Repository navigation
fix(#7550)!: export per-model usage as trace components - #7553
fullsend-ai-coder[bot] wants to merge 6 commits into
Conversation
Pi already records one PerModelUsage entry per child model plus the parent, but the agent span exported only the parent identity with totals that included every child. Mixed-vendor spend was therefore attributed to the parent in OTLP backends. When an iteration has more than one spec, emit one Internal usage child of the agent span per entry, carrying that spec's serving endpoint, model, tokens, request count, and cost. Mark the agent span as a rollup and omit gen_ai.usage.* there so backends that auto-sum those keys cannot double-count. Single-model runs and runtimes without PerModelUsage keep the previous agent span. The query contract is documented: sum fullsend.usage.component children; do not add the parent rollup. metrics.json is unchanged. Closes #7550
|
🤖 Finished Review · ✅ Success · Started 4:48 PM UTC · Completed 5:11 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $7.48 |
Site previewPreview: https://559fd1d1-site.fullsend-ai.workers.dev Commit: |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
Risk Assessment: moderate (2/5) DetailsTier 1 signals remain unchanged with zero protected paths, no security-sensitive files, and no dependency modifications despite a large diff, and while Tier 2 reflects high churn across central runner files, full acceptance criteria coverage for the linked issue and re-review anchoring preserve the moderate score of 2. Previous runRisk Assessment: moderate (2/5) DetailsA large diff with breaking trace attribute modifications is balanced by bot authorship, zero protected or security-sensitive paths, complete acceptance criteria coverage on the linked issue, and extensive test additions. Previous run (2)Risk Assessment: moderate (2/5) DetailsTier 1 signals are unchanged from the prior review (bot author, no protected/security paths, no CI/dependency changes, large diff), and Tier 2 remains driven by the same high-churn hotspot files (run.go, architecture.md, runtimes.md) with no new files or protected paths introduced by the follow-up commit, which only refined an already-new usage_spans.go/test.go pair and a doc; per re-review anchoring the prior moderate score of 2 is preserved. Previous run (3)Risk Assessment: moderate (2/5) DetailsTier 1 is low-risk (trusted bot author, no protected/security-sensitive paths, no CI/dependency changes) despite a large diff; Tier 2 is elevated because touched files (run.go, architecture.md, runtimes.md) are high-churn hotspots with heavy historical fix/revert activity; Tier 3 is moderate given plausible-but-unverified acceptance-criteria coverage and no rollback flag for a behavior-modifying telemetry change. The 50/30/20 weighted blend rounds to a moderate overall score of 2. |
ReviewFindingsLow
Earlier findings
2 finding(s) below the Previous run<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJyZXF1ZXN0LWNoYW5nZXMiLCJmaW5kaW5ncyI6W3sic2V2ZXJpdHkiOiJoaWdoIiwiY2F0ZWdvcnkiOiJzZWNyZXQtZXhwb3N1cmUiLCJmaWxlIjoiaW50ZXJuYWwvY2xpL3VzYWdlX3NwYW5zLmdvIiwiaWQiOiJmX2I4OTRjNWIzMTAzZGRhNDIiLCJsaW5lIjoxODUsImFjdGlvbmFibGUiOnRydWUsImRpbWVuc2lvbiI6InNlY3VyaXR5In0seyJzZXZlcml0eSI6Im1lZGl1bSIsImNhdGVnb3J5IjoidGVzdC1pbmFkZXF1YXRlIiwiZmlsZSI6ImludGVybmFsL2NsaS91c2FnZV9zcGFuc190ZXN0LmdvIiwiaWQiOiJmXzUxNDE0ZDUxMTExZmM3Y2EiLCJsaW5lIjo0NTMsImFjdGlvbmFibGUiOnRydWUsImRpbWVuc2lvbiI6ImNvcnJlY3RuZXNzIn0seyJzZXZlcml0eSI6ImxvdyIsImNhdGVnb3J5IjoiZG9jLXN0eWxlIiwiZmlsZSI6ImRvY3MvZ3VpZGVzL2luZnJhc3RydWN0dXJlL2Rpc3RyaWJ1dGVkLXRyYWNpbmcubWQiLCJpZCI6ImZfNmIwMGIwNjRlYjEzY2VlYiIsImxpbmUiOjQ4MywiYWN0aW9uYWJsZSI6dHJ1ZSwiZGltZW5zaW9uIjoic3R5bGUtY29udmVudGlvbnMifSx7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJicmVha2luZy1zY2hlbWEiLCJmaWxlIjoiaW50ZXJuYWwvY2xpL3J1bi5nbyIsImlkIjoiZl82NGQwNjkzOGE5NzE3YjY5IiwiYWN0aW9uYWJsZSI6ZmFsc2UsImRpbWVuc2lvbiI6ImNyb3NzLXJlcG8tY29udHJhY3RzIn0seyJzZXZlcml0eSI6ImxvdyIsImNhdGVnb3J5IjoiYmFja3dhcmQtaW5jb21wYXRpYmxlIiwiZmlsZSI6ImludGVybmFsL2V2YWxtZWFzdXJlL2ZpdG5lc3MuZ28iLCJpZCI6ImZfYTljZTdmYWEwOWJjOGM5MiIsImxpbmUiOjIwNSwiYWN0aW9uYWJsZSI6ZmFsc2UsImRpbWVuc2lvbiI6ImNyb3NzLXJlcG8tY29udHJhY3RzIn1dfQ== --> ReviewFindingsHigh
Medium
Low
Labels: PR changes Go telemetry code and the tracing documentation. Next steps:
Previous run (2)ReviewFindingsHigh
Medium
Low
Next steps:
Previous run (3)ReviewFindingsHigh
Medium
Low
Next steps:
|
|
🤖 Finished Fix · ✅ Success · Started 5:13 PM UTC · Completed 5:34 PM UTC Commit: Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $4.47 |
… DoS Address review feedback on PR #7553: - Scan each PerModelUsage key through security.OutputPipeline() before it reaches a usage span's name or attributes, matching the treatment toolSpanTracker.safeName already gives the structurally identical execute_tool case. The key is read back from a sandbox-writable file (foldPiSubagentUsage, internal/runtime/pi_subagents.go) and scanOutputFiles skips the telemetry JSONL on the invariant that stream-derived span strings are scanned before assembly. - Route gen_ai.system / gen_ai.provider.name through boundedStringAttr instead of the unbounded stringAttr. - Cap the number of usage children one iteration may emit (maxUsageSpansPerIteration, derived from tool_spans.go's maxToolSpansPerIteration) and record the drop count on the parent agent span (fullsend.usage.dropped), mirroring the execute_tool overflow contract (fullsend.tool_spans.dropped) so an adversarial burst of distinct model keys can't evict the agent span from the OTLP queue. - Give the usage span name its own length bound (maxUsageSpanNameBytes) instead of reusing tool_spans.go's tool-name-specific constant. - Correct the tracing doc's per-model cost rounding claim: independently rounded component costs are not guaranteed to sum to the rounded iteration rollup (documented with the review's own $0.006 x 3 example), and add a breaking-change callout for the gen_ai.usage.* removal from mixed-model agent spans, matching the doc's existing callout convention. Not addressed here (recorded in the fix agent's structured output): PR description/title updates are post-script/human responsibilities outside this agent's mutation scope. BREAKING CHANGE: agent spans for mixed-model Pi iterations (introduced by this PR) no longer carry gen_ai.usage.input_tokens / output_tokens / cache_creation.input_tokens / cache_read.input_tokens. Downstream consumers that sum those keys directly off the agent span must instead filter on fullsend.usage.component children. Single-model iterations are unaffected. The PR title also needs the "!" suffix before merge. Addresses #7553
🔧 Fix agent — iteration 1 (bot-triggered)Addressed all 7 review findings. Code: scan PerModelUsage keys through security.OutputPipeline() before they reach a usage span (secret-exposure), bound gen_ai.system/provider.name (part of the same finding), cap usage-span cardinality with overflow recorded on the parent span (data-exposure), and give usage spans their own name-length constant instead of reusing tool_spans.go's (naming-conventions). Docs: corrected the false per-model cost rounding claim with a demonstrating test (logic-error) and added a breaking-change callout for the gen_ai.usage.* removal (api-compatibility). Commit message: added the Fixed (7):
Tests: passed Decision points
Next steps:
|
|
🤖 Finished Review · ✅ Success · Started 5:36 PM UTC · Completed 5:54 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $6.87 |
|
🤖 Finished Fix · ✅ Success · Started 5:57 PM UTC · Completed 6:17 PM UTC Commit: Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $2.40 |
Address review feedback on PR #7553: - docs/guides/dev/tracing.md: the "usage spans" section documented the happy path but omitted the security/robustness mechanisms the prior fix-iteration added (security.OutputPipeline() sanitization of each PerModelUsage key, the maxUsageSpanNameBytes name bound, the maxUsageSpansPerIteration cap, and the fullsend.usage.dropped overflow attribute), unlike the neighboring execute_tool spans section, which documents its equivalent protections in full. - docs/guides/infrastructure/distributed-tracing.md: the fullsend.usage.dropped attribute description and the "Per-model usage components" section referenced "the usage-span cap" without ever stating the actual limit (1,024), unlike fullsend.tool_spans.dropped's description, which cites the 1,024-span cap explicitly. Not addressed here (recorded in the fix agent's structured output): - breaking-change-authorization (PR title missing the "!" suffix) and adr-process (PR description not mentioning the ADR 0050 annotation) are PR-metadata edits outside this agent's mutation scope, per the prior fix-iteration's same determination. - test-weakened (usage_spans_test.go:453): verified against the actual code and disagreed - the test's filler value already exceeds both the redactor's 8-char structural-match threshold and its 10-char prefix+ellipsis masking threshold, and the test's assert.Equal only passes because real redaction fires (it would fail against the much longer unredacted string otherwise). Addresses #7553
🔧 Fix agent — iteration 2 (bot-triggered)Addressed the two actionable documentation findings by extending the usage-spans doc sections with the security/cap details they were missing. Verified and disagreed with the test-weakened finding (the test's filler already clears the redactor's thresholds and exercises real redaction). Recorded the PR-title and PR-description findings as valid but outside this agent's forge-mutation scope, consistent with the prior fix-iteration's determination. All non-Fullsend project CI checks on the PR are currently passing. Fixed (2):
Disagreed (4):
Tests: passed CI inspections (12):
Decision points
Next steps:
|
|
🤖 Finished Review · ❌ Failure (post-script /home/runner/work/fullsend/fullsend/.fullsend/.fullsend-cache/resources/sha256/f93bf432ead6bfd81509e281c866ca9d50995c3561249ae53ed09c58df8e249d/scripts/post-review.sh failed: exit status 5) · Started 6:19 PM UTC · Completed 6:40 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $4.44 |
|
/fs-review |
|
🤖 Finished Review · ❌ Failure (post-script /home/runner/work/fullsend/fullsend/.fullsend/.fullsend-cache/resources/sha256/df4695a536d58e6f4754a7a25396e74d5ffce2c2741bd06854f47dfa37378d7f/scripts/post-review.sh failed: exit status 5) · Started 5:26 PM UTC · Completed 5:42 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $3.91 |
Merge main and resolve the four tracing documentation conflicts while preserving current content-capture, Codex usage, and cost guidance. Use the iteration serving endpoint for Claude and Codex model-id usage components, including gateway routes, and cover both with regression tests. Replace a synthetic API-key fixture that triggered the staged secret scan. BREAKING CHANGE: Mixed-model iterations emit input, output, and cache token attributes on fullsend.usage.component children instead of the agent rollup. Consumers must sum the components and exclude the parent rollup. Signed-off-by: Adam Scerra <ascerra@redhat.com>
Signed-off-by: Adam Scerra <ascerra@redhat.com>
|
🤖 Finished Review · ❌ Failure (post-script /home/runner/work/fullsend/fullsend/.fullsend/.fullsend-cache/resources/sha256/eb05ab46875ecfb67f461de43e976d2cadf0913251419c784718aba608589835/scripts/post-review.sh failed: exit status 5) · Started 11:55 AM UTC · Completed 12:08 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $2.85 |
|
/fs-review |
|
🤖 Finished Review · ✅ Success · Started 12:37 PM UTC · Completed 12:56 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $3.10 |
|
🤖 Finished Fix · ✅ Success · Started 12:58 PM UTC · Completed 1:09 PM UTC Commit: Runtime: claude · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $0.55 |
Route PerModelUsage keys through redactText (runner env literal pass, whole-string masking, URL userinfo/query/fragment masking) instead of a bare output-pipeline scan, matching toolSpanTracker.safeName. Add a regression test for URL and runner-env redaction, rewrap long doc paragraphs, and update the dev tracing guide. Notes: pre-commit could not fetch hook repos (certificate failure); gofmt, go vet and trailing-whitespace checks were run directly. Unrelated TestStartFetchService_* tests fail in the sandbox (listener permission denied), with and without this change. Addresses #7553 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
🔧 Fix agent — iteration 3 (bot-triggered)Fixed the high-severity secret-exposure finding by routing usage-span model specs through redactText and added a regression test. Rewrapped the long doc paragraphs. Disagreed with the test-inadequate finding, because the file already uses a 26-character value and the test passes. Declined the two optional low-severity cross-repo notes. Pre-commit could not fetch its hook repositories (certificate failure), so gofmt, go vet and a whitespace check were run directly. Unrelated TestStartFetchService_* tests fail in the sandbox because listeners are denied; they fail the same way without this change. Fixed (2):
Disagreed (3):
Tests: passed CI inspections (6):
Next steps:
|
|
🤖 Finished Review · ✅ Success · Started 1:11 PM UTC · Completed 1:28 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $2.57 |
Superseded by updated review
Summary
Mixed-model iterations already record per-model usage in
metrics.json, but their agent span attributes attributed every child's tokens and cost to the parent model. Export oneusage <model>child per breakdown entry so telemetry consumers can attribute usage to the model and serving endpoint that produced it.Closes #7550.
Changes
per_model_usagehas more than one entry; mark the parentfullsend.usage.rollup=trueand retain its cost rollup. Single-model iterations keep their existing token attributes.fullsend.usage.dropped.docs/ADRs/0050-distributed-tracing-instrumentation.md) describing usage attribution without rewriting its original decision.7aedba229). Replace a synthetic API-key test fixture with an explicit placeholder so the secret scan passes.Breaking change
Mixed-model iterations move
gen_ai.usage.input_tokens,gen_ai.usage.output_tokens,gen_ai.usage.cache_creation.input_tokens, andgen_ai.usage.cache_read.input_tokensfrom the agent span to children markedfullsend.usage.component=true. Consumers must sum those components and exclude the parent rollup. Reasoning tokens remain on the agent span. Historical traces andmetrics.jsonare unchanged.Validation
go test ./...: all 74 tested packages passed after removing a relative entry from the local test process PATH.