Skip to content

fix(#7550)!: export per-model usage as trace components - #7553

Open
fullsend-ai-coder[bot] wants to merge 6 commits into
mainfrom
agent/7550-per-model-usage-spans
Open

fullsend-ai-coder[bot] wants to merge 6 commits into
mainfrom
agent/7550-per-model-usage-spans

Conversation

@fullsend-ai-coder

@fullsend-ai-coder fullsend-ai-coder Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Mixed-model iterations already record per-model usage in metrics.json, but their agent span attributes attributed every child's tokens and cost to the parent model. Export one usage <model> child per breakdown entry so telemetry consumers can attribute usage to the model and serving endpoint that produced it.

Closes #7550.

Changes

  • Emit components when per_model_usage has more than one entry; mark the parent fullsend.usage.rollup=true and retain its cost rollup. Single-model iterations keep their existing token attributes.
  • Parse Pi provider/model specs; preserve Claude and Codex model ids and use their iteration serving endpoint, including gateway routes. Unknown models remain in the unknown bucket.
  • Sanitize model specs through the output security pipeline, bound names and attributes, cap components at 1,024 per iteration, and report overflow as fullsend.usage.dropped.
  • Teach EM-001 trace fitness to recognize usage components and document query, rounding, and overflow behavior.
  • Add a dated annotation to Accepted ADR 0050 (docs/ADRs/0050-distributed-tracing-instrumentation.md) describing usage attribution without rewriting its original decision.
  • Resolve four documentation conflicts and incorporate current main (7aedba229). Replace a synthetic API-key test fixture with an explicit placeholder so the secret scan passes.

Breaking change

Mixed-model iterations move gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_creation.input_tokens, and gen_ai.usage.cache_read.input_tokens from the agent span to children marked fullsend.usage.component=true. Consumers must sum those components and exclude the parent rollup. Reasoning tokens remain on the agent span. Historical traces and metrics.json are unchanged.

Validation

  • go test ./...: all 74 tested packages passed after removing a relative entry from the local test process PATH.
  • Targeted telemetry regression tests passed after the final main merge, including Claude/Codex direct and gateway routes, unknown models, cancellation, cardinality limits, and real redaction.
  • Approximate changed-production-line coverage: 98.5%.
  • Applicable pre-commit checks passed, including vet, formatting, secret scanning, ADR checks, and Markdown links.
  • Race tests could not run locally because the environment lacks a C compiler; fresh GitHub CI is running.
  • Human-directed update commits carry DCO sign-off.

Pi already records one PerModelUsage entry per child model plus the
parent, but the agent span exported only the parent identity with
totals that included every child. Mixed-vendor spend was therefore
attributed to the parent in OTLP backends.

When an iteration has more than one spec, emit one Internal usage
child of the agent span per entry, carrying that spec's serving
endpoint, model, tokens, request count, and cost. Mark the agent
span as a rollup and omit gen_ai.usage.* there so backends that
auto-sum those keys cannot double-count. Single-model runs and
runtimes without PerModelUsage keep the previous agent span.

The query contract is documented: sum fullsend.usage.component
children; do not add the parent rollup. metrics.json is unchanged.

Closes #7550
@fullsend-ai-coder
fullsend-ai-coder Bot requested a review from a team September 22, 2026 16:46
@fullsend-ai-coder fullsend-ai-coder Bot added the ready-for-review Triggers review agent dispatch label Sep 22, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 4:48 PM UTC · Completed 5:11 PM UTC

Commit: 0e0dba3 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $7.48

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Site preview

Preview: https://559fd1d1-site.fullsend-ai.workers.dev

Commit: fbf6fd2836114413cd4d89ae1617c85e70c35dd6

@codecov

codecov Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.82353% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/evalmeasure/fitness.go 93.33% 0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Risk Assessment: moderate (2/5)

Details

Tier 1 signals remain unchanged with zero protected paths, no security-sensitive files, and no dependency modifications despite a large diff, and while Tier 2 reflects high churn across central runner files, full acceptance criteria coverage for the linked issue and re-review anchoring preserve the moderate score of 2.

Previous run

Risk Assessment: moderate (2/5)

Details

A large diff with breaking trace attribute modifications is balanced by bot authorship, zero protected or security-sensitive paths, complete acceptance criteria coverage on the linked issue, and extensive test additions.

Previous run (2)

Risk Assessment: moderate (2/5)

Details

Tier 1 signals are unchanged from the prior review (bot author, no protected/security paths, no CI/dependency changes, large diff), and Tier 2 remains driven by the same high-churn hotspot files (run.go, architecture.md, runtimes.md) with no new files or protected paths introduced by the follow-up commit, which only refined an already-new usage_spans.go/test.go pair and a doc; per re-review anchoring the prior moderate score of 2 is preserved.

Previous run (3)

Risk Assessment: moderate (2/5)

Details

Tier 1 is low-risk (trusted bot author, no protected/security-sensitive paths, no CI/dependency changes) despite a large diff; Tier 2 is elevated because touched files (run.go, architecture.md, runtimes.md) are high-churn hotspots with heavy historical fix/revert activity; Tier 3 is moderate given plausible-but-unverified acceptance-criteria coverage and no rollback flag for a behavior-modifying telemetry change. The 50/30/20 weighted blend rounds to a moderate overall score of 2.

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Review

Findings

Low

  • [breaking-schema] internal/cli/run.go — Mixed-model iterations no longer put gen_ai.usage.* token attributes on the parent agent span. Usage moves to usage <model> child spans, and the parent is marked fullsend.usage.rollup=true. External consumers (dashboards, MLflow queries) that read tokens from the agent span will see them missing for mixed-model runs. The PR is marked breaking (!) and the docs describe the migration, so this is advisory only.
    Remediation: Keep the breaking marker and the migration notes in the docs.

Earlier findings

  • internal/cli/usage_spans.go:185, secret-exposure — resolved. sanitizeModelSpec now runs the whole spec through redactText (runner env literal pass, whole-string masking, URL userinfo/query/fragment masking) before splitting or truncating, and finalizeAgentSpan passes the tracker's runnerEnv.
  • internal/cli/usage_spans_test.go:453, test-inadequate — reclassified to info (withdrawn). The raw fixture is a 22-character value, so it does satisfy the env_assignment {8,} pattern. It masks to zzzz..., and the assertions are consistent. The earlier concern came from a display-masked literal.
  • docs/guides/infrastructure/distributed-tracing.md:483, doc-style — resolved. The three paragraphs are re-wrapped to about 80 columns.
  • internal/evalmeasure/fitness.go:205, backward-incompatible — reclassified to info. usageOK checks the run and agent spans first and only adds a child-component fallback, so nothing that passed before can fail.

2 finding(s) below the low severity threshold were not posted.

Previous run

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJyZXF1ZXN0LWNoYW5nZXMiLCJmaW5kaW5ncyI6W3sic2V2ZXJpdHkiOiJoaWdoIiwiY2F0ZWdvcnkiOiJzZWNyZXQtZXhwb3N1cmUiLCJmaWxlIjoiaW50ZXJuYWwvY2xpL3VzYWdlX3NwYW5zLmdvIiwiaWQiOiJmX2I4OTRjNWIzMTAzZGRhNDIiLCJsaW5lIjoxODUsImFjdGlvbmFibGUiOnRydWUsImRpbWVuc2lvbiI6InNlY3VyaXR5In0seyJzZXZlcml0eSI6Im1lZGl1bSIsImNhdGVnb3J5IjoidGVzdC1pbmFkZXF1YXRlIiwiZmlsZSI6ImludGVybmFsL2NsaS91c2FnZV9zcGFuc190ZXN0LmdvIiwiaWQiOiJmXzUxNDE0ZDUxMTExZmM3Y2EiLCJsaW5lIjo0NTMsImFjdGlvbmFibGUiOnRydWUsImRpbWVuc2lvbiI6ImNvcnJlY3RuZXNzIn0seyJzZXZlcml0eSI6ImxvdyIsImNhdGVnb3J5IjoiZG9jLXN0eWxlIiwiZmlsZSI6ImRvY3MvZ3VpZGVzL2luZnJhc3RydWN0dXJlL2Rpc3RyaWJ1dGVkLXRyYWNpbmcubWQiLCJpZCI6ImZfNmIwMGIwNjRlYjEzY2VlYiIsImxpbmUiOjQ4MywiYWN0aW9uYWJsZSI6dHJ1ZSwiZGltZW5zaW9uIjoic3R5bGUtY29udmVudGlvbnMifSx7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJicmVha2luZy1zY2hlbWEiLCJmaWxlIjoiaW50ZXJuYWwvY2xpL3J1bi5nbyIsImlkIjoiZl82NGQwNjkzOGE5NzE3YjY5IiwiYWN0aW9uYWJsZSI6ZmFsc2UsImRpbWVuc2lvbiI6ImNyb3NzLXJlcG8tY29udHJhY3RzIn0seyJzZXZlcml0eSI6ImxvdyIsImNhdGVnb3J5IjoiYmFja3dhcmQtaW5jb21wYXRpYmxlIiwiZmlsZSI6ImludGVybmFsL2V2YWxtZWFzdXJlL2ZpdG5lc3MuZ28iLCJpZCI6ImZfYTljZTdmYWEwOWJjOGM5MiIsImxpbmUiOjIwNSwiYWN0aW9uYWJsZSI6ZmFsc2UsImRpbWVuc2lvbiI6ImNyb3NzLXJlcG8tY29udHJhY3RzIn1dfQ== -->

Review

Findings

High

  • [secret-exposure] internal/cli/usage_spans.go:185 — sanitizeModelSpec runs model specs only through security.OutputPipeline(). It does not use the shared redactText helper that toolSpanTracker.safeName uses (tool_spans.go:181), so three protections are missing: the runner-env literal pass, whole-string masking when the normalizer stripped something or the text has invalid UTF-8, and maskURLs (URL userinfo, query and fragment). A PerModelUsage key comes from a sandbox-writable file, so a key such as openai/https://example.invalid/?sig=opaque-value, or one containing a runner-env secret that no pattern matches, is exported unchanged in the span name, gen_ai.request.model and fullsend.usage.model_spec. scanOutputFiles skips the telemetry JSONL on the assumption that span strings were already scanned, so nothing removes it later.
    Remediation: Sanitize model specs with redactText(pipeline, runnerEnv, spec, &findings) before splitting or truncating them, passing in the runner environment. Add regression tests showing that URL userinfo, query parameters, fragments and runner-env literals never appear in any usage span name or string attribute.

Medium

  • [test-inadequate] internal/cli/usage_spans_test.go:453 — TestEmitPerModelUsageSpans_SanitizesSecretInSpec sets secretValue = "zzzz...", which is 7 characters, but the env_assignment pattern (internal/security/redactor.go:252) requires a value of 8 or more characters, so the value is never redacted. The assertions also contradict each other: lines 468-471 assert no span name or attribute contains zzzz..., while line 480 requires gen_ai.request.model to equal model API_KEY=***, which contains it. The test cannot pass and does not exercise the intended masking path. Remediation: Use a synthetic value of at least 10 characters, for example strings.Repeat("z", 16)`, and keep the expected masked form. Assert that the full original value is absent from every span name and attribute and that the masked model is present.

Low

  • [doc-style] docs/guides/infrastructure/distributed-tracing.md:483 — Under "Per-model usage components", the paragraphs at lines 483, 485 and 487 are each a single very long line of several hundred characters. Surrounding paragraphs are wrapped at about 80 columns, as docs/contributing/documentation.md requires.
    Remediation: Rewrap lines 483, 485 and 487 at word boundaries around 80 columns to match the surrounding paragraphs.
  • [breaking-schema] internal/cli/run.go — On mixed-model iterations (len(PerModelUsage) > 1), the agent span no longer emits gen_ai.usage.input_tokens, output_tokens, cache_creation.input_tokens or cache_read.input_tokens. Those values move to the new usage <model> child spans and the agent span gets fullsend.usage.rollup=true instead. The break is declared (the ! in the PR title and a callout in the tracing guide). Downstream consumers that read gen_ai.usage.* from agent spans will see missing values on mixed-model iterations, and there is no dual-emit transition period.
    Remediation: Optional: add a migration note to the release notes: sum gen_ai.usage.* over spans with fullsend.usage.component=true, and treat fullsend.usage.rollup=true as meaning "no tokens on this span".
  • [backward-incompatible] internal/evalmeasure/fitness.go:205 — EM-001 usageOK now passes when any span in the trace with fullsend.usage.component=true carries gen_ai.usage.*. The check is not tied to a parent agent span marked as a rollup, so a stray component span could satisfy it. The code comment and docs say "mixed-model Pi", but Claude and Codex mixed iterations also emit component spans.
    Remediation: Optionally accept only components whose parent is an agent span with fullsend.usage.rollup=true, and widen the comment and doc wording beyond Pi.

Labels: PR changes Go telemetry code and the tracing documentation.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR
Previous run (2)

Review

Findings

High

  • [breaking-change-authorization] internal/cli/run.go:3576 — Partially resolved since the prior review. The follow-up commit's message now correctly carries the ! suffix (fix(#7550)!:) and a BREAKING CHANGE: trailer. However, the PR title — the source GoReleaser uses to build release notes per COMMITS.md — still reads fix(#7550): export per-model usage as mixed-model Pi span children with no ! suffix. COMMITS.md states an unmarked PR title makes a breaking change invisible in release notes even when the commit message is marked, and AGENTS.md classifies a missing ! as an important-severity finding. The commit message itself acknowledges the gap ("The PR title also needs the "!" suffix before merge") but does not close it.
    Remediation: Update the PR title to fix(#7550)!: export per-model usage as mixed-model Pi span children before merge.

Medium

  • [test-weakened] internal/cli/usage_spans_test.go:453 — TestEmitPerModelUsageSpans_SanitizesSecretInSpec exercises the real security.OutputPipeline() path (not a mock), but the payload can't trigger the redactor: the env_assignment pattern requires an 8+ character value (internal/security/redactor.go), while secretValue is 7 characters ("zzzz..."). With no finding raised, sanitizeModelSpec returns the spec unchanged, so the assertions pass without ever exercising redaction — the test can't actually prove the secret-exposure fix works against a credential-shaped value.
    Remediation: Use a filler of at least 10 characters that env_assignment matches (e.g. "zzzzxxxxxxxx"), assert the raw value is absent from every span name/attribute, and assert the redactor's masked form appears instead of asserting equality against the unsanitized string.

Low

  • [adr-process] docs/ADRs/0050-distributed-tracing-instrumentation.md:182 — This PR appends a policy-compliant dated annotation to Accepted ADR 0050, but docs/contributing/adrs.md expects edits to an Accepted ADR to be explicitly called out in the PR description. The current description still doesn't mention the ADR 0050 annotation.
    Remediation: Add an explicit line to the PR description noting the ADR 0050 annotation.

  • [documentation-completeness] docs/guides/dev/tracing.md:144 — The "usage spans" section documents emitPerModelUsageSpans and its attributes, but omits the security/robustness mechanisms this PR's follow-up commit added: the 1,024-spec cap (maxUsageSpansPerIteration), the fullsend.usage.dropped overflow attribute, security.OutputPipeline() sanitization, and maxUsageSpanNameBytes truncation. The neighboring "execute_tool spans" section documents its equivalent protections in full.
    Remediation: Update the "usage spans" section to document the cap, the overflow attribute, sanitization, and name-length bounding.

  • [documentation-accuracy] docs/guides/infrastructure/distributed-tracing.md:260 — The attribute table describes fullsend.usage.dropped as present when the breakdown exceeds "the usage-span cap," but the actual limit (1,024, via maxUsageSpansPerIteration) is never stated in this document, unlike fullsend.tool_spans.dropped's description which explicitly cites "the 1,024-span cap."
    Remediation: State the 1,024 limit explicitly in the fullsend.usage.dropped description and in "Per-model usage components."

  • [sub-agent-failure] The challenger adjudication pass returned a malformed response (a bare array instead of the expected {adjudicated_findings, removed_findings} object) that silently dropped 2 of 6 input findings with no removal justification. This review falls back to the pre-challenger, per-dimension finding set above rather than trusting the malformed output.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR
Previous run (3)

Review

Findings

High

  • [secret-exposure] internal/cli/usage_spans.go:137 — Per-model usage children copy sandbox/agent-writable PerModelUsage map keys (foldPiSubagentUsage in internal/runtime/pi_subagents.go keys entries by rec.Model, read from a JSONL file the sandbox can write; its own comments acknowledge a malformed line may be "an agent-authored line") into OTel span attributes and the span name without running them through security.OutputPipeline(). scanOutputFiles (internal/cli/run.go) explicitly skips run-telemetry.jsonl on the stated invariant that tool names/ids were already scanned at assembly via toolSpanTracker.safeName/safeID before landing on a span. usageComponentSpanAttrs uses plain stringAttr for gen_ai.system/gen_ai.provider.name (no length bound, no content scan) and boundedStringAttr for gen_ai.request.model/fullsend.usage.model_spec (size-bounded only), and usageSpanName only does ToValidUTF8+truncate. This breaks the exemption invariant for a structurally identical untrusted-name case the codebase already has an established mitigation for.
    Remediation: Scan spec (and/or the split provider/model) through security.OutputPipeline() before recording span attributes and the span name, matching toolSpanTracker.safeName's pattern (redact or drop on findings). Route gen_ai.system/gen_ai.provider.name through boundedStringAttr rather than stringAttr.

  • [breaking-change-authorization] internal/cli/run.go:3576 — For mixed-model Pi iterations, agentSpanEndAttrs now omits gen_ai.usage.input_tokens/output_tokens/cache_creation.input_tokens/cache_read.input_tokens from the parent agent span (previously the rollup total across all children). COMMITS.md lists "Output format changes that downstream consumers parse" as a breaking-change criterion and requires the ! suffix plus a BREAKING CHANGE: trailer; AGENTS.md separately calls a missing ! an important-severity finding. Neither the PR title (fix(#7550): export per-model usage as mixed-model Pi span children) nor the commit message carries ! or a BREAKING CHANGE: trailer. Any existing OTLP consumer summing gen_ai.usage.* directly off the agent span (not only MLflow, which this PR targets) will silently see missing/zero token counts for mixed-model iterations going forward.
    Remediation: Mark the PR title and commit with the ! suffix (e.g. fix(#7550)!: ...) and add a BREAKING CHANGE: trailer describing the emission-shape change and migration path (filter on fullsend.usage.component children instead of the parent span's gen_ai.usage.* for mixed-model iterations).

Medium

  • [logic-error] docs/guides/infrastructure/distributed-tracing.md:298 — The documented per-model usage query contract claims independently rounded component fullsend.cost_usd values sum to the rounded iteration rollup ("they already sum to it"; "so they sum to the rounded iteration cost"). This is false: usageComponentSpanAttrs emits roundUSD(u.CostUSD) per component (internal/cli/usage_spans.go:144) while the parent agent span emits roundUSD(m.TotalCostUSD) of the raw, unrounded sum (internal/cli/run.go:3577). sum(round(x_i)) is not always round(sum(x_i)) — e.g. three components at $0.006 each round individually to $0.01 each (sum $0.03) while the raw total $0.018 rounds to $0.02. TestEmitPerModelUsageSpans_MixedVertexXAIGoogle uses component costs already at cent precision (0.90, 0.20, 0.13), so it cannot catch this. The file already documents an analogous agent-vs-root rounding caveat a few paragraphs later.
    Remediation: Align the query-contract prose with the existing agent-vs-root rounding caveat (children may differ from roundUSD(iteration total) by up to $0.01 per component). Add a test with sub-cent per-component costs demonstrating the real behavior, or allocate the rounding remainder onto one component so sums exactly reconcile and test that invariant.

  • [data-exposure] internal/cli/usage_spans.go:118 — emitPerModelUsageSpans starts and ends one Internal child span per PerModelUsage key with no cardinality cap. tool_spans.go caps the structurally similar execute_tool span stream at maxToolSpansPerIteration = 1024 because the OTLP batch processor's default queue (2048, drop-newest) would otherwise evict the agent span under an adversarial burst. PerModelUsage keys come from a sandbox-writable file capped at 1 MiB (piSubagentUsageMaxBytes, ~4k child records per its own comment); isMixedModelUsage only requires len(PerModelUsage) > 1, so a legitimate parent entry plus attacker-controlled distinct keys is enough to emit thousands of children in a burst right before the agent span ends.
    Remediation: Cap the number of usage children emitted per iteration (reuse maxToolSpansPerIteration or a dedicated bound) and record overflow on the parent agent span the way recordToolSpanOverflow does for execute_tool spans.

Low

  • [api-compatibility] docs/guides/infrastructure/distributed-tracing.md:227 — This file already uses a "> Breaking change — Pi runtime:" blockquote convention for an earlier gen_ai.system rename. This PR's equally consumer-visible removal of gen_ai.usage.* from mixed-model agent spans is documented only as attribute-table/section prose, without a matching callout where an integrator would look for contract breaks.
    Remediation: Add a "> Breaking change — mixed-model Pi iterations:" callout stating that agent spans for mixed-model Pi iterations no longer carry gen_ai.usage.* and that consumers must filter on fullsend.usage.component children instead.

  • [adr-process] docs/ADRs/0050-distributed-tracing-instrumentation.md:181 — This PR appends a new dated annotation to Accepted ADR 0050. The annotation itself is policy-compliant (consistent style, no rewrite of Context/Decision/Consequences), but docs/contributing/adrs.md expects edits to an Accepted ADR to be explicitly called out in the PR description; this PR's "Changes" section only implies the touch via general doc-update bullets.
    Remediation: Add an explicit line to the PR description noting the ADR 0050 annotation, per the ADR review checklist.

  • [naming-conventions] internal/cli/usage_spans.go:96 — usageSpanName bounds the model-name component using maxToolSpanNameBytes, a constant defined and documented in tool_spans.go as specifically bounding the tool-name portion of execute_tool spans. Reusing it couples an unrelated span kind to tool-span limits.
    Remediation: Define a dedicated constant in internal/cli/usage_spans.go (e.g. maxUsageSpanNameBytes = 128) and use it in usageSpanName.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-coder

fullsend-ai-coder Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 5:13 PM UTC · Completed 5:34 PM UTC

Commit: 0e0dba3 · View workflow run →

Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $4.47

… DoS

Address review feedback on PR #7553:

- Scan each PerModelUsage key through security.OutputPipeline() before it
  reaches a usage span's name or attributes, matching the treatment
  toolSpanTracker.safeName already gives the structurally identical
  execute_tool case. The key is read back from a sandbox-writable file
  (foldPiSubagentUsage, internal/runtime/pi_subagents.go) and
  scanOutputFiles skips the telemetry JSONL on the invariant that
  stream-derived span strings are scanned before assembly.
- Route gen_ai.system / gen_ai.provider.name through boundedStringAttr
  instead of the unbounded stringAttr.
- Cap the number of usage children one iteration may emit
  (maxUsageSpansPerIteration, derived from tool_spans.go's
  maxToolSpansPerIteration) and record the drop count on the parent agent
  span (fullsend.usage.dropped), mirroring the execute_tool overflow
  contract (fullsend.tool_spans.dropped) so an adversarial burst of
  distinct model keys can't evict the agent span from the OTLP queue.
- Give the usage span name its own length bound (maxUsageSpanNameBytes)
  instead of reusing tool_spans.go's tool-name-specific constant.
- Correct the tracing doc's per-model cost rounding claim: independently
  rounded component costs are not guaranteed to sum to the rounded
  iteration rollup (documented with the review's own $0.006 x 3 example),
  and add a breaking-change callout for the gen_ai.usage.* removal from
  mixed-model agent spans, matching the doc's existing callout convention.

Not addressed here (recorded in the fix agent's structured output):
PR description/title updates are post-script/human responsibilities
outside this agent's mutation scope.

BREAKING CHANGE: agent spans for mixed-model Pi iterations (introduced by
this PR) no longer carry gen_ai.usage.input_tokens / output_tokens /
cache_creation.input_tokens / cache_read.input_tokens. Downstream consumers
that sum those keys directly off the agent span must instead filter on
fullsend.usage.component children. Single-model iterations are unaffected.
The PR title also needs the "!" suffix before merge.

Addresses #7553
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 1 (bot-triggered)

Addressed all 7 review findings. Code: scan PerModelUsage keys through security.OutputPipeline() before they reach a usage span (secret-exposure), bound gen_ai.system/provider.name (part of the same finding), cap usage-span cardinality with overflow recorded on the parent span (data-exposure), and give usage spans their own name-length constant instead of reusing tool_spans.go's (naming-conventions). Docs: corrected the false per-model cost rounding claim with a demonstrating test (logic-error) and added a breaking-change callout for the gen_ai.usage.* removal (api-compatibility). Commit message: added the ! suffix and a BREAKING CHANGE trailer (breaking-change-authorization). Two findings need a PR-metadata edit (title ! suffix; a PR-description line noting the ADR 0050 annotation) that this agent cannot perform — flagged in the actions list for a human or the post-script to complete before merge.

Fixed (7):

  1. [secret-exposure] usage_spans.go:137 — PerModelUsage keys reached spans unscanned (internal/cli/usage_spans.go): PerModelUsage keys (agent/sandbox-writable via foldPiSubagentUsage) reached span names/attributes without running through security.OutputPipeline(), unlike the structurally identical execute_tool case. Added sanitizeModelSpec, which runs each spec through security.OutputPipeline() (same treatment as toolSpanTracker.safeName) before splitModelSpec/usageSpanName/usageComponentSpanAttrs ever see it; a spec that sanitizes to nothing is dropped to the unknown bucket. Also switched gen_ai.system/gen_ai.provider.name from stringAttr to boundedStringAttr. Added TestEmitPerModelUsageSpans_SanitizesSecretInSpec (internal/cli/usage_spans_test.go) covering end-to-end redaction into span name and attributes.
  2. [data-exposure] usage_spans.go:118 — usage spans had no cardinality cap (internal/cli/usage_spans.go): emitPerModelUsageSpans had no cardinality cap; PerModelUsage keys come from a sandbox-writable file (~4k records possible), so an adversarial burst could emit thousands of children and evict the agent span from the OTLP queue, unlike the capped execute_tool stream. Added maxUsageSpansPerIteration (derived from tool_spans.go's maxToolSpansPerIteration) to cap usage children per iteration, and recordUsageSpanOverflow to mark the parent agent span with fullsend.usage.dropped when specs are refused, mirroring recordToolSpanOverflow/fullsend.tool_spans.dropped. Documented the new attribute in docs/guides/infrastructure/distributed-tracing.md. Added TestEmitPerModelUsageSpans_CapsCardinality and TestEmitPerModelUsageSpans_NoOverflowUnderCap.
  3. [breaking-change-authorization] run.go:3576 — breaking gen_ai.usage. removal not marked with ! / trailer* ((commit message)): Mixed-model agent spans now omit gen_ai.usage.* (previously the rollup), a breaking output-format change per COMMITS.md, but neither the PR title nor commit messages carried the required ! suffix / BREAKING CHANGE trailer. This fix commit's title carries the ! suffix (fix(bug(telemetry): mixed-model Pi subagent usage is attributed to the parent model #7550)!: ...) and its body includes a BREAKING CHANGE trailer describing the removed attributes and the fullsend.usage.component migration path. The PR title itself (currently fix(#7550): export per-model usage as mixed-model Pi span children) still needs the ! suffix before merge — editing the PR title is a post-script/human responsibility outside this agent's mutation scope, so it is flagged here for the summary comment.
  4. [logic-error] distributed-tracing.md:298 — false claim that rounded component costs sum to the rollup (docs/guides/infrastructure/distributed-tracing.md): The doc claimed component fullsend.cost_usd values 'already sum to' the rounded agent-span rollup; this is false when independently-rounded per-component costs cross cent boundaries differently than the raw sum (e.g. three $0.006 components sum-round to $0.03 vs. the raw $0.018 rounding to $0.02). Rewrote the query-contract prose (lines ~296-298) to state that component costs are rounded independently and can differ from the agent-span rollup by up to a cent per component, using the review's own $0.006 x 3 example, and cross-referenced the existing agent-vs-root rounding caveat. Added TestUsageComponentCosts_RoundedSumCanDifferFromRollup demonstrating the real (non-reconciling) behavior with sub-cent per-component costs, since the existing mixed-model test's fixture costs are already at cent precision and can't show this.
  5. [api-compatibility] distributed-tracing.md:227 — missing breaking-change callout for gen_ai.usage. removal* (docs/guides/infrastructure/distributed-tracing.md): The doc's existing 'Breaking change' blockquote convention (used for the gen_ai.system Pi rename) was not applied to this PR's equally consumer-visible removal of gen_ai.usage.* from mixed-model agent spans. Added a matching '> Breaking change — mixed-model Pi iterations:' blockquote immediately after the existing Pi-runtime breaking-change callout, stating the removed attributes and the fullsend.usage.component migration path.
  6. [naming-conventions] usage_spans.go:96 — reused tool_spans.go's tool-name-bound constant (internal/cli/usage_spans.go): usageSpanName reused tool_spans.go's maxToolSpanNameBytes (documented as bounding the tool-name portion of execute_tool spans), coupling an unrelated span kind to tool-span limits. Added a dedicated maxUsageSpanNameBytes = 128 constant in usage_spans.go and switched usageSpanName to use it; updated TestUsageSpanName_BoundsModel to reference the new constant.
  7. [adr-process] ADR 0050:181 — Accepted-ADR annotation not called out in PR description ((PR description)): This PR annotates an Accepted ADR, which docs/contributing/adrs.md expects to be explicitly called out in the PR description; the current description only implies it via general doc-update bullets. The ADR annotation itself was already reviewed as policy-compliant (style/no rewrite of Context/Decision/Consequences) — no ADR content change was needed. Adding the explicit callout line to the PR description is a PR-metadata edit outside this agent's mutation scope (post-script/human responsibility), so it is flagged here for the summary comment rather than left silently unaddressed.

Tests: passed

Decision points
  • Reused tool_spans.go's maxToolSpansPerIteration value for the new usage-span cardinality cap, via a dedicated maxUsageSpansPerIteration constant rather than a hardcoded literal or a fully independent bound. (alternatives: Invent a new independent numeric bound for usage spans, Reuse maxToolSpansPerIteration directly at call sites, Derive a dedicated constant from maxToolSpansPerIteration (chosen); rationale: The cap exists to protect the shared OTLP batch-processor queue (2048, drop-newest) from eviction, the same physical resource tool spans protect — reusing the reasoned-about value avoids inventing a new number to justify, while a dedicated constant keeps the two bounds free to diverge later, unlike the maxToolSpanNameBytes reuse the naming-conventions finding flagged.)
  • Fixed the rounding-claim discrepancy by correcting the documentation and adding a demonstrating test, rather than changing cost allocation to make rounded component sums reconcile exactly with the rollup. (alternatives: Allocate the rounding remainder onto one component so sums reconcile exactly, Correct the doc prose and add a test proving the real (non-reconciling) behavior (chosen); rationale: The remediation offered either option; changing cost allocation would alter already-shipped billing semantics beyond what the review asked for, while documenting the true behavior (matching the existing agent-vs-root rounding caveat already in the same doc) is the minimal correct fix.)

Next steps:

  • /fs-review — request a re-review of the changes
  • /fs-fix <your instruction> — run another fix pass with specific guidance
  • Push commits directly — review re-runs automatically on push
    Updated by fullsend fix agent

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 5:36 PM UTC · Completed 5:54 PM UTC

Commit: dbc8a10 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $6.87

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-coder

fullsend-ai-coder Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 5:57 PM UTC · Completed 6:17 PM UTC

Commit: dbc8a10 · View workflow run →

Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $2.40

Address review feedback on PR #7553:

- docs/guides/dev/tracing.md: the "usage spans" section documented the
  happy path but omitted the security/robustness mechanisms the prior
  fix-iteration added (security.OutputPipeline() sanitization of each
  PerModelUsage key, the maxUsageSpanNameBytes name bound, the
  maxUsageSpansPerIteration cap, and the fullsend.usage.dropped overflow
  attribute), unlike the neighboring execute_tool spans section, which
  documents its equivalent protections in full.
- docs/guides/infrastructure/distributed-tracing.md: the
  fullsend.usage.dropped attribute description and the "Per-model usage
  components" section referenced "the usage-span cap" without ever
  stating the actual limit (1,024), unlike fullsend.tool_spans.dropped's
  description, which cites the 1,024-span cap explicitly.

Not addressed here (recorded in the fix agent's structured output):

- breaking-change-authorization (PR title missing the "!" suffix) and
  adr-process (PR description not mentioning the ADR 0050 annotation)
  are PR-metadata edits outside this agent's mutation scope, per the
  prior fix-iteration's same determination.
- test-weakened (usage_spans_test.go:453): verified against the actual
  code and disagreed - the test's filler value already exceeds both the
  redactor's 8-char structural-match threshold and its 10-char
  prefix+ellipsis masking threshold, and the test's assert.Equal only
  passes because real redaction fires (it would fail against the much
  longer unredacted string otherwise).

Addresses #7553
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 2 (bot-triggered)

Addressed the two actionable documentation findings by extending the usage-spans doc sections with the security/cap details they were missing. Verified and disagreed with the test-weakened finding (the test's filler already clears the redactor's thresholds and exercises real redaction). Recorded the PR-title and PR-description findings as valid but outside this agent's forge-mutation scope, consistent with the prior fix-iteration's determination. All non-Fullsend project CI checks on the PR are currently passing.

Fixed (2):

  1. documentation-completeness: usage spans section omits the security/robustness mechanisms the prior fix-iteration added (docs/guides/dev/tracing.md): Extended the "usage spans" section to document security.OutputPipeline() sanitization of each PerModelUsage key (with the unknown-bucket fallback on a fully-redacted spec), the maxUsageSpanNameBytes span-name bound, the maxUsageSpansPerIteration cap, and the fullsend.usage.dropped overflow attribute, mirroring how the neighboring execute_tool spans section documents its equivalent protections.
  2. documentation-accuracy: fullsend.usage.dropped description and the per-model usage components section never state the actual 1,024 cap (docs/guides/infrastructure/distributed-tracing.md): Stated the 1,024-spec limit explicitly in the fullsend.usage.dropped attribute-table description and added a sentence in "Per-model usage components" noting the 1,024-child cap and the overflow attribute, matching fullsend.tool_spans.dropped's existing explicit citation of its 1,024-span cap.

Disagreed (4):

  1. breaking-change-authorization: the PR title still lacks the "!" suffix required by COMMITS.md: The finding is valid, but editing the PR title is a forge-level PR-metadata mutation, not a repository file change, and falls outside this agent's authorized mutation scope (push/PR/comment/label edits are reserved for the deterministic post-script). The current HEAD commit already carries the "!" suffix and a BREAKING CHANGE trailer as COMMITS.md requires; the prior fix-iteration reached and recorded the same determination. A human or the post-script needs to update the PR title to fix(#7550)!: export per-model usage as mixed-model Pi span children before merge.
  2. adr-process: the PR description does not call out the ADR 0050 annotation as docs/contributing/adrs.md expects: Valid observation, but the remediation is to edit the PR description, which is forge PR metadata outside this agent's mutation scope (the same category as the PR-title finding above). Flagging for a human/post-script to add a line to the PR description noting the ADR 0050 annotation before merge.
  3. test-weakened: TestEmitPerModelUsageSpans_SanitizesSecretInSpec's filler value is too short to trigger the redactor's env_assignment pattern: Verified directly against the committed file rather than trusting the finding: the secretValue filler is well over both the redactor's 8-character structural-match minimum and its 10-character prefix+ellipsis masking threshold, not the 7 characters the finding describes. Running the test in isolation confirms real redaction fires — the assert.Equal on the redacted attribute only passes because the value was actually shortened by masking; it would fail if compared against the much longer unredacted string. No change made; the test already exercises the redaction path it claims to cover.
  4. sub-agent-failure: the review's challenger-adjudication pass returned a malformed response and fell back to the pre-challenger finding set: This is meta-commentary about the review pipeline's own process, not an actionable code or documentation finding, so no fix applies.

Tests: passed

CI inspections (12):

  1. functional-tests (passing, success)
  2. gate (Functional Tests) (passing, success)
  3. behaviour (passing, success)
  4. e2e (passing, success)
  5. gate (E2E Tests) (passing, success)
  6. test-sandbox-darwin (passing, success)
  7. build (Build Site) (passing, success)
  8. commit-lint (passing, success)
  9. web (passing, success)
  10. test (passing, success)
  11. codecov/patch (passing, success)
  12. DCO (passing, success)
Decision points
  • Left the PR title and PR description unedited despite both findings' remediations asking for forge-metadata changes (alternatives: Call the forge CLI directly to edit the PR title/description, Leave PR-metadata edits to the post-script/human and document the need in structured output; rationale: PR title/description are forge PR mutations outside this agent's authorized scope (push/PR/comment/label edits are reserved for the deterministic post-script); the prior fix-iteration's commit message recorded the identical determination for the same PR.)
  • Disagreed with the test-weakened finding after direct verification instead of lengthening the test's filler further (alternatives: Trust the finding and lengthen the filler regardless, Verify the actual committed byte content and the test's real pass/fail behavior before acting; rationale: Zero-trust principle: the finding's stated character count did not match the committed file, and an isolated test run showed genuine redaction already occurring, so no code change was warranted.)

Next steps:

  • /fs-review — request a re-review of the changes
  • /fs-fix <your instruction> — run another fix pass with specific guidance
  • Push commits directly — review re-runs automatically on push
    Updated by fullsend fix agent

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ❌ Failure (post-script /home/runner/work/fullsend/fullsend/.fullsend/.fullsend-cache/resources/sha256/f93bf432ead6bfd81509e281c866ca9d50995c3561249ae53ed09c58df8e249d/scripts/post-review.sh failed: exit status 5) · Started 6:19 PM UTC · Completed 6:40 PM UTC

Commit: 64de88e · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $4.44

@ascerra

ascerra commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

/fs-review

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ❌ Failure (post-script /home/runner/work/fullsend/fullsend/.fullsend/.fullsend-cache/resources/sha256/df4695a536d58e6f4754a7a25396e74d5ffce2c2741bd06854f47dfa37378d7f/scripts/post-review.sh failed: exit status 5) · Started 5:26 PM UTC · Completed 5:42 PM UTC

Commit: 64de88e · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $3.91

@ascerra ascerra changed the title fix(#7550): export per-model usage as mixed-model Pi span children fix(#7550)!: export per-model usage as trace components Nov 5, 2026
Merge main and resolve the four tracing documentation conflicts while
preserving current content-capture, Codex usage, and cost guidance.
Use the iteration serving endpoint for Claude and Codex model-id usage
components, including gateway routes, and cover both with regression tests.
Replace a synthetic API-key fixture that triggered the staged secret scan.

BREAKING CHANGE: Mixed-model iterations emit input, output, and cache token
attributes on fullsend.usage.component children instead of the agent rollup.
Consumers must sum the components and exclude the parent rollup.

Signed-off-by: Adam Scerra <ascerra@redhat.com>
Signed-off-by: Adam Scerra <ascerra@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ❌ Failure (post-script /home/runner/work/fullsend/fullsend/.fullsend/.fullsend-cache/resources/sha256/eb05ab46875ecfb67f461de43e976d2cadf0913251419c784718aba608589835/scripts/post-review.sh failed: exit status 5) · Started 11:55 AM UTC · Completed 12:08 PM UTC

Commit: e5df188 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $2.85

@ascerra

ascerra commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

/fs-review

@fullsend-ai-review

fullsend-ai-review Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 12:37 PM UTC · Completed 12:56 PM UTC

Commit: e5df188 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $3.10

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added go Pull requests that update go code component/docs User-facing documentation labels Nov 5, 2026
@fullsend-ai-coder

fullsend-ai-coder Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 12:58 PM UTC · Completed 1:09 PM UTC

Commit: e5df188 · View workflow run →

Runtime: claude · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $0.55

Route PerModelUsage keys through redactText (runner env literal pass,
whole-string masking, URL userinfo/query/fragment masking) instead of a
bare output-pipeline scan, matching toolSpanTracker.safeName. Add a
regression test for URL and runner-env redaction, rewrap long doc
paragraphs, and update the dev tracing guide.

Notes: pre-commit could not fetch hook repos (certificate failure); gofmt,
go vet and trailing-whitespace checks were run directly. Unrelated
TestStartFetchService_* tests fail in the sandbox (listener permission
denied), with and without this change.

Addresses #7553

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 3 (bot-triggered)

Fixed the high-severity secret-exposure finding by routing usage-span model specs through redactText and added a regression test. Rewrapped the long doc paragraphs. Disagreed with the test-inadequate finding, because the file already uses a 26-character value and the test passes. Declined the two optional low-severity cross-repo notes. Pre-commit could not fetch its hook repositories (certificate failure), so gofmt, go vet and a whitespace check were run directly. Unrelated TestStartFetchService_* tests fail in the sandbox because listeners are denied; they fail the same way without this change.

Fixed (2):

  1. Model spec sanitization skipped the shared redactText helper (runner-env literals, whole-string masking, URL masking) (internal/cli/usage_spans.go): sanitizeModelSpec now calls redactText with the runner environment on the whole spec before splitting or truncating it. emitPerModelUsageSpans takes runnerEnv, and finalizeAgentSpan passes the tool-span tracker's runner env. Added a regression test covering URL userinfo, query, fragment and a runner-env literal across span names and string attributes.
  2. Long unwrapped paragraphs in the per-model usage section of the tracing guide (docs/guides/infrastructure/distributed-tracing.md): Rewrapped the three paragraphs at about 80 columns without changing their content. Also updated docs/guides/dev/tracing.md so its description of sanitizeModelSpec matches the redactText behavior.

Disagreed (3):

  1. TestEmitPerModelUsageSpans_SanitizesSecretInSpec uses a 7-character value and has contradictory assertions: The file already uses a 26-character filler value, which the env_assignment pattern matches. The expected masked model keeps a 4-character prefix plus an ellipsis, which does not contain the full original value, so the assertions are consistent. The test passes against the real pipeline. The reviewer probably saw a display-masked copy of the literal.
  2. Mixed-model agent spans no longer carry gen_ai.usage. (add release-note migration guidance)*: The reviewer marked this optional and non-actionable. The break is already declared with the ! in the PR title, the BREAKING CHANGE trailer and the guide callout. Release notes are authored outside this PR's code.
  3. EM-001 usageOK could be satisfied by any usage component span; tie it to a rollup parent and widen the Pi-only wording: The reviewer marked this optional and non-actionable. Tightening the fitness check changes eval semantics beyond the scope of this review round.

Tests: passed

CI inspections (6):

  1. CI / test (passing, success)
  2. CI / commit-lint (passing, success)
  3. PR Title Lint / pr-title-lint (passing, success)
  4. codecov/patch (passing, success)
  5. E2E Tests / playback (transient-infra, cancelled): The run was cancelled rather than failed. Rerun the E2E jobs after pushing.
  6. E2E Tests / behaviour (transient-infra, cancelled): The run was cancelled rather than failed. Rerun the E2E jobs after pushing.

Next steps:

  • /fs-review — request a re-review of the changes
  • /fs-fix <your instruction> — run another fix pass with specific guidance
  • Push commits directly — review re-runs automatically on push
    Updated by fullsend fix agent

@fullsend-ai-review

fullsend-ai-review Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 1:11 PM UTC · Completed 1:28 PM UTC

Commit: fbf6fd2 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $2.57

@fullsend-ai-review
fullsend-ai-review Bot dismissed stale reviews from themself November 5, 2026 13:27

Superseded by updated review

This branch was successfully deployed

1 active deployment
site-preview — fbf6fd28 Deployed Nov 5, 2026 by github-actions[bot]
dev — fbf6fd28 Deployed Nov 5, 2026 by fullsend-ai-coder[bot] via behaviour #18963
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/docs User-facing documentation go Pull requests that update go code ready-for-merge All reviewers approved — ready to merge ready-for-review Triggers review agent dispatch risk/moderate PR risk: moderate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(telemetry): mixed-model Pi subagent usage is attributed to the parent model

1 participant