Repository navigation
fix(#9386)!: emit gen_ai.usage.reasoning.output_tokens - #9387
fullsend-ai-coder[bot] wants to merge 2 commits into
Conversation
The agent span recorded reasoning tokens only as gen_ai.usage.reasoning_tokens, which is not a GenAI semantic convention key. Since semconv v1.41.0 the registered attribute is gen_ai.usage.reasoning.output_tokens, so GenAI-aware backends did not see fullsend's reasoning tokens. agentSpanEndAttrs now emits gen_ai.usage.reasoning.output_tokens and keeps the legacy gen_ai.usage.reasoning_tokens alongside it with the same value for one release, mirroring gen_ai.system / gen_ai.provider.name. Both are still omitted when reasoning is zero. Tests assert both keys when non-zero and neither key when zero. Docs: the token table, jq example (new key, legacy fallback) and the docs/cli/run.md token_usage note name the new key. The message schema links now point at open-telemetry/semantic-conventions-genai pinned to 371eb36a, the last commit before the unreleased structured-content change (#634). The per-runtime notes record that Codex's gen_ai.usage.output_tokens excludes reasoning, departing from the convention's SHOULD; Claude and pinned pi include it. Note: pre-commit could not fetch remote hook repos (TLS failure in the sandbox). Hooks were run directly instead: pre-commit-hooks v6.0.0 checks, gofmt, go vet, lint-broken-symlinks, lint-docs-links, lychee and gitleaks all passed. semgrep-scan was skipped because it could not download its pinned ruleset. Six internal/cli tests (TestStartFetchService_*, TestPostScriptEnv_NoSchema*) fail identically on main in this sandbox and are unrelated. Closes #9386 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
🤖 Finished Review · ✅ Success · Started 7:09 PM UTC · Completed 7:16 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $1.48 |
Site previewPreview: https://8770bb1f-site.fullsend-ai.workers.dev Commit: |
|
Risk Assessment: moderate (2/5) DetailsModerate risk maintained at score 2: while the second commit introduces an unflagged breaking change to telemetry output token semantics, zero protected paths and aligned issue scope are balanced against high churn and multi-author contention in internal/cli/run.go. Previous runRisk Assessment: moderate (2/5) DetailsSmall additive telemetry update with clear issue alignment, elevated to moderate risk by high churn and multi-author contention in internal/cli/run.go. Previous run (2)Risk Assessment: moderate (2/5) DetailsSmall additive telemetry update with clear issue alignment, elevated to moderate risk by high churn and multi-author contention in internal/cli/run.go. |
ReviewFindingsLow
Previous run<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJhcHByb3ZlIiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJzY29wZS1leGNlZWRlZCIsImZpbGUiOiJpbnRlcm5hbC9jbGkvcnVuLmdvIiwiaWQiOiJmX2FhZDFiZjU5N2FmMzRkODEiLCJsaW5lIjo0ODM3LCJhY3Rpb25hYmxlIjpmYWxzZSwiZGltZW5zaW9uIjoiaW50ZW50LWNvaGVyZW5jZSJ9LHsic2V2ZXJpdHkiOiJsb3ciLCJjYXRlZ29yeSI6Im1pc3NpbmctZGVwcmVjYXRpb24iLCJmaWxlIjoiaW50ZXJuYWwvY2xpL3J1bi5nbyIsImlkIjoiZl80MzhmM2FjYzE5MDRkODdiIiwibGluZSI6NDg2MywiYWN0aW9uYWJsZSI6ZmFsc2UsImRpbWVuc2lvbiI6ImNyb3NzLXJlcG8tY29udHJhY3RzIn1dfQ== --> ReviewFindingsLow
Previous run (2)<!-- fullsend:review-diff-hash:0dc1f070e03fe79c7efcd73088943e957ea57b8a562b0b7b0e7b157b257e5376 --> Looks good to me Previous run (3)<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJhcHByb3ZlIiwiZmluZGluZ3MiOltdfQ== --> Looks good to me Previous run (4)<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJhcHByb3ZlIiwiZmluZGluZ3MiOltdfQ== --> Looks good to me Labels: PR updates user-facing tracing documentation alongside a telemetry attribute change. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
ascerra
left a comment
There was a problem hiding this comment.
The Go change is correct, and the upstream facts the PR cites check out: the key name and its v1.41.0 origin, the pinned semantic-conventions-genai commit and both schema paths, and the jq example on old and new lines. Three things need to change before this merges. The detail is in the inline comments.
Blocking
- The new key is emitted on Codex and OpenCode spans, where
gen_ai.usage.output_tokensexcludes reasoning. A reader that applies the convention then subtracts reasoning a second time. - Issue item 4 is not done for OpenCode. The pinned OpenCode 1.19.1 excludes reasoning from output, the same as Codex, and the docs do not say so.
- The "one release, then removed" promise names no release, and nothing tracks it once this PR closes #9386.
The PR body also says Closes #9386 while two of the issue's items are still open: item 4 (above) and the acceptance item that asks for run-telemetry.jsonl from a real run. The closing line needs a maintainer's edit; the fix agent cannot change it.
Not blocking, same files
- Span-level test. The new assertions in
internal/cli/telemetry_run_test.gocheck the slice thatagentSpanEndAttrsreturns.TestFinalizeAgentSpanrecords a real span but never setsReasoningTokens. A case there gives the span-level test the issue asks for. - Attribute table.
docs/guides/infrastructure/distributed-tracing.md:345still lists only the input, output and cache keys foragentspans. - Deprecation wording.
CONTRIBUTING.md("Deprecation notices in documentation") asks for(deprecated)in table cells and a> **Deprecated:**blockquote. Lines 578 and 586-589 of the tracing guide say "legacy". - Schema link. At the pinned commit,
finish_reasonis deprecated and no longer required (schema). Lines 124-125 of the tracing guide still present it as part of following the schema; one clause saying it is deprecated upstream keeps the sentence accurate. Comments atinternal/cli/content_collector.go:330andinternal/cli/run.go:2881still call it schema-required. Those are outside this diff. - PR description. "Backends that read the convention (MLflow, Langfuse) never saw fullsend's reasoning tokens" does not match either backend. MLflow reads no reasoning key at all (translator). Langfuse already stored the legacy key as its own usage bucket. This text becomes the squash commit message.
Outside this PR
- MLflow is unaffected by this change. It computes total tokens as input plus output and reads no reasoning key (code). On Codex and OpenCode runs its totals leave reasoning out, and will until
gen_ai.usage.output_tokensincludes it on those runtimes. - The Codex parser subtracts reasoning on a wrong premise. Its comment says Claude Code and pi report reasoning separately from output. Both report it inside output, as the Claude and pi bullets in the tracing guide say.
RunMetrics.ReasoningTokenstherefore means two different things, and the liveTOKENS … total=line (renderer.go:94) counts reasoning twice on Claude and pi. - Input tokens depart from the convention on every runtime. Since semconv v1.40.0,
gen_ai.usage.input_tokensSHOULD include cached tokens (registry). Fullsend emits uncached input.
The Langfuse and MLflow behaviour described in this review is read from their source at the linked commits. It was not observed on a running instance.
ascerra
left a comment
There was a problem hiding this comment.
Requesting changes for the three blocking items in my earlier review. That review was filed as a comment by mistake; its content stands.
- The new key is emitted on Codex and OpenCode spans, where
gen_ai.usage.output_tokensexcludes reasoning. - The OpenCode check from issue item 4 is not done, and the docs do not say OpenCode departs the same way as Codex.
- The legacy-key removal names no release and has no tracker.
The Closes #9386 line also needs a maintainer's edit while issue item 4 and the real-run acceptance item are open.
|
/fs-review please consider this recent PR as well #9563 |
|
🤖 Finished Review · ✅ Success · Started 7:19 PM UTC · Completed 7:24 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $0.75 |
Superseded by updated review
The agent span now reports gen_ai.usage.output_tokens with reasoning tokens included on every runtime, as the GenAI semantic convention defines it. Codex and OpenCode metrics keep reasoning out of output, so the span adds it back for those two. Claude Code and pi already count reasoning inside output and their spans do not change. The previous commit on this branch emitted gen_ai.usage.reasoning.output_tokens next to an output count that left reasoning out on Codex and OpenCode. The convention defines that key as a subset of output, so a backend that subtracts it from output would have subtracted reasoning twice on those runtimes. - internal/runtime: OutputExcludesReasoning names the runtimes whose metrics keep reasoning out of output. A test lists every name in config.ValidRuntimes, so a new one fails until it is classified. - internal/cli: agentSpanEndAttrs adds reasoning to output for those runtimes. RunMetrics and metrics.json keep their values. - Tests: a per-runtime table, a recorded-span case in TestFinalizeAgentSpan, and a file-sink test that reads both reasoning keys and the inclusive output count back from run-telemetry.jsonl. - Tracing guide: states the span rule and the metrics.json rule separately, documents that the pinned OpenCode 1.19.1 subtracts reasoning from output, marks gen_ai.usage.reasoning_tokens as deprecated, and makes the jq example read files written before the new key the same way as new ones. It also notes that finish_reason is deprecated in the pinned message schema. - Runtime implementation guide: lists the classification as a touchpoint for new runtimes and for version bumps. - Comments in the Codex parser and its testdata README said Claude Code and pi report reasoning separately from output. They count it inside output, and the comments now say so. The acceptance item in #9386 that asks for run-telemetry.jsonl from a real run is still unchecked, and removing the deprecated key is still to do. BREAKING CHANGE: On agent spans from the Codex and OpenCode runtimes, gen_ai.usage.output_tokens now includes reasoning tokens. Consumers that added reasoning to output for those runtimes must stop doing so on spans that carry gen_ai.usage.reasoning.output_tokens. metrics.json token_usage is unchanged. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Adam Scerra <ascerra@redhat.com>
|
Pushed 0a5d7de for the three blocking items in my review. The PR description is updated to match.
The non-blocking items are in the same commit, with one exception: the code comments that call Still open:
One correction to my review: I wrote that the PR description becomes the squash commit message. PRs here land as merge commits, so the first commit's message stays in history as written. |
|
🤖 Finished Review · ✅ Success · Started 11:43 PM UTC · Completed 11:52 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $1.83 |
Superseded by updated review
Summary
The
agentspan now emits reasoning tokens asgen_ai.usage.reasoning.output_tokens, the GenAI semantic convention key since semconv v1.41.0, and itsgen_ai.usage.output_tokensnow includes reasoning tokens on every runtime, as the convention defines it. The oldgen_ai.usage.reasoning_tokenskey is deprecated. It is still emitted with the same value through the first release that has the new key, the same waygen_ai.systemandgen_ai.provider.nameare emitted together. The tracing docs are updated to match.Breaking change: on
agentspans from the Codex and OpenCode runtimes,gen_ai.usage.output_tokensnow includes reasoning tokens. Consumers that added reasoning to output for those runtimes must stop doing so on spans that carrygen_ai.usage.reasoning.output_tokens. Claude and pi spans,RunMetricsandmetrics.jsondo not change.This description was edited after the second commit. The original is in the edit history.
Why there are two commits
The first commit (code agent) added the new key and documented that Codex output leaves reasoning out. The convention defines the reasoning key as a subset of
gen_ai.usage.output_tokens, and Codex and OpenCode metrics keep reasoning out of output. A backend that follows the convention and subtracts the reasoning key from output, as Langfuse does, would have subtracted reasoning a second time on those two runtimes.The second commit makes the span follow the convention instead of documenting the departure. It is the output-side counterpart of #9563, which does the same for
gen_ai.usage.input_tokensand cached tokens. Neither PR needs the other to land first. Both edit the "Persisted token fields" section of the tracing guide, so the second one to merge has a conflict to resolve there and inagentSpanEndAttrs.For backends that read no reasoning key, such as MLflow, the effect is through the output count: Codex and OpenCode reasoning is now part of it. The backend behaviour described here is read from their source at the linked refs, not observed on a running instance.
Related Issue
Addresses the issue linked below. Merging closes it, because the first commit's message and the line at the end of this description both carry the closing keyword. The issue also covers two things this PR does not do: removing the deprecated key after it has shipped for a release, and checking
run-telemetry.jsonlfrom a real run. The issue needs reopening after the merge, or a follow-up issue, so that the removal stays tracked.Changes
internal/cli/run.go(agentSpanEndAttrs):ReasoningTokens > 0. Neither key is emitted when it is zero, as before.gen_ai.usage.output_tokensfor runtimes whose metrics keep it out of output.internal/runtime/token_semantics.go:OutputExcludesReasoningnames those runtimes (Codex and OpenCode). A test lists every registered runtime, so a new one fails until it is classified.internal/cli/telemetry_run_test.go:TestFinalizeAgentSpanasserts the keys on a recorded span.TestAgentSpan_ReasoningKeysReachFileSinkreads both keys and the inclusive output count back fromrun-telemetry.jsonl.docs/guides/infrastructure/distributed-tracing.md:metrics.jsonrule (per runtime) separately.1.19.1: it subtracts reasoning from output before fullsend reads it, like Codex. This completes item 4 of the issue.gen_ai.usage.reasoning_tokensas deprecated, in the formCONTRIBUTING.mdasks for, and warns that a backend which keeps unrecognizedgen_ai.usage.*attributes as extra counters can count reasoning twice while both keys are present.jqexample adds reasoning back for Codex and OpenCode spans written before the new key existed, so old and new files read the same way. The worked example prices four terms.output_tokenscome frommetrics.jsonand exclude reasoning on Codex.open-telemetry/semantic-conventions-genai, pinned to commit371eb36a5258cfef7d80328755d5ef65251c2145, the last commit before the upstream structured-content change that fullsend has not adopted. That repo has no release tags yet, so a commit is the only pin available. The guide now saysfinish_reasonis deprecated in that schema.docs/cli/run.md: thetoken_usagenote covers Codex and OpenCode and says how the span differs.docs/contributing/runtime-implementation.md: lists the classification as a touchpoint for new runtimes and for version bumps.internal/runtime/codex_progress.goand the Codex testdata README: comments said Claude Code and pi report reasoning separately from output. They count it inside output; the comments now say so.Code comments that cite semconv
v1.37.0(internal/telemetry/content.go,internal/cli/tool_spans.go) are left alone because the issue did not include them.Testing
Second commit, run locally:
go test ./internal/cli/passes.go test ./internal/runtime/ ./internal/evalmeasure/ ./internal/telemetry/passes, except theTestCodexAdapter_*hook tests ininternal/runtime. They fail the same way without this change, because they need an environment variable that the local shell does not set.pre-commithooks on the changed files: gofmt,go vet, lint-md-links (lychee, offline, with fragments), lint-docs-links, gitleaks and the whitespace, end-of-file, line-ending, merge-conflict and private-key checks.jqexample was run on spans with both keys, the deprecated key only, the new key only and no reasoning, for Codex, OpenCode, Claude and pi.run-telemetry.jsonlfrom a real run to show both keys. The file-sink test above is the closest check without a real agent run.First commit, as reported by the code agent:
go test ./internal/cli/ -run 'AgentSpanEndAttrs|AggregateRunMetrics|Telemetry'passes.make lint:pre-commitcould not fetch its remote hook repos in the sandbox (TLS error), so the hooks were run directly instead. pre-commit-hooks v6.0.0 checks, gofmt,go vet ./..., lint-broken-symlinks, lint-docs-links, lychee (offline, with fragments) and gitleaks all passed. semgrep-scan was skipped because it could not download its pinned ruleset.Checklist
!for breaking changes). The second commit changes the value of an existing attribute, so the title and that commit carry!.🤖 Generated with Claude Code
Closes #9386
Post-script verification (first commit)
agent/9386-reasoning-output-tokens)7338abefc1c8f618b8149053ee80a7a75e45fb99..HEAD)