test(budget): pin the usage-attribution invariant — child cost bills, child context never leaks (#418) - #465
Merged
Conversation
#418 asked whether graff ever lets a subagent's tokens inflate its parent's context-window estimate, which would fire compaction early on fleet-heavy runs. Audited the four named surfaces and the answer is no, by construction rather than by a save/restore dance: * the BILL is process-wide. Agent.recordCost writes pricing.g_cost, so a child's spend lands in the [usage] footer, /cost and the --json cost_usd without anyone forwarding it. * the METER is per-Agent. last_context_tokens / context_local_tokens are fields on the Agent that made the request; runSub builds the child its OWN Agent, and ToolCtx carries no pointer back to the parent, so a child has nothing to write to. * run_budget/phase_budget ledger model CALLS, not tokens. Neither has a token field to conflate. A completed child hands back SubRun{output, usage}. The foreground path (execSubagent) keeps output alone; the background path (agentJobPump -> agent_output) stores usage on the job and renders it through agentStatusText -- as text the model reads, never as a number the parent's meter adds. The in-place layers that DO reuse the root Agent (review.Context, compact()) already capture and pin the meter back. Nothing to fix, so the test is the deliverable: two parents seeded identically, one of them fleeted with eight workers that each end past the parent's compaction threshold. Their meters stay equal field for field, neither trips inputOverCompactThreshold, and the run tally moves by exactly the children's tokens and USD. The second test pins the one place a child's context total does cross over -- the report line -- to the bytes it actually costs, not the 480k it names. Suite: 1118 pass (was 1116). Tier 1 green. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
Found while auditing #418. behavior_upload.recordApi accumulates the CALLER's meter into one turn-wide total, root and subagent alike, so a turn that fanned out eight workers reports roughly nine windows' worth under a field named context_tokens. Correct for upload, and never read back by the harness -- but it is exactly the field a future reader reaches for when they want "how full was the context on this turn", and the name gives them no warning. Answering that from here would re-introduce the bug #418 exists to prevent, one layer over. Documented at the type (every field is a sum over the turn's agents), at the field (the authoritative per-agent occupancy is Agent.last_context_tokens / effectiveContextTokens, which is what the compaction gates read), and at the accumulator (the summing is deliberate and belongs on this side of the boundary). Comments only. Suite still 1118 pass; tier 1 green. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #418. Audit outcome: no bug. The invariant already holds, and it holds structurally rather than by a discipline someone could forget. The test is the deliverable.
The question, answered
No. The two accounts are separate by construction:
Agent.recordCostwritespricing.g_cost, a global tally. A child bills the run automatically — that is why the[usage]footer includes the fleet.last_context_tokens/context_local_tokensare fields on the Agent that made the request.runSubgives the child a fresh Agent with both at their0defaults, andToolCtxcarries no pointer back to the parent Agent — so a child has nothing to write to even if it tried.inputOverCompactThreshold→effectiveContextTokens→contextEstimatereads onlyself.messages,self.systemPrompt(),self.toolsJson()and the twoselfanchors, and both gate call sites passroot.run_budget.zig/phase_budget.zighave no token field at all — they ledger model calls, depth and concurrency. There is nothing there to conflate.All 7
runSubcall sites were checked, across both spawn paths: foregroundexecSubagentkeepsrun.outputand dropsrun.usage; backgroundagentJobPumpstoresjob.usageandagent_outputrenders it as text; the workflow/retry/judge/pipeline sites all returnrun.output. No asymmetry between them, which is where a bug of this shape would most likely hide.Worth noting the layers that do reuse the root Agent already implement exactly the capture-and-pin-back the issue describes:
review.Context.begin/restoreandcompact().The tests
parent context is identical with and without an attributed child; only the bill differs— two identically seeded roots, one of which runs eight workers each ending at 164k context, deliberately past the parent's 80kcompactAt()so a single leaked token would trip the gate. Assertslast_context_tokens,context_local_tokens,effectiveContextTokens()andinputOverCompactThreshold()all equal the control field-for-field, and thatg_costmoved by exactly 8× the children's in/cache/out tokens and USD.a child's context total reaches the parent as report bytes, never as context— a child reporting 480k has its status line appended as a tool result; the parent grows by under 200 tokens rather than 480k, and stays under threshold.One latent trap found, documented rather than changed
behavior_upload.zigsumscontext_tokensacross parent and subagent API calls, so a fleet turn'sTurnMetrics.context_tokensis a sum-of-agents, not an occupancy. It is write-only telemetry that no harness decision reads back, so it is not a bug today — but it is exactly the field someone would later mistake for a context reading, so it now says so in place.Verification
zig build test→ 1118/1118, +2 from the 1116 baseline.scripts/eval-tier1.shgreen (reachability confirms 1079 declared tests compiled in).Not traced: whether the GUI/
--jsonfrontend aggregates theagent_usageevent'scontext_tokensfor display. That is a wire event to a separate process and cannot feed the Zig meter, so it is outside this invariant, but the GUI side was not read.