fix(opencode): add model attribution to run --format json events - #40545
fix(opencode): add model attribution to run --format json events#40545macurandb wants to merge 6 commits into
Conversation
|
The following comment was made by an LLM, it may be inaccurate: Based on my search results, no duplicate PRs found for this PR addressing model attribution in JSON step events. The search did surface two related but distinct PRs that work with JSON output formatting:
These are complementary improvements to JSON output, not duplicates of the current PR (#40545). |
|
Thanks for updating your PR! It now meets our contributing guidelines. 👍 |
9b00e27 to
ddbb4fe
Compare
The step_start/step_finish events emitted by `run --format json` carried no model information, so a headless consumer could not attribute token usage or cost to a model. The data was already available in the same loop: the CLI reads `info.modelID` off `message.updated` to print the model header in the default format, but never passed it to the JSON path. Track providerID/modelID per messageID and include them in the step events.
ddbb4fe to
7a0daff
Compare
|
I hit #40544 too and landed on the same approach before I saw this, so I closed mine (#40581) in favour of it. Two small deltas from my version, take or leave:
Your live two-model check is better evidence than anything I had, so this is the one to merge. |
step_start/step_finish carried the model but text, reasoning and tool_use did not, so a consumer reading only those had to correlate back to a step event. Same lookup, hoisted once per part.
|
Thanks — both deltas are in, credited in the description. Stamping the other events was the right call: I'd optimised for the smallest diff and left the schema half-and-half, where
What does make it useful is the parent's own agent varying — |
2b898c2 to
f48dc9e
Compare
|
Heads up: I pulled the It's a one-line addition sitting on the same message ( This PR is now strictly the model. If you want the agent too, the commit is ready on top of this branch: macurandb/opencode@run-json-model...run-json-agent — say the word and I'll push it here. If you'd rather not, nothing to do. Verified live either way: with |
|
Fair correction on the subagent point - I checked, and you're right: And splitting it back out is the right call. It's the maintainers' to take, not mine, and keeping this PR strictly the missing attribution makes it a smaller thing to say yes to. Happy to leave |
reasoning was the one stamped event type with no attribution assertion: reverting its emit left the suite green. The attribution run now covers all five part-bearing types.
|
Thanks for checking — and no harm done, the One more round of review turned up a gap in my tests: |
|
Good catch. That revert-one-line check is the only way to know a shape test is actually asserting anything, and nice that you ran it per field rather than on the whole stamp. Only thing left from my side is the note already in the description: whenever #38504's |
Every existing test passes --model, so an implementation reading the model off the argv passed the whole suite. This one runs without --model and only passes if the value really comes from the assistant message.
Issue for this PR
Closes #40544
Type of change
What does this PR do?
run --format jsonevents carried no model, so a headless consumer couldn't attribute tokens or cost. With two models in one session, the events are byte-identical apart from ids.The value was already in the same loop: the default format reads
info.modelIDoffmessage.updatedto print the model header, but it never reached the JSON path. So this tracks providerID/modelID per messageID as those events arrive, and stamps them onto every part-bearing event (step_start,step_finish,text,reasoning,tool_use). It works becausemessage.updatedfor an assistant message is published before that message's parts (session.tsupdateMessageruns beforeprocessor.createinprompt.ts), so the map is populated by the time a part arrives.The
message.updatedguard is split: the map is populated for every assistant message, while the header print stays behindargs.format !== "json"— default-format output is unchanged.Additive: no existing field changes,
partpayload untouched.How did you verify your code works?
Updated the existing
--format jsonshape test — it asserts the exact event shape on purpose ("a future event-emit change has to update this expectation"). Reverting onlyrun.tsmakes it fail on exactly the two missing fields.Added a test that runs a non-default model (
test/test-model-alt, new in the test provider) through a tool call, so the run spansstep_start/text/tool_use/step_finish, and asserts every one of them reports that model — the attribution can't pass by returning a fixed value.Added a test that runs without
--model: every other test passes the flag, so an implementation reading the model off the argv passed the whole suite. Confirmed it — that implementation now fails this test and only this one.bun test test/cli/run/run-process.test.ts→ 15 pass / 0 fail. The other suites using the shared test provider are unaffected:test/provider/header-timeout,test/server/httpapi-session,test/server/httpapi-sdk→ 45 pass / 0 fail,test/cli/acp/→ 15 pass / 0 fail.Live, two models in one session:
step_finishreportsopencode/ling-3.0-flash-freeon the first turn andopencode/mimo-v2.5-freeon the second. Before, both were indistinguishable.oxlinton the touched file reports the same 17 pre-existing warnings as it does without the change.Typecheck:
tsgo --noEmitonpackages/opencoderuns clean uncached (turbo reports 30/30, but that run is cached — the uncached one is the meaningful check).To reproduce the live check:
bun dev run --format json -m <model-a> "hi", thenbun dev run --format json --session <sessionID> -m <model-b> "hi".The multi-turn case is verified live but not in the suite: the CLI test harness can't continue a session across processes (
--session <id>from a second spawn errorsSession not found, and--continuestarts a new session), so the automated test covers a single turn on a non-default model instead.Note on the unknown case: the map is only filled from messages seen on this stream, so a step whose message was created before the subscription (attaching to a turn already in flight) emits without the fields rather than guessing — absent means unknown, never wrong. Happy to emit explicit
nulls instead if you'd rather the event shape stay fixed.Screenshots / recordings
Not a UI change.
Checklist
I put the fields next to
sessionIDrather than insidepart, to leave the Part payload as-is.The wider stamping came from @rawsun007, who hit the same issue independently and closed #40581 in favour of this one.
Note for whichever lands second: #38504 adds a
part_deltaevent using the same map-in-the-loop shape, and it would want these fields too.