Skip to content

fix(opencode): add model attribution to run --format json events - #40545

Open
macurandb wants to merge 6 commits into
anomalyco:devfrom
macurandb:run-json-model
Open

fix(opencode): add model attribution to run --format json events#40545
macurandb wants to merge 6 commits into
anomalyco:devfrom
macurandb:run-json-model

Conversation

@macurandb

@macurandb macurandb commented Aug 5, 2026

Copy link
Copy Markdown

Issue for this PR

Closes #40544

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

run --format json events carried no model, so a headless consumer couldn't attribute tokens or cost. With two models in one session, the events are byte-identical apart from ids.

The value was already in the same loop: the default format reads info.modelID off message.updated to print the model header, but it never reached the JSON path. So this tracks providerID/modelID per messageID as those events arrive, and stamps them onto every part-bearing event (step_start, step_finish, text, reasoning, tool_use). It works because message.updated for an assistant message is published before that message's parts (session.ts updateMessage runs before processor.create in prompt.ts), so the map is populated by the time a part arrives.

The message.updated guard is split: the map is populated for every assistant message, while the header print stays behind args.format !== "json" — default-format output is unchanged.

Additive: no existing field changes, part payload untouched.

How did you verify your code works?

  • Updated the existing --format json shape test — it asserts the exact event shape on purpose ("a future event-emit change has to update this expectation"). Reverting only run.ts makes it fail on exactly the two missing fields.

  • Added a test that runs a non-default model (test/test-model-alt, new in the test provider) through a tool call, so the run spans step_start/text/tool_use/step_finish, and asserts every one of them reports that model — the attribution can't pass by returning a fixed value.

  • Added a test that runs without --model: every other test passes the flag, so an implementation reading the model off the argv passed the whole suite. Confirmed it — that implementation now fails this test and only this one.

  • bun test test/cli/run/run-process.test.ts → 15 pass / 0 fail. The other suites using the shared test provider are unaffected: test/provider/header-timeout, test/server/httpapi-session, test/server/httpapi-sdk → 45 pass / 0 fail, test/cli/acp/ → 15 pass / 0 fail.

  • Live, two models in one session: step_finish reports opencode/ling-3.0-flash-free on the first turn and opencode/mimo-v2.5-free on the second. Before, both were indistinguishable.

  • oxlint on the touched file reports the same 17 pre-existing warnings as it does without the change.

  • Typecheck: tsgo --noEmit on packages/opencode runs clean uncached (turbo reports 30/30, but that run is cached — the uncached one is the meaningful check).

To reproduce the live check: bun dev run --format json -m <model-a> "hi", then bun dev run --format json --session <sessionID> -m <model-b> "hi".

The multi-turn case is verified live but not in the suite: the CLI test harness can't continue a session across processes (--session <id> from a second spawn errors Session not found, and --continue starts a new session), so the automated test covers a single turn on a non-default model instead.

Note on the unknown case: the map is only filled from messages seen on this stream, so a step whose message was created before the subscription (attaching to a turn already in flight) emits without the fields rather than guessing — absent means unknown, never wrong. Happy to emit explicit nulls instead if you'd rather the event shape stay fixed.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

I put the fields next to sessionID rather than inside part, to leave the Part payload as-is.

The wider stamping came from @rawsun007, who hit the same issue independently and closed #40581 in favour of this one.

Note for whichever lands second: #38504 adds a part_delta event using the same map-in-the-loop shape, and it would want these fields too.

@github-actions github-actions Bot added the needs:compliance This means the issue will auto-close after 2 hours. label Aug 5, 2026
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

The following comment was made by an LLM, it may be inaccurate:

Based on my search results, no duplicate PRs found for this PR addressing model attribution in JSON step events.

The search did surface two related but distinct PRs that work with JSON output formatting:

These are complementary improvements to JSON output, not duplicates of the current PR (#40545).

@github-actions github-actions Bot removed the needs:compliance This means the issue will auto-close after 2 hours. label Aug 5, 2026
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Thanks for updating your PR! It now meets our contributing guidelines. 👍

The step_start/step_finish events emitted by `run --format json` carried no
model information, so a headless consumer could not attribute token usage or
cost to a model. The data was already available in the same loop: the CLI reads
`info.modelID` off `message.updated` to print the model header in the default
format, but never passed it to the JSON path.

Track providerID/modelID per messageID and include them in the step events.
@rawsun007

Copy link
Copy Markdown

I hit #40544 too and landed on the same approach before I saw this, so I closed mine (#40581) in favour of it. Two small deltas from my version, take or leave:

  • I stamped the fields onto text, reasoning and tool_use as well, not just step_start / step_finish. Same lookup, and it means a consumer that only reads text or tool events does not have to correlate back to a step event to know the model.
  • I also carried agent alongside providerID / modelID. A subagent turn and a parent turn on the same model are otherwise indistinguishable, and the value is right there on the message.

Your live two-model check is better evidence than anything I had, so this is the one to merge.

step_start/step_finish carried the model but text, reasoning and tool_use
did not, so a consumer reading only those had to correlate back to a step
event. Same lookup, hoisted once per part.
@macurandb

Copy link
Copy Markdown
Author

Thanks — both deltas are in, credited in the description.

Stamping the other events was the right call: I'd optimised for the smallest diff and left the schema half-and-half, where step_finish said the model and text didn't. It's now on step_start, step_finish, text, reasoning and tool_use, with the lookup hoisted once per part. The attribution test runs a non-default model through a tool call so it covers all four event types in one run.

agent is in too, though not for the reason you gave — worth flagging in case it bit you elsewhere. Subagent turns can't show up on this stream: the task tool runs the subagent in a child session (sessionID: nextSession.id, tool/task.ts:204) and the loop drops anything that isn't the current session (if (part.sessionID !== sessionID) continue). So a parent turn and a subagent turn are never both in this output.

What does make it useful is the parent's own agent varying — --agent plan vs build — and the default format already printing agent and model together. Verified live: with --agent plan every event reports agent: plan.

@macurandb

Copy link
Copy Markdown
Author

Heads up: I pulled the agent field back out of this PR.

It's a one-line addition sitting on the same message (info.agent), and the default format already prints agent and model together, so a consumer aggregating cost per agent would want it. But it's additive functionality rather than the missing attribution #40544 is about, and CONTRIBUTING routes features through a design conversation — so I'd rather ask than assume.

This PR is now strictly the model. If you want the agent too, the commit is ready on top of this branch: macurandb/opencode@run-json-model...run-json-agent — say the word and I'll push it here. If you'd rather not, nothing to do.

Verified live either way: with --agent plan, every event reports agent: plan rather than the default build.

@rawsun007

Copy link
Copy Markdown

Fair correction on the subagent point - I checked, and you're right: task.ts runs it under nextSession.id and the loop drops anything that isn't the current session, so those parts never reach this stream. My justification for agent was wrong; the parent's own --agent plan vs build is the real one.

And splitting it back out is the right call. It's the maintainers' to take, not mine, and keeping this PR strictly the missing attribution makes it a smaller thing to say yes to. Happy to leave agent on your branch until someone asks for it.

reasoning was the one stamped event type with no attribution assertion:
reverting its emit left the suite green. The attribution run now covers
all five part-bearing types.
@macurandb

Copy link
Copy Markdown
Author

Thanks for checking — and no harm done, the agent justification being wrong is exactly the kind of thing that's easier to catch from outside.

One more round of review turned up a gap in my tests: reasoning was the only stamped event type with no attribution assertion, so reverting its emit left the whole suite green. The attribution test now runs with --thinking and a reason() in the flow, covering all five part-bearing types — verified it fails on exactly that field when the line is reverted.

@rawsun007

Copy link
Copy Markdown

Good catch. That revert-one-line check is the only way to know a shape test is actually asserting anything, and nice that you ran it per field rather than on the whole stamp.

Only thing left from my side is the note already in the description: whenever #38504's part_delta lands, it wants the same stamp, and now the same per-field revert check too.

Every existing test passes --model, so an implementation reading the model
off the argv passed the whole suite. This one runs without --model and only
passes if the value really comes from the assistant message.
@macurandb macurandb changed the title fix(opencode): add model attribution to run --format json step events fix(opencode): add model attribution to run --format json events Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

run --format json events don't say which model produced them

2 participants