Repository navigation
runtime: record image omission as a durable projection transition #4569
Description
Last piece of #4283. Everything else that issue asked for is on main; this is the one lossy edit to model-visible history that still lives outside the ledger.
Problem
When a provider rejects a mid-turn request as too long, the runtime drops historical image Tool Results from the request and retries once. The set it dropped is kept only in memory:
packages/runtime/src/ai-sdk-compaction.ts: the ContextLength branch buildsstate.omittedImageToolResultson the mid-turn state, andprojectHistoricalImageOmissionsre-applies that map to every later request in the same turn.packages/runtime/src/provider-image-overflow-recovery.ts:collectHistoricalImageToolResultsandomitHistoricalImageToolResultsdo the selection and the rewrite.
Nothing about the omission is written to the operational ledger. On the next Turn, after a restart, or on a branch, the effective-history reducer rebuilds from the durable projections, the images come back, and the request either overflows again (one more rejected round trip and one more fold) or goes through with content the model had already lost. Either way the prompt-cache prefix the previous request established is gone.
The invariant #4283 was opened for:
A successful model-visible history is append-only. Any lossy change to already-visible history must first become a durable successor in the append-only operational ledger, and no later replay, compaction, branch, or restart may restore the replaced form.
Tool Result archiving already honours it: packages/runtime/src/tool-result-archive-transition.ts builds a ModelProjectionTransition (@maka/core/model-projection-transition) whose replacement is the archived placeholder projection, records it through recordTransition, and lets the reducer apply it everywhere. Image omission is the same shape of change and should use the same record.
Change
- Each omitted image Tool Result becomes one
ModelProjectionTransitionagainst itsfunction_responseevent, with a replacement projection that carries the omission textomissionTextalready produces, recorded through the samerecordTransitionthe archive path uses. - The reducer applies those transitions like any other, so live continuation, the next Turn, restart and branch all see the omitted form.
state.omittedImageToolResultsandprojectHistoricalImageOmissionsgo away; the mid-turn re-application falls out of the reducer. - Selection stays all-at-once. fix(runtime): let the provider decide fit, anchor estimates on real usage, retire the local verdict #4486 bills every materialized image at one constant, so there is no per-image cost to order by. Do not add one.
Acceptance
- After a ContextLength rejection that omits images, the ledger holds one transition per omitted Tool Result, and the retried request matches what the reducer now produces.
- The next Turn, a restart, and a branch from that Session send the omitted form; the image does not come back.
- No new in-memory map of omissions on the turn state.
- Existing archive transitions and their reducer tests keep passing;
packages/runtimetest:distcovers it.
Sequencing
Land this after #4559. That change rewrites the tail and overflow path in ai-sdk-compaction.ts, and a transition-based omission written against today's code would conflict with it. Rebase on whatever #4559 leaves.
Out of scope
- Removing
midTurnRequestPayloadCharsas a sizing authority. - Per-image cost on the artifact projection part. Rejected on Context budget: an undeclared window fabricates a capacity that kills live turns #4458.
Activity
liuxiaocs7 commentedon Sep 2, 2026
take
liuxiaocs7 commentedon Sep 2, 2026
wait #4559