[Bug]: V2 stores tool-result image bytes twice per item, and no client reads them #16647

Activity

1. juliusmarminge commented on Oct 7, 2026

   

   juliusmarminge

   on Oct 7, 2026

   Member

   More actions

   Note

   Grok responding on behalf of Julius.

   Triage: confirmed on current main (517188b)

   I checked this by reading the code, not by running it. Nothing between the provider adapter and SQLite removes tool-result image bytes, and the only thing that serves image bytes back out doesn't recognize these shapes.

   Where the bytes come in, unchanged

   • Claude: claudeToolResultEntriesFromMessage passes message.tool_use_result through as the structured output (apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts L2217, claudeNativeToolOutputFromStructuredResult L1906–1915). That value then lands on the dynamic_tool item as output (L3934, L3989). The Read preview uses viewedImagePath (L3920–3926), not these bytes.
   • ACP (Grok): rawOutput ?? content goes to output as-is (AcpAdapterV2.ts L3253, L3329 via acpMcpToolCallOutput L797–810, and L3355 for read).
   • Cursor: cursorToolOutput returns result.value unchanged (CursorAdapterV2.ts L362–368, used at L1245–1247 and L1294–1296).
   • dynamic_tool.output is Schema.Unknown in both the domain schema and the JSON schema (packages/contracts/src/orchestrationV2.ts L1570, L2345).

   Where it gets written twice

   • ProviderEventIngestor.ts L463–471 wraps the turn item into a turn-item.updated domain event without changing it.
   • orchestration_events: appendAgentEvents writes the encoded payload to payload_json (apps/server/src/persistence/OrchestrationEventStore.ts L433–473).
   • orchestration_v2_projection_turn_items: the turn-item.updated projector writes encodeTurnItemPayload(event.payload) to payload_json (ProjectionStore.ts L2360–2405, encoder L991–993).
   • compactEventStore deliberately skips turn-item.updated (ProjectionMaintenance.ts L239, L297–308), so the event-log copy is never cleaned up.

   Why nothing reads them

   • readToolOutputImage only matches a top-level { type: "image", data, mimeType } or { type: "image", source: { type: "base64", … } } block (packages/shared/src/toolOutput.ts L203–214). Claude's output.file.base64, Grok's ImageContent.data and Cursor's content[].image.data don't fit either shape.
   • Because of that, the tool-output-image asset route never returns them (apps/server/src/assets/AssetAccess.ts L226–238 → toolOutputImages, toolOutput.ts L230–239). The client asset list behaves the same way (packages/client-runtime/src/work-log/itemDetail.ts L208–219).
   • The timeline projection drops output (WireProjection.ts L113–121).
   • The on-demand detail read (projectTurnItemForDetail → boundToolOutput, WireProjection.ts L155–166, L181–186) uses omitToolOutputImageData, which doesn't recognize these shapes either. So up to 256 KB of the base64 still goes over the wire on a detail read. For the Claude shape, the client then shows it as [image] (itemDetail.ts L28). That's transport, not use.

   Suggested fix

   • Remove or replace unrecognized inline image payloads once, when the item is ingested, before both writes. The adapters or ProviderEventIngestor are good places for this. Keep a small marker such as { type: "image", mimeType, omitted: true } so the stored rows stay tiny.
   • Leave alone the blocks toolOutputImages already recognizes, because the tool-output-image asset route (fix: tool screenshots show as images, not base64 text #16199) needs them.
   • Optional: convert the provider shapes to the recognized block shape, but only if those images should appear as assets. That's a separate choice.
   • Reclaiming bytes that are already stored (a rewrite or migration plus VACUUM) is a separate storage decision for maintainers. The fix going forward doesn't depend on it.

   Related

   • fix: tool screenshots show as images, not base64 text #16199 (merged): added the tool-output-image asset path for recognized blocks. It doesn't cover these shapes.
   • [Bug]: iOS app fails to open a thread on slow links ("syncing → failed to connect") after an image Read — server inlines full base64 images into thread activity payloads #4595 (closed): stopped inlining images into the thread activity sent over the wire. That was about transport, not storage.
   • preview_snapshot includeImage embeds full-res base64 PNGs into tool history; providers that reject inline images brick the session permanently #11295 (open) and fix(server): require explicit inline preview screenshots #11380 (open PR): both about preview_snapshot screenshots, which are the images the asset route does serve. Neither covers this.
   • I found no open PR that addresses this.
2. juliusmarminge

   added

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   on Oct 7, 2026
3. t3dotgg

   closed this as completedin #16652on Oct 7, 2026