Skip to content

[Bug]: Claude subagent progress re-stores the full subagent prompt on every update (~50x write amplification in statev2.sqlite) #18204

Closed
Closed
[Bug]: Claude subagent progress re-stores the full subagent prompt on every update (~50x write amplification in statev2.sqlite)#18204
Labels
bugSomething is broken or behaving incorrectly.via-triageFiled through npx t3 triage

Description

@Luzivog
Luzivog
opened on Oct 11, 2026

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. On Orchestrator V2 with the Claude provider, start a thread that launches several subagents doing long, tool-heavy work (each runs for many minutes and emits frequent progress).
  2. Open statev2.sqlite read-only and compare stored bytes with final state per subagent:
WITH x AS (
  SELECT json_extract(payload_json, '$.id') AS id,
         count(*) AS updates,
         count(DISTINCT json_extract(payload_json, '$.prompt')) AS distinct_prompts,
         sum(length(payload_json)) AS stored,
         max(length(payload_json)) AS final
  FROM orchestration_events
  WHERE event_type = 'turn-item.updated'
    AND json_extract(payload_json, '$.type') = 'subagent'
  GROUP BY 1
)
SELECT count(*) AS subagents, sum(updates) AS updates,
       sum(distinct_prompts = 1) AS prompt_never_changed,
       sum(stored) AS stored_bytes, sum(final) AS final_bytes
FROM x;

Run the same query with event_type = 'subagent.updated' (and without the $.type filter) for the second copy.

Expected behavior

Event-store growth is roughly proportional to what changed. A progress tick that changes only progress stores the change, not the whole subagent record again.

Actual behavior

Each Claude subagent progress tick is written as two full snapshots: a turn-item.updated (item type subagent) and a subagent.updated. Both carry the complete prompt, although the prompt almost never changes between ticks. ProviderEventIngestor persists every subagent.updated as-is, and nothing coalesces or diffs these events.

Measurements from one install, one UTC day (aggregate numbers only):

  • 123,376 V2 events, 334 MiB of payload_json + metadata_json.
  • By event type (count, payload MiB): turn-item.updated 62,295 / 200.0; subagent.updated 15,469 / 68.2; node.updated 35,068 / 28.2; message.updated 2,599 / 4.4; provider-turn.updated 3,617 / 3.3; checkpoint.captured 245 / 2.3. Every other type was under 1.2 MiB.
  • Subagents: 137 subagents produced 15,469 progress updates (mean 113, max 1,258 for one subagent). Each update is stored twice, so there were 30,938 events.
    • The prompt (mean 3.3 KiB, max 7.4 KiB) was byte-identical across all updates for 100 of the 137 subagents, and had at most 5 distinct values for the rest.
    • The field that changes is progress (13,229 distinct values).
    • Repeated prompt text alone was 103.8 MiB, 31% of the day's event bytes.
  • Amplification: turn-item.updated subagent items stored 73.9 MiB for 1.5 MiB of final state (49.8x); subagent.updated stored 68.2 MiB for 1.43 MiB of final state.
  • For comparison, other item types in the same day stay close to their final size: command_execution 69.9 MiB stored / 51.0 MiB final (1.4x), dynamic_tool 30.7 / 29.5, reasoning 16.7 / 16.3, assistant_message 3.4 / 3.3.
  • Across six consecutive days, subagent snapshot events were 17–42% of each day's event bytes, 797 of 2,634 MiB in total (30%); repeated prompt text was 552 MiB (21%).

Effect on the database:

  • statev2.sqlite is 15.6 GB after one week on V2. That includes history imported from V1 at the migration.
  • V2 event bytes per day were 248, 630, 320, 579, 523 and 334 MiB. The data directory grew 0.8–2.0 GiB a day (mean about 1.35 GiB). The difference is indexes and projection rows; that split is an estimate, as I did not run a full dbstat per day.
  • PRAGMA auto_vacuum is 0 (NONE), so deleted rows would not shrink the file without a full VACUUM.
  • ProjectionMaintenanceV2.compactEventStore has no caller in the installed builds or on main (18656a3) apart from FoundationPersistence.test.ts; fix(server): projection maintenance finally has an entrypoint #15759, which scheduled it, was closed pending approval. Even when it runs, it would not help here: it keeps turn-item.updated by design ("replay assigns positions on first write") and does not treat subagent.updated as supersedable.

Impact

Minor bug or occasional failure

Version or commit

0.0.46-nightly.20261007.2787 (measured). On main @ 18656a3, ProviderEventIngestor still persists every subagent.updated payload as-is.

Environment

Linux x64 server (t3 serve), Orchestrator V2, statev2.sqlite in WAL mode, Claude provider (Claude Agent SDK) with heavy subagent use (100+ subagents a day).

Suggestions, in order of impact

  1. Progress events carry only changed fields (for example progress, status, timestamps), never the unchanged prompt. Alternatively, skip a subagent.updated / turn-item.updated whose payload equals the last stored one apart from progress, as perf(server): ACP tool updates no longer persist a snapshot per streamed chunk #16682 did for ACP tool updates and perf(server): OpenCode 2 stops writing unchanged session status updates #16775 proposes for OpenCode 2.
  2. Once a subagent finishes, collapse its progress snapshots to the final one (the approach compactEventStore already takes for node.updated and message.updated).
  3. Run event compaction on a schedule (the cadence question from fix(server): projection maintenance finally has an entrypoint #15759).
  4. Create databases with auto_vacuum = INCREMENTAL, convert existing ones with a one-time VACUUM at upgrade, and run PRAGMA incremental_vacuum after compaction so freed pages return to the filesystem.

Related

Workaround

None supported. Deleting threads does not remove their events (#17845), and there is no compaction command.

Activity

  1. juliusmarminge commented on Oct 11, 2026

    @juliusmarminge
    juliusmarminge
    on Oct 11, 2026
    Member

    Note

    Grok responding on behalf of Julius.

    Confirmed write amplification on current main (18656a340). Every Claude subagent progress tick is persisted as a full snapshot, and the unchanged prompt rides along.

    What main does. ClaudeAdapterV2.updateClaudeSubagentNode emits both subagent.updated (payload = the whole OrchestrationV2Subagent, including prompt) and turn_item.updated with type: "subagent" and prompt: task.prompt. Progress frames call that path with only progress / workflow telemetry changing. ProviderEventIngestor then writes subagent.updated as-is (payload: input.event.subagent); unlike turn_item.updated, there is no strip/diff/coalesce step. So each tick stores two full copies of a mostly-static record.

    Compaction does not cover this today. ProjectionMaintenanceV2.compactEventStore has no production caller — only FoundationPersistence.test.ts. Even if it ran, it keeps turn-item.updated by design ("replay assigns positions on first write") and only supersedes message.updated, node.updated, and a fixed set of thread-state events. subagent.updated is not in that set. Scheduling that job and changing auto_vacuum stay with the product call that closed #15759; they are out of scope for this issue.

    Related, not duplicates. Same full-snapshot class as #16558 (linked-PR sync) and the growth/retention pressure in #17845 / #12758. #16682 already shipped the skip-unchanged gate for ACP tool updates; #16775 proposes the same idea for OpenCode 2. Neither touches Claude subagent progress.

    A gate that drops a subagent.updated / subagent turn-item.updated when only progress (and timestamps) changed — or that persists only the delta fields — matches the #16682 precedent and would cut the amplification the report measured. Collapsing finished subagents' progress history is a separate follow-up once compaction policy is decided.

  2. juliusmarminge
    added
    bugSomething is broken or behaving incorrectly.
    via-triageFiled through npx t3 triage
    on Oct 11, 2026

  3. ScottN-PV commented on Oct 11, 2026

    @ScottN-PV
    ScottN-PV
    on Oct 11, 2026
    Contributor

    Read on main ed0f7b9, source only: each accepted Claude task_progress frame emits a full subagent.updated and a full subagent turn-item.updated, both carrying the unchanged prompt. The numbers below are the report's; we have not reproduced it or measured a fix.

    The gate the triage describes, dropping a subagent.updated or subagent turn-item.updated when only progress changed, would also stop live progress in the clients. Those two events are the only path into the server and client projections, and subagentDetailPreview leads with progress for a running subagent, in the web tooltip and the mobile agents list. The gate would freeze that text at its first value. #16682 kept every visible-output change and sampled only streamed arguments, so it does not cover dropping progress.

    Skipping snapshots that are identical apart from timestamps would skip few updates here. The report counts 13,229 distinct progress values in 15,469 updates, and the adapter already skips empty progress frames.

    That leaves two fixes:

    1. Progress events without the prompt. Make prompt optional on OrchestrationV2Subagent and the subagent turn item, and merge on upsert in both projections (server upsertById, client upsertEntity), which replace the record today. This stops repeating the prompt, but changes the event contract (these updates carry full records today) across contracts, server and client.
    2. Bounded progress sampling in ClaudeAdapterV2. Keep full snapshots but publish progress-only changes at most once every few seconds per subagent, always emitting the first update, lifecycle and other non-progress changes, and the final state. Adapter-local. Running subagents refresh progress less often, and it is a partial reduction: each kept snapshot still repeats the prompt, and the saving depends on how fast ticks arrive.

    I'd start with 2 and leave 1 for a contracts change you own. Which do you want?

    Written by Claude Opus 5.5 and Claude Fable 5.1 on behalf of ScottN-PV.

  4. AdEx-Partners-DE commented on Oct 11, 2026

    @AdEx-Partners-DE
    AdEx-Partners-DE
    on Oct 11, 2026
    Contributor

    A second install, in case it helps to size this. I ran the two queries from the report read-only against my own statev2.sqlite (Windows 11, Claude and Codex, moderate subagent use; the file is 1,590 MiB and holds 250,812 events with 597.6 MiB of payload_json, including history imported from V1).

    The shape is the same, the share is smaller:

    subagents updates stored final state factor
    turn-item.updated, item type subagent 108 3,659 21.9 MiB 0.96 MiB 22.7x
    subagent.updated 108 3,669 20.7 MiB 0.93 MiB 22.2x
    • The prompt was byte-identical across all updates for 102 of the 108 subagents (mean prompt 2.8 KiB, mean 34 updates per subagent, at most 259).
    • Repeated prompt text is 15.7 MiB in each of the two copies, against 0.29 MiB if each prompt were stored once: 31.4 MiB together, 5.3% of all event payload bytes in this database.
    • Both snapshot streams together are 7.1% of event payload bytes here, compared with the 17 to 42% per day in the report. I launch far fewer subagents, so the cost scales with subagent use, as you would expect.
    • Other item types stay near their final size here too: command_execution 164.4 MiB stored for 139.3 MiB final (1.18x), dynamic_tool 79.6 for 71.8 (1.11x), reasoning 24.8 for 20.7 (1.20x).
    • PRAGMA auto_vacuum is 0 here as well.

    So on a lighter install the amplification per subagent is still about 22x, and the prompt is the bulk of it: storing it once would remove roughly 98% of the repeated prompt bytes. Not measured: per-day growth, and anything about a fix.

    Would it be enough to drop prompt from the progress snapshots and keep it only on the first event of each subagent, so that live progress still reaches the clients?

    Posted by Claude (Opus 5.5, Claude Code via T3 Code) on behalf of @AdEx-Partners-DE.

  5. juliusmarminge commented on Oct 14, 2026

    @juliusmarminge
    juliusmarminge
    on Oct 14, 2026
    Member

    Note

    Grok responding on behalf of Julius.

    Fixed by #18375, now on main. New Claude subagent progress events no longer store the unchanged prompt again; the prompt is stored once per subagent and restored on read. Rows written before this change keep their full prompts, and reclaiming that existing space is tracked in #15759.

  6. juliusmarminge
    closed this as completedon Oct 14, 2026
  7. AdEx-Partners-DE
    added a commit that references this issue on Oct 14, 2026

    fix(server): Claude subagent progress no longer re-stores the unchang…

    9bc2e63

  8. AdEx-Partners-DE commented on Oct 14, 2026

    @AdEx-Partners-DE
    AdEx-Partners-DE
    on Oct 14, 2026
    Contributor

    Thanks for the fix. I took #18375 into our Windows fork right after it merged, and it does what it says on a real install. Read-only queries on that server's statev2.sqlite, subagent.updated events from Claude subagents:

    48 hours before the build with #18375 first 3 hours after
    events 4,871 from 110 subagents 184 from 2 subagents
    events that store the prompt text 4,871 (all), about 3,050 characters each 2 (one per subagent)
    payload per event, average / largest 4,354 / 16,214 bytes 1,261 / 6,297 bytes

    The subagent turn-item.updated events moved the same way: 4,636 bytes on average before, 1,527 after.

    Limits of this: the after sample is two subagents in three hours, so the averages will move as more run, and I compared stored rows only; I did not check how the prompt reads back in the UI. As you wrote, the rows from before keep their size until #15759 or something like it reclaims them.

    Posted by Claude (Opus 5.5, Claude Code via T3 Code) on behalf of @AdEx-Partners-DE.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is broken or behaving incorrectly.via-triageFiled through npx t3 triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions