Skip to content

Oversized Codex diff event exhausts desktop backend heap #10924

Closed
#12305
Closed
Oversized Codex diff event exhausts desktop backend heap#10924
#12305
Labels
acceptedfeature request acceptedbugSomething is broken or behaving incorrectly.via-triageFiled through npx t3 triage

Description

@alexfertel
alexfertel
opened on Sep 9, 2026

What happened

The T3 Code desktop backend stops without warning. The UI then marks active work as failed and shows:

Provider session did not survive a server restart. Send a new message to continue.

The fault happened three times over two days.

Diagnosis

The desktop backend runs out of V8 heap while it handles a large Codex turn/diff/updated notification.

CodexAdapter places the same diff in two fields of the canonical event:

  • raw.payload.diff
  • payload.unifiedDiff

Source:

  • apps/server/src/provider/Layers/CodexAdapter.ts:985
  • apps/server/src/provider/Layers/CodexAdapter.ts:1646

ProviderService sends this canonical event to the provider event logger before it publishes the event:

  • apps/server/src/provider/Layers/ProviderService.ts:916

EventNdjsonLogger serializes the complete event and holds the resulting line in memory:

  • apps/server/src/provider/Layers/EventNdjsonLogger.ts:497
  • apps/server/src/provider/Layers/EventNdjsonLogger.ts:620

In the observed crash, one redacted canonical log record exceeded 250 MiB. The duplicate diff and temporary serialization copies pushed the backend past its heap limit.

After the backend restarts, reconcileProviderSessions marks sessions whose provider process is gone as failed. This creates the user-facing restart message:

  • apps/server/src/serverRuntimeStartup.ts:338

Expected behavior: T3 Code should bound, omit, or summarize large raw diff data before diagnostic serialization. One provider event should not stop the backend or unrelated work.

Steps to reproduce

Proposed source-level reproduction:

  1. Start the T3 Code desktop backend with the Codex provider.
  2. Send a valid turn/diff/updated notification with a large generated string in params.diff.
  3. Let CodexAdapter convert it to a canonical event.
  4. Let the canonical provider event logger serialize the event.
  5. Observe that the serialized record contains the diff twice.
  6. With a large enough diff or existing heap use, V8 stops the backend with an out-of-memory error.

A standalone synthetic test has not yet confirmed the minimum payload size.

Version

0.0.41-nightly.20260909.1439, commit 6c583620ff7a

This was the newest nightly, and its commit matched main when checked.

Environment

  • T3 Code desktop app with local backend
  • Darwin 25.6.0, arm64
  • Node.js 25.6.0
  • Codex CLI 0.153.4

Evidence

<--- Last few GCs --->

Scavenge 1856.5 (...) -> 1856.7 (...) MB
Scavenge 1856.7 (...) -> 1848.5 (...) MB

FATAL ERROR: Zone Allocation failed - process out of memory

Sanitized provider log measurement:

stream: CANON
method: turn/diff/updated
record size: greater than 250 MiB
payload content: redacted

No secrets, source paths, thread identifiers, thread titles, commands, query data, or payload content are included.

Related issues

Fix applied or workaround

Restarting the desktop app restores the backend for a time. Avoiding very large single-event diffs reduces the risk. No local data was changed.

Filed by

Codex, GPT-5, via t3 triage

Activity

  1. juliusmarminge commented on Sep 9, 2026

    @juliusmarminge
    juliusmarminge
    on Sep 9, 2026
    Member

    Triage

    Confirmed on current main (6c583620ff7a, same commit as the nightly in the report). This is a real backend heap bug, not a duplicate of #8648 or #7075.

    What the code does

    CodexAdapter builds every canonical event with the native payload attached, then maps turn/diff/updated onto payload.unifiedDiff as well:

    // apps/server/src/provider/Layers/CodexAdapter.ts (~985, ~1646)
raw: { source, method, payload: event.payload ?? {} }
// …
type: "turn.diff.updated",
payload: { unifiedDiff: payload.diff }

    ProviderService.publishRuntimeEvent writes that object to the canonical provider log before publish (ProviderService.ts ~919). EventNdjsonLogger.write then JSON-encodes the whole event and keeps the resulting line (EventNdjsonLogger.ts ~497, ~620).

    turn.diff.updated is not in the transient skip set (that filter is for high-churn deltas). There is no per-record size cap. File rotation (10 MiB) and the flush watermark (1 MiB) run after serialization, so a 250 MiB CANON record is fully materialized first.

    The user-facing restart copy is a consequence, not a separate bug: after the Node process dies, reconcileProviderSessions marks sessions whose provider process is gone as failed (serverRuntimeStartup.ts ~339).

    Orchestration does not store unifiedDiff. ProviderRuntimeIngestion only uses turn.diff.updated to dispatch a placeholder checkpoint (status: "missing", files: []). The large string exists for diagnostics (and an unused contract field), then gets serialized twice.

    A >250 MiB CANON record implies an underlying diff on the order of ~125 MiB written twice, plus the native log line, plus the [ts] CANON: … copy. That is enough to push V8 over the edge on top of a live desktop heap.

    Related, not the same

    No open PR bounds this logger path or removes the dual-field diff.

    Suggested fix

    1. In EventNdjsonLogger, omit or summarize oversized payloads before serializeEvent (keep method/type + byte count). That guards native and canonical across providers.
    2. In CodexAdapter, do not attach the full diff on both raw.payload and payload.unifiedDiff for turn/diff/updated.
    3. Add a regression test that a large turn/diff/updated cannot produce a multi-hundred-MiB CANON line.

    Restarting the desktop app is only a temporary workaround.

  2. juliusmarminge
    added
    bugSomething is broken or behaving incorrectly.
    acceptedfeature request accepted
    via-triageFiled through npx t3 triage
    on Sep 9, 2026
  3. Koushik890
    added 8 commits that reference this issue on Sep 9, 2026

    fix(server): stop an oversized provider diff from exhausting the back…

    af54f40

    fix(server): stop an oversized provider diff from exhausting the back…

    8b0e056

    fix(server): stop an oversized provider diff from exhausting the back…

    5400141
  4. juliusmarminge
    closed this as completedin #12305on Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    acceptedfeature request acceptedbugSomething is broken or behaving incorrectly.via-triageFiled through npx t3 triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions