You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: In-memory ingestion backlog is lost on restart, bricking threads in "starting" #6560
While ingestion is backlogged, send a message to a healthy thread on another provider (Codex in the incident) and let the provider complete the whole turn. The canonical events are published, and CheckpointReactor even captures a checkpoint, but ingestion has not processed them yet.
Quit or restart the app before ingestion catches up.
After restart, open the thread and try the stop/interrupt button.
Expected behavior
Either the canonical runtime event stream should be durable enough to replay after a restart, or startup should reconcile threads whose projections show a pending or running turn with provider reality. The session directory still contains the resume cursor, and the provider-side thread shows that the turn completed.
At minimum, interrupting a thread with no active turn should clear the pending turn-start and return the session to ready, allowing the user to escape the stuck state.
Actual behavior
ProviderRuntimeIngestion.ts consumes providerService.streamEvents through a single serial DrainableWorker backed by an unbounded, in-memoryTxQueue. Ingestion stopped dispatching at 03:26:17 — whether the worker was wedged behind the backlog or its subscription had already died, the unprocessed events existed only in this in-memory pipeline, so the server shutdown (provider.stopAll at 03:39:08) discarded them.
The canonical events exist nowhere else in orchestration state; they remain only in the debug NDJSON provider log. The loss is therefore permanent:
The thread's two completed Codex turns never produced assistant thread.message-sent events or the running/ready thread.session-set transitions — only the client message-sent and the initial "starting" session-set from each turn request exist. Both assistant replies are still missing from the thread.
The session projection remains stuck at status = "starting", with a leaked pending turn-start row: a projection_turns row with a NULL turn_id.
There is no interrupt- or startup-driven escape from this state:
Interrupt is a no-op — processTurnInterruptRequested (ProviderCommandReactor.ts:1187) only calls providerService.interruptTurn. During the incident, this actually recovered the Codex session (session/connecting → session/ready, MCP servers ready), but there was no active turn to interrupt and nothing cleared the pending state.
The session cannot leave "starting" — Ingestion deliberately maps session/ready back to "starting" while a pending turn-start exists (ProviderRuntimeIngestion.ts:1564). As a result, even the successful session recovery in step 2 left the UI spinner running forever.
Incident timeline (UTC, 2026-08-13), reconstructed from orchestration_command_receipts, orchestration_events, provider_session_runtime, and the provider NDJSON logs:
03:22–03:26 Grok tool_call_update storm floods ingestion (#6556)
03:26:17 last ingestion dispatch (receipts with `provider:` command-id prefix stop)
03:26:48 user message → Codex thread; turn completes 03:29:12 (canonical events
published; checkpoint captured by CheckpointReactor) — never ingested
03:36:50 second user message; turn completes 03:37:49 — same fate
03:39:08 server shutdown (provider.stopAll) — in-memory ingestion backlog discarded
03:41:09 server back; ingestion dispatches resume for OTHER threads' fresh events only
04:16:37 user hits interrupt: Codex session reconnects and reports ready, but the
projection stays "starting"; second interrupt is a silent no-op. Frozen.
This is distinct from #5781, where the adapter consumer fiber dies and events never reach the bus. Here, the events demonstrably reached the bus: CheckpointReactor consumed them and captured checkpoints for turns whose messages were never persisted. That inconsistency also provides a useful signal for a reconciler.
macOS 26.5.2 (Darwin 25.5.0), Codex app-server provider (gpt-5.6-sol) on the affected thread, Grok CLI 1.0.3 as the flood source
Logs or stack traces
All excerpts below are from `~/.t3/userdata/state.sqlite` and `~/.t3/userdata/logs/provider/events.3540be8c-….log`, lightly annotated, with long lines truncated at `…`. State rows were captured ~2026-08-13T04:30Z, while the thread was still frozen.
Command receipts for the affected thread, 03:26–04:17. Note the total absence of `provider:`-prefixed receipts (runtime ingestion dispatches) between 03:26:48 and 04:16:37, while`server:checkpoint-*` receipts land at 03:29:12 and 03:37:49 for the very turns whose messages were never persisted:
sqlite> SELECT command_id, accepted_at FROM orchestration_command_receipts
...> WHERE aggregate_id='3540be8c-edc6-4d96-9460-decfbf8001ea'
...> AND accepted_at BETWEEN '2026-08-13T03:26' AND '2026-08-13T04:17' ORDER BY accepted_at;
e89cdc52-2416-44b9-a089-8a4d54c1ec02|2026-08-13T03:26:48.012Z -- user turn-start
server:provider-session-set:a9f5d624-1045-4a7a-803d-e15c48ff75c3|2026-08-13T03:26:48.012Z
server:checkpoint-turn-diff-complete:b3fb918e-7144-4147-b28b-23d6c7225049|2026-08-13T03:29:12.794Z
server:checkpoint-captured-activity:cd9dd77b-3125-4054-9ebf-6e43976608f2|2026-08-13T03:29:12.794Z
7118e14f-ef65-48f9-a5b8-550104e312ac|2026-08-13T03:36:50.279Z -- user turn-start
server:provider-session-set:606ead09-d1d4-488a-889a-e3e17f709e7c|2026-08-13T03:36:50.279Z
server:checkpoint-turn-diff-complete:c160cba2-c6c0-4e6f-a463-34b1127f6cd6|2026-08-13T03:37:49.707Z
server:checkpoint-captured-activity:77640006-c6b5-401e-b93f-20121c692e8e|2026-08-13T03:37:49.707Z
741930ea-92a2-4eea-bd1b-8837c1427229|2026-08-13T04:16:37.821Z -- user interrupt
provider:df1e199f-2425-44dd-9701-6c1283f5aaed:thread-session-set:1412d1de-…|2026-08-13T04:16:37.847Z
provider:9b3f44ce-0f90-4221-b683-ca85baf2714f:thread-session-set:98f272c0-…|2026-08-13T04:16:37.969Z
provider:03c8a382-0c91-409c-8e48-e13ddd66a435:thread-activity-append:4982f2a4-…|2026-08-13T04:16:37.970Z
501b5b4a-5a05-42b5-8e33-2f679e27ad30|2026-08-13T04:16:38.993Z -- second interrupt: no receipts follow
State rows while frozen — session directory says the session is fine (recovered, resume cursor intact, last event `provider.stopAll` at shutdown), but the projection is stuck `starting` with a leaked pending turn row:
sqlite> SELECT * FROM provider_session_runtime WHERE thread_id='3540be8c-…';
3540be8c-edc6-4d96-9460-decfbf8001ea|codex|codex|auto|running|2026-08-13T04:16:37.969Z|{"threadId":"019fbbac-1f21-7953-96c9-6e67bd93a5cf"}|{"cwd":"/Users/kelchm/.t3/worktrees/home-lab/t3code-6046b50c","model":"gpt-5.6-sol","activeTurnId":null,"lastError":null,…,"lastRuntimeEvent":"provider.stopAll","lastRuntimeEventAt":"2026-08-13T03:39:08.912Z"}|codex
sqlite> SELECT * FROM projection_thread_sessions WHERE thread_id='3540be8c-…';
3540be8c-edc6-4d96-9460-decfbf8001ea|starting|codex|||||2026-08-13T04:16:37.969Z|auto|codex
sqlite> SELECT turn_id, state, requested_at, started_at, completed_at FROM projection_turns
...> WHERE thread_id='3540be8c-…' ORDER BY requested_at DESC LIMIT 2;
019ff931-c406-7f42-af01-ae5b90c6543a|completed|2026-08-13T03:37:49.707Z|2026-08-13T03:37:49.707Z|2026-08-13T03:37:49.707Z
|pending|2026-08-13T03:36:50.279Z||
Provider NDJSON log proves the turn completed and the reply text exists — it just never reached orchestration:
[2026-08-13T03:37:49.712Z] NTIVE: {"id":"b04f5a5f-72c5-4dcc-85f9-5bd859e8483f","provider":"codex","providerInstanceId":"codex","createdAt":"2026-08-13T03:37:49.707Z","kind":"notification","threadId":"3540be8c-edc6-4d96-9460-decfbf8001ea","method":"turn/completed","turnId":"019ff931-c406-7f42-af01-ae5b90c6543a","payload":{"threadId":"019fbbac-1f21-7953-96c9-6e67bd93a5cf","turn":{"completedAt":1786592269,"durationMs":53518,"error":null,…
[2026-08-13T03:37:49.712Z] NTIVE: {"id":"efcc8fa0-7b91-49cf-a3ae-62bb0bb99251",…,"method":"item/completed","turnId":"019ff931-c406-7f42-af01-ae5b90c6543a","itemId":"msg_039b496747c19434016a7d3c079e888195bb20b71e97ad3fda","payload":{"completedAtMs":1786592269433,"item":{…,"phase":"final_answer","text":"Going well. Current state:\n\n- [Draft PR #306](https://gi…[2026-08-13T03:37:49.719Z] CANON: {"eventId":"b04f5a5f-72c5-4dcc-85f9-5bd859e8483f","provider":"codex","threadId":"3540be8c-edc6-4d96-9460-decfbf8001ea","createdAt":"2026-08-13T03:37:49.707Z","turnId":"019ff931-c406-7f42-af01-ae5b90c6543a","providerRefs":{"providerTurnId":"019ff931-c406-7f42-af01-ae5b90c6543a"},"raw":{"source":"codex.app-server.notification","method":"turn/completed",…
Screenshots, recordings, or supporting files
No response
Workaround
Send a new message to the frozen thread. The new turn-start supersedes the stale pending one, and the recovered session handles it — confirmed on the affected thread roughly 21 hours later: the first new message immediately started a turn that ran and completed normally. The lost assistant replies can be recovered manually from ~/.t3/userdata/logs/provider/events.<threadId>.log; the item.completedagentMessage entries contain the full text.
changed the title [-][Bug]: ProviderRuntimeIngestion backlog is in-memory only — a restart during a flood permanently loses turn lifecycle events and bricks the thread in "starting"[/-][+][Bug]: In-memory ingestion backlog is lost on restart, bricking threads in "starting"[/+]on Aug 14, 2026
Independent repro on macOS with T3 Code Nightly 0.0.34-nightly.20260817.1113.
Affected T3 thread: 3bbf530c-b3a3-4756-913f-fe9064c2d1f7
The provider turn completed successfully at 2026-08-17T05:11:38.759Z and the checkpoint was persisted. The final assistant response is still recoverable in the provider NDJSON log. However, no assistant thread.message-sent event or completed session lifecycle transition was projected.
State after restart:
projection_turns: completed
provider_session_runtime: stopped
projection_thread_sessions: starting
pending_messages: orphan row with a null turn ID
projection_thread_messages: user prompt only, no assistant message
UI: still shows Working with only the prompt visible
The restart dropped the in-memory ingestion backlog and left the thread internally contradictory. The source flood in this run is documented in #5681. Runtime events adjacent to this thread checkpoint were about 31 minutes old when the checkpoint was persisted.
This looks like the same durability boundary described here, with a concrete case where the provider result survived on disk but the canonical projection did not.
Before submitting
Area
apps/server
Steps to reproduce
Observed in production use on 2026-08-13 (timeline below). A deterministic reproduction should be possible with these steps:
tool_call_updatestorm in [Bug]: Grok ACP resends full cumulative terminal output in everytool_call_update, flooding ingestion for all threads #6556 or the Codex subagent progress flood in [Bug]: Codex subagent progress floods orchestration ingestion and delays unrelated threads #5681.CheckpointReactoreven captures a checkpoint, but ingestion has not processed them yet.Expected behavior
Either the canonical runtime event stream should be durable enough to replay after a restart, or startup should reconcile threads whose projections show a pending or running turn with provider reality. The session directory still contains the resume cursor, and the provider-side thread shows that the turn completed.
At minimum, interrupting a thread with no active turn should clear the pending turn-start and return the session to
ready, allowing the user to escape the stuck state.Actual behavior
ProviderRuntimeIngestion.tsconsumesproviderService.streamEventsthrough a single serialDrainableWorkerbacked by an unbounded, in-memoryTxQueue. Ingestion stopped dispatching at 03:26:17 — whether the worker was wedged behind the backlog or its subscription had already died, the unprocessed events existed only in this in-memory pipeline, so the server shutdown (provider.stopAllat 03:39:08) discarded them.The canonical events exist nowhere else in orchestration state; they remain only in the debug NDJSON provider log. The loss is therefore permanent:
thread.message-sentevents or the running/readythread.session-settransitions — only the client message-sent and the initial "starting" session-set from each turn request exist. Both assistant replies are still missing from the thread.status = "starting", with a leaked pending turn-start row: aprojection_turnsrow with a NULLturn_id.There is no interrupt- or startup-driven escape from this state:
processTurnInterruptRequested(ProviderCommandReactor.ts:1187) only callsproviderService.interruptTurn. During the incident, this actually recovered the Codex session (session/connecting→session/ready, MCP servers ready), but there was no active turn to interrupt and nothing cleared the pending state.session/readyback to"starting"while a pending turn-start exists (ProviderRuntimeIngestion.ts:1564). As a result, even the successful session recovery in step 2 left the UI spinner running forever.Incident timeline (UTC, 2026-08-13), reconstructed from
orchestration_command_receipts,orchestration_events,provider_session_runtime, and the provider NDJSON logs:This is distinct from #5781, where the adapter consumer fiber dies and events never reach the bus. Here, the events demonstrably reached the bus:
CheckpointReactorconsumed them and captured checkpoints for turns whose messages were never persisted. That inconsistency also provides a useful signal for a reconciler.Impact
Major degradation or frequent failure
Version or commit
Desktop 0.0.33 / main @ e5c82d7
Environment
macOS 26.5.2 (Darwin 25.5.0), Codex app-server provider (gpt-5.6-sol) on the affected thread, Grok CLI 1.0.3 as the flood source
Logs or stack traces
Screenshots, recordings, or supporting files
No response
Workaround
Send a new message to the frozen thread. The new turn-start supersedes the stale pending one, and the recovered session handles it — confirmed on the affected thread roughly 21 hours later: the first new message immediately started a turn that ran and completed normally. The lost assistant replies can be recovered manually from
~/.t3/userdata/logs/provider/events.<threadId>.log; theitem.completedagentMessageentries contain the full text.