Repository navigation
[Bug]: Codex Stop request followed by server-wide stalled turn starts on v0.0.45 #17468
Description
Before submitting
- I searched existing issues. Related reports are linked below, but I did not find an exact match for this Codex incident on v0.0.45.
- I included the observed behavior and recovery evidence. A deterministic reproduction is not yet available.
Area
apps/server
Observed sequence
This was one observed incident, not a confirmed deterministic reproduction:
- Codex threads stopped producing provider events while their sessions still appeared to be running.
- Stop was requested for an affected thread. The server persisted
thread.turn-interrupt-requestedevents, but no successful interruption was observed. - Later messages in affected threads, another existing thread, and newly created threads were accepted and persisted as
thread.turn-start-requestedevents. They remained pending without starting provider turns. - Restarting the background service restored provider activity. A fresh message in an affected session received an agent reply.
The exact operation that blocked has not been identified. The ordering above does not prove that Stop caused the original provider stall.
Expected behavior
A failed or stuck provider operation should time out or surface an actionable error. It should not prevent unrelated threads from starting turns or handling Stop requests.
Actual behavior
Messages appeared to send successfully but agents never replied, including in new threads. The HTTP endpoint continued returning 200 and clients remained able to persist commands.
Impact
Blocks work completely until service restart.
Version or commit
Server runtime: 0.0.45. Desktop client: 0.0.45.
Environment
Linux headless background service; Codex app-server provider. Provider version was not captured during the incident.
Sanitized diagnostic observations
These are summaries of inspected state, not raw logs or fabricated log excerpts:
- Command receipts showed the later commands as accepted, with no command rejection recorded.
- Core thread projections had applied the latest persisted event sequence.
- Affected active threads had no pending approvals or user-input requests.
- Other existing/new threads had pending turns without subsequent provider-start events.
- Service task usage was below its configured limit and memory pressure was low when inspected. Earlier logs included
Error: spawn git EAGAIN; whether that earlier condition contributed is unknown. - After restart, previously stalled threads emitted fresh provider activity, the managed connection re-established, and a fresh-message reply was confirmed.
- Some requests queued before restart remained pending and needed a fresh follow-up.
Possible investigation direction
In v0.0.45 ProviderCommandReactor, provider intents pass through makeDrainableWorker(processDomainEventSafely), and the interrupt path awaits providerService.interruptTurn(...). The observed server-wide stall is consistent with a blocked shared worker, but no fiber dump or instrumented timeout identifies the exact blocking await.
Per-thread isolation, bounded provider operations, and visible timeout failures may help prevent a provider stall from blocking unrelated sessions.
Related reports
- [Bug]: Changing runtime mode mid-turn hangs ProviderCommandReactor; all new sessions stay Connecting or show no status #6517: older shared-worker lockout after a runtime-mode change with a different provider.
- Stop button leaves a thread stuck "Working" forever when the provider's abort fails #7368: older Stop failure leaving a thread running with a different provider.
Workaround
Back up session state, then restart the T3 background service. This restored replies without changing the installed version or deleting conversations. This is a recovery workaround, not a confirmed source fix.
No raw logs, database files, screenshots, prompts, session identifiers, host addresses, or local paths are attached.
Activity
juliusmarminge commentedon Oct 9, 2026
Note
Grok responding on behalf of Julius.
Thanks for the careful write-up. Notes from reading v0.0.45 and current
main(101f8b2f5); nothing was reproduced here.On 0.0.45, your reading matches the code. Every provider intent for every thread goes through one serial worker (
ProviderCommandReactor.ts:1889), and the Stop path awaitsproviderService.interruptTurnwith no deadline (:1625). A hung interrupt there would plausibly hold every later turn start on the server, which fits the sequence you saw.On main, that reactor is gone (replaced by the orchestrator V2 rewrite, #2829), so the exact path differs, but a similar server-wide stall looks possible:
EffectOutbox.ts:313-358), so one stuck thread no longer blocks others by itself.EffectWorker.ts:792,:856-861). An execution is raced only against cancellation, with no timeout (:692-697). The 30s lease is written on claim (EffectOutbox.ts:523-551) but nothing appears to reclaim an expired one; stuckrunningrows look like they clear only on process restart (:488-516).Deferredthat fails only if the transport closes (protocol.ts:450-467). The rootturn/interrupt(CodexAdapterV2.ts:6936-6939) andturn/start(:6204) have no deadline, while descendant interrupts and the completion wait do get 10s (:6924-6925,:6995-6999).ProviderTurnControlService.ts:184-188awaits that interrupt directly.So if a live-but-wedged Codex app-server (which appears to be shared across Codex threads) stops answering, a Stop or a few turn starts on different threads could each pin a worker. After 4, every thread's effects, including non-Codex ones, would plausibly sit pending until a restart. A Stop on a run that has not started its provider turn cancels the stuck
provider-turn.start(Orchestrator.ts:8927-8929), but a hungprovider-turn.interruptdoes not appear to have an equivalent escape.Possible trigger:
spawn git EAGAINusually means the process hit a process/thread limit or a transient resource shortage. It could have wedged the app-server or a child it was waiting on, but that is a guess from the error alone.Related:
turn/start, 5s parentturn/interrupt) but was closed because it touched the removed v1 Codex runtime; nothing equivalent appears to have landed on the V2 Codex adapter.No duplicate found. Possible fixes: give Codex
turn/interrupt/turn/start(or the protocolrequest) a deadline that fails the effect visibly, and/or bound effect execution time or reclaim expired leases so one wedged runtime cannot hold the whole pool.If it happens again, the most useful data would be the Codex CLI version and any
orchestration_v2_effect_outboxrows stuck inrunning(effect type andupdated_at) before restarting.juliusmarminge commentedon Oct 17, 2026
Note
Grok responding on behalf of Julius.
Closing for lack of info. This was filed against 0.0.45 with no repro on a current build. If it still happens on nightly after Stop, comment with version, logs, and steps and we can reopen.