Skip to content

[Bug]: Codex Stop request followed by server-wide stalled turn starts on v0.0.45 #17468

Closed as not planned

Description

@Rastadco

Before submitting

  • I searched existing issues. Related reports are linked below, but I did not find an exact match for this Codex incident on v0.0.45.
  • I included the observed behavior and recovery evidence. A deterministic reproduction is not yet available.

Area

apps/server

Observed sequence

This was one observed incident, not a confirmed deterministic reproduction:

  1. Codex threads stopped producing provider events while their sessions still appeared to be running.
  2. Stop was requested for an affected thread. The server persisted thread.turn-interrupt-requested events, but no successful interruption was observed.
  3. Later messages in affected threads, another existing thread, and newly created threads were accepted and persisted as thread.turn-start-requested events. They remained pending without starting provider turns.
  4. Restarting the background service restored provider activity. A fresh message in an affected session received an agent reply.

The exact operation that blocked has not been identified. The ordering above does not prove that Stop caused the original provider stall.

Expected behavior

A failed or stuck provider operation should time out or surface an actionable error. It should not prevent unrelated threads from starting turns or handling Stop requests.

Actual behavior

Messages appeared to send successfully but agents never replied, including in new threads. The HTTP endpoint continued returning 200 and clients remained able to persist commands.

Impact

Blocks work completely until service restart.

Version or commit

Server runtime: 0.0.45. Desktop client: 0.0.45.

Environment

Linux headless background service; Codex app-server provider. Provider version was not captured during the incident.

Sanitized diagnostic observations

These are summaries of inspected state, not raw logs or fabricated log excerpts:

  • Command receipts showed the later commands as accepted, with no command rejection recorded.
  • Core thread projections had applied the latest persisted event sequence.
  • Affected active threads had no pending approvals or user-input requests.
  • Other existing/new threads had pending turns without subsequent provider-start events.
  • Service task usage was below its configured limit and memory pressure was low when inspected. Earlier logs included Error: spawn git EAGAIN; whether that earlier condition contributed is unknown.
  • After restart, previously stalled threads emitted fresh provider activity, the managed connection re-established, and a fresh-message reply was confirmed.
  • Some requests queued before restart remained pending and needed a fresh follow-up.

Possible investigation direction

In v0.0.45 ProviderCommandReactor, provider intents pass through makeDrainableWorker(processDomainEventSafely), and the interrupt path awaits providerService.interruptTurn(...). The observed server-wide stall is consistent with a blocked shared worker, but no fiber dump or instrumented timeout identifies the exact blocking await.

Per-thread isolation, bounded provider operations, and visible timeout failures may help prevent a provider stall from blocking unrelated sessions.

Related reports

Workaround

Back up session state, then restart the T3 background service. This restored replies without changing the installed version or deleting conversations. This is a recovery workaround, not a confirmed source fix.

No raw logs, database files, screenshots, prompts, session identifiers, host addresses, or local paths are attached.

Activity

  1. juliusmarminge commented on Oct 9, 2026

    @juliusmarminge
    Member

    Note

    Grok responding on behalf of Julius.

    Thanks for the careful write-up. Notes from reading v0.0.45 and current main (101f8b2f5); nothing was reproduced here.

    On 0.0.45, your reading matches the code. Every provider intent for every thread goes through one serial worker (ProviderCommandReactor.ts:1889), and the Stop path awaits providerService.interruptTurn with no deadline (:1625). A hung interrupt there would plausibly hold every later turn start on the server, which fits the sequence you saw.

    On main, that reactor is gone (replaced by the orchestrator V2 rewrite, #2829), so the exact path differs, but a similar server-wide stall looks possible:

    So if a live-but-wedged Codex app-server (which appears to be shared across Codex threads) stops answering, a Stop or a few turn starts on different threads could each pin a worker. After 4, every thread's effects, including non-Codex ones, would plausibly sit pending until a restart. A Stop on a run that has not started its provider turn cancels the stuck provider-turn.start (Orchestrator.ts:8927-8929), but a hung provider-turn.interrupt does not appear to have an equivalent escape.

    Possible trigger: spawn git EAGAIN usually means the process hit a process/thread limit or a transient resource shortage. It could have wedged the app-server or a child it was waiting on, but that is a guess from the error alone.

    Related:

    No duplicate found. Possible fixes: give Codex turn/interrupt/turn/start (or the protocol request) a deadline that fails the effect visibly, and/or bound effect execution time or reclaim expired leases so one wedged runtime cannot hold the whole pool.

    If it happens again, the most useful data would be the Codex CLI version and any orchestration_v2_effect_outbox rows stuck in running (effect type and updated_at) before restarting.

  2. added
    bugSomething is broken or behaving incorrectly.
    via-triageFiled through npx t3 triage
    needs more infoInitial triage showed no bug. Awaiting more info
    and removed
    bugSomething is broken or behaving incorrectly.
    on Oct 9, 2026
  3. juliusmarminge commented on Oct 17, 2026

    @juliusmarminge
    Member

    Note

    Grok responding on behalf of Julius.

    Closing for lack of info. This was filed against 0.0.45 with no repro on a current build. If it still happens on nightly after Stop, comment with version, logs, and steps and we can reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs more infoInitial triage showed no bug. Awaiting more infovia-triageFiled through npx t3 triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions