Skip to content

[Bug]: One unresponsive provider holds all four effect workers and stalls every thread #16636

Open
#16790
Open
[Bug]: One unresponsive provider holds all four effect workers and stalls every thread#16636
#16790
Labels
bugSomething is broken or behaving incorrectly.via-triageFiled through npx t3 triage

Description

@3mdistal
3mdistal
opened on Oct 7, 2026

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. Make one provider stop answering control requests without exiting. Ours was the shared Codex app-server on Windows after it ran out of blocking threads ([Bug]: Codex app-server keeps per-thread MCP servers running after threads settle #15979, [Windows] Idle local STDIO MCP servers can exhaust Tokio blocking pool and stall the app-server openai/codex#42740); a stub provider whose startTurn never resolves should do the same.
  2. Send messages to four or more threads on that provider.
  3. Send a message to a thread on a different provider, such as Claude.

Expected behavior

The Claude thread starts. A stalled provider fails or times out its own effects without holding up the others.

Actual behavior

Every thread stalls, on every provider. Each unanswered provider-turn.start (and later provider-turn.interrupt) holds one of the orchestration effect workers, and there are four (DEFAULT_EFFECT_WORKER_CONCURRENCY in orchestration-v2/EffectWorker.ts). Each claim has a 30-second lease, but an expired lease isn't a deadline: the effect keeps running, no worker frees up, and Claude launches, checkpoints, and deliveries wait in orchestration_v2_effect_outbox behind it. At the first lockup, four effects (three provider-turn.start, one provider-turn.interrupt) had been running for 20 to 40 minutes, long past their leases, with eleven pending behind them: five starts, four checkpoints, a steer, and a title generation. Recovery needed restarting the Codex app-server once and the T3 backend once, which interrupted every in-flight run.

Impact

Blocks work completely

Version or commit

0.0.46-nightly.20261005.2702 (first lockup) and 0.0.46-nightly.20261006.2735 (second)

Environment

Windows 11, T3 Code desktop, codex-cli 0.160.1 as the shared Codex app-server, Claude as a second provider

Logs or stack traces

orchestration_v2_effect_outbox at 12:40 PM, first lockup:
  running  provider-turn.start      claimed 12:00:19  attempt_count 1  lease long expired
  running  provider-turn.start      claimed 12:01:04  attempt_count 1  lease long expired
  running  provider-turn.interrupt  claimed 12:19:26  attempt_count 1  lease long expired
  running  provider-turn.start      claimed 12:19:40  attempt_count 1  lease long expired
  pending  11 more: 5 starts, 4 checkpoints, 1 steer, 1 title generation
Codex provider log: turn/start, thread/start, turn/interrupt, thread/resume sent, never answered;
the app-server stayed alive and kept sending notifications.
For comparison, provider-turn.start normally finishes in p50 6.4 s, p95 26 s, max 33 s.

Suggested fix: give provider control effects an execution deadline that fails the run, and run each provider's effects on its own workers, so one wedged provider can't take every worker.

Activity

  1. juliusmarminge commented on Oct 7, 2026

    @juliusmarminge
    juliusmarminge
    on Oct 7, 2026
    Member

    Note

    Grok responding on behalf of Julius.

    Confirmed on current main (517188b). One provider that stops answering can occupy every orchestration effect worker.

    Root cause.

    • Fixed pool of four workers.

      • apps/server/src/orchestration-v2/EffectWorker.ts:781 sets DEFAULT_EFFECT_WORKER_CONCURRENCY = 4.
      • runDaemonWithOptions (:784-853) starts that many identical runWorker loops, and layerDaemon (:857-858) uses the default.
      • The loops are not partitioned by provider. Per-thread ordering in EffectOutbox.ts:314-353 (claimableCandidatePredicate) only stops two effects on the same thread from running together, so the four slots are shared by every thread on every provider.

    • No execution deadline. runOnce (EffectWorker.ts:681-686) awaits Effect.raceFirst(executor.execute(...), cancellation). The only other branch is an explicit cancellation, with no timeout.

      • provider-turn.start → ProviderTurnStartService.start and provider-turn.interrupt → ProviderTurnControlService.interrupt (dispatch at EffectWorker.ts:152-204) add no timeout either. ProviderTurnControlService.ts:194-207 has a 2s deadline, but it only covers the poll after interruptTurn returns.
      • In the Codex adapter, client.request("turn/start", ...) (CodexAdapterV2.ts:5890) and the thread/start / thread/resume requests (~:6079-6110) are unbounded.
      • packages/effect-codex-app-server/src/client.ts has no default request timeout.
      • Only some interrupt sub-steps are bounded: timeoutOption("10 seconds") around waiting for completion (:6550, :6624). The bare turn/interrupt request at :6561 is not.

    • The lease is never enforced while the process is alive.

      • claimNext (EffectOutbox.ts:518-552) writes lease_expires_at but only claims status = 'pending' rows (:324).
      • Nothing compares lease_expires_at with the current time, and there is no renewal or reclaim of expired leases.
      • Expired running rows are only handled by reconcileAfterProcessLoss (:483-512) on the next server start.
      • workerId is per process (EffectWorker.ts:538), so the ownership checks in succeed/fail/retry still pass long after the lease has expired.

      In practice the 30s lease is metadata only. A fiber blocked on a silent provider holds its slot until that provider answers or the server restarts, which matches the outbox snapshot in the report.

    Suggested fix direction. Both parts from the report seem right, and they work independently:

    1. Give provider-control effects an execution deadline that fails the effect on timeout. Either raise it above the observed p95/max for provider-turn.start (~33s) or derive it from the lease and renew the lease while the call is healthy. On timeout, interrupt the fiber so its slot frees up, and settle the run through the existing retry/fail path.
    2. Partition worker capacity, either with per-provider (or per-provider-instance) lanes or a per-provider cap on running effects inside the claim query. That way one wedged provider can't take every slot.

    A unit test with a stub executor whose provider-turn.start never resolves, plus four or more threads, would reproduce this deterministically.

    Related.

    I didn't find an open PR that adds a deadline or per-provider lanes.

  2. juliusmarminge
    added
    bugSomething is broken or behaving incorrectly.
    via-triageFiled through npx t3 triage
    on Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is broken or behaving incorrectly.via-triageFiled through npx t3 triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions