[Bug]: Stuck OpenCode deletion blocks new chats in ThreadDeletionReactor.drainThrough #11889

Activity

1. juliusmarminge commented on Sep 15, 2026

   

   juliusmarminge

   on Sep 15, 2026

   Member

   More actions

   Triage

   Confirmed. This is a real server availability bug in current main, not a one-off 0.0.40 glitch. A stuck OpenCode thread deletion can block every new chat on that process.

   What we see

   The reported sequence matches the code and the live probe:

   1. Delete an OpenCode thread while prompt admission/cancellation is still unsettled.
   2. processThreadDeleted stays in cancelPendingOpenCodePrompt (observed 242,562,870 ms, then “Interrupted during server shutdown”).
   3. A later thread.create / first-message bootstrap persists the empty thread, then waits in ThreadDeletionReactor.drainThrough and never reaches thread.turn-start-requested.
   4. The client remains connected; HTTP still succeeds. Restart clears the stall (replay acknowledged in 18 ms).

   The exact admission race that left cancellation unsettled is not deterministic. The global drain block is.

   Why

   Two cooperating defects.

   1. OpenCode teardown waits without a bound

   cancelPendingOpenCodePrompt does:

   • Fiber.interrupt(admission.promptFiber) when the fiber exists
   • then Deferred.await(admission.submissionSettled)

   Neither has a timeout. Remote abort (abortOpenCodeSessionForTeardown, 1s) runs after this wait, so a stuck cancel never reaches the already-bounded abort. The same unbounded pair is in interruptTurn.

   submissionSettled is only completed by the early-cancel check in sendTurn, or by promptEffect's onExit. Admission is installed before promptFiber is assigned, so a delete in that window waits on a deferred that sendTurn has not completed yet.

   The in-flight path is worse. runOpenCodeSdk is Effect.tryPromise. Interrupt aborts the AbortSignal, but the fiber does not finish until the promise settles. If session.promptAsync ignores abort, Fiber.interrupt never returns, onExit never runs, and submissionSettled never completes. The 10s promptAsync timeout is part of that same fiber: interrupting it also cancels the timeout, so the bound disappears exactly when teardown needs it.

   Existing coverage bounds a hung abort (aborts a held teardown request before closing the session scope). It does not hold prompt cancellation open across stopSession.

   2. One stuck cleanup is a process-wide create barrier

   ThreadDeletionReactor uses a single DrainableWorker. drainThrough waits for the event watermark, then worker.drain (queue empty and current item finished). ws.ts awaits that on explicit thread.create and on first-message bootstrap, so a reused thread id cannot own provider/terminal resources before the prior incarnation’s cleanup finishes (#8226).

   That fence is global, not per-thread. One hung providerService.stopSession keeps outstanding > 0, so every later create waits. Failures are logged and skipped; a hang is neither, so the worker never idles. Shutdown interrupting the 67-hour span matches a stuck worker item, not a crashed process.

   Not a duplicate

   Related but distinct:

   • #5241 — orphaned opencode serve after backend crash
   • #9065 — bundled serve holding OpenCode SQLite so CLI run hangs
   • #11730 — OpenCode background delegations look Settled
   • #8796 — forget the deleted thread’s provider binding; still calls unbounded stopSession on the same shared worker
   • #11613 / #8939 / #10805 — other OpenCode admission/abort work; none bound this wait or isolate drainThrough

   Next step

   Keep this open. Fix both layers; either one alone still leaves an instance-level stall.

   1. Bound OpenCode cancel/teardown. cancelPendingOpenCodePrompt and the matching interruptTurn wait should finish or fail in a short, explicit period (timeout Fiber.interrupt / Deferred.await, or Fiber.interruptFork plus a deferred timeout), then still run the existing remote abort. Do not let a non-settling SDK promise pin stopSession.
   2. Stop using one in-flight deletion as a process-wide create barrier. drainThrough should wait only for deletions that can conflict with the new thread (same id / reincarnation), or fail open after a bound so unrelated creates proceed. A hung cleanup can stay isolated and logged.
   3. Regression. Hold an OpenCode prompt cancel open (before promptFiber assignment and during an uninterruptible/hanging promptAsync), delete that thread, then create/start an unrelated thread. Teardown must terminate or report a bounded failure; the new thread must complete without restart.

   Workaround remains: confirm no active turns, restart the headless T3 process. HTTP health checks will not see this.
2. juliusmarminge

   added

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   acceptedfeature request acceptedfeature request accepted

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   on Sep 15, 2026
3. shivamhwp

   closed this as completedin #12003on Sep 29, 2026