bug(server): Claude sendTurn hangs on a silent CLI, then Stop reports 'Provider turn start failed'

[nohat]

Problem

A Claude thread showed Provider turn start failed: ProviderAdapterSessionClosedError: claudeAgent adapter thread is closed (thread 67f4d809-8d85-48ea-95c0-068b5e368989, production release 5561b14f9c, 2026-10-02). The thread had looked stuck for about ten minutes. The error appeared only after Stop was pressed, so it reads as a crash when it is the result of the stop. The two messages sent during the stall were never queued.

What the logs show (all times UTC)

Time	Event
15:19	Last provider event for the thread, then 18 minutes of silence (events per minute: 15:19 = 26, 15:20 to 15:37 = 0, 15:38 = 86)
15:28:00	sendTurn #1 starts (model claude-sonnet-5-5, interaction mode default) and never returns
15:38:20	sendTurn #2 starts and also never returns
15:38:31.76	Both sendTurn spans fail with ProviderAdapterSessionClosedError; cause is Error: Query closed before response received
15:38:31.76 to :33.77	interruptTurn runs stopSessionInternal, which takes 2006 ms
15:38 onward	New system/init events and the thread works again

Sources: ~/.t3/userdata/logs/server.trace.ndjson.2 (the four sendTurn failure spans and the interruptTurn span) and ~/.t3/userdata/logs/provider/events.67f4d809-8d85-48ea-95c0-068b5e368989.log (event counts).

Root cause (probable)

Before sendTurn queues the message in apps/server/src/provider/Layers/ClaudeAdapter.ts, it awaits SDK control requests: query.setModel(...) when the model changed, and query.setPermissionMode(...) on every send that carries an interaction mode. These wait for the Claude CLI to answer and have no timeout. While the CLI is silent they block, and every later send on the thread queues behind the same stall.

interruptTurn then calls stopSessionInternal, whose query.close() rejects the pending request with Query closed before response received. toSessionError maps any message containing "closed" to ProviderAdapterSessionClosedError, which the orchestrator reports as a failed turn start. Stopping the thread on purpose is therefore reported as a provider failure.

Not verified

• Which control call was pending. The trace spans carry no attribute for it, so setModel and setPermissionMode cannot be told apart.
• Why the CLI stopped emitting events at 15:19 (a long tool or subagent, or a hung process). The thread had background Task subagents running (task_progress events, scaffold-dev).
• Whether upstream main behaves the same. This is the same code path as upstream's, but only the fork's packaged build was observed.

Proposed fixes

1. Bound the control requests in sendTurn with a timeout, and skip setPermissionMode when the mode is unchanged (track it as currentApiModelId is tracked), so a busy CLI cannot hold a message.
2. When the session was closed by our own stopSessionInternal (context.stopped), report an interrupted or stopped outcome instead of a turn start failure. Narrow the includes("closed") match in toSessionError.
3. Show a visible waiting state when sendTurn has been pending for more than a few seconds, so the thread does not look dead.
4. Add the pending control method to the sendTurn span attributes so the next occurrence is diagnosable.

Severity

Medium. Stop recovers the thread, but messages sent during the stall are lost and the error text misleads. It was seen on the scaffold orchestrator thread, which runs long background subagents, so it may recur there.

Related

Upstream pingdotgg#2336 (closed) covers a different failure, a dangling resume cursor after the CLI is killed before its first turn. It is not this bug.

Reported from the production build; not yet reproduced on demand.

[nohat]

Second occurrence, same thread, 08:56 local (15:56 UTC). Raising to high: this is now a recurring blocker on the scaffold orchestrator thread.

• thread.turn.start was dispatched at t+0. Everything before the adapter (baseline, session lookup) finished within 60 ms. sendTurn then sat for 16.1 s until Stop (thread.turn.interrupt at +16.14 s) closed the query and failed it with the same ProviderAdapterSessionClosedError.
• Provider events had been silent since 15:40:01 UTC (16.5 minutes). In all three silences on this thread (15:19 to 15:38, 15:40 to 15:56, and an earlier 62-minute gap) the last events were subagent task_progress lines.
• Correction to the first report: the 2 s stopSessionInternal is not evidence of an unresponsive CLI (the interrupt grace is 3 s and resumed background tasks keep the CLI alive on purpose).

Narrowed cause. The prompt queue is unbounded, the model matched the running session (claude-sonnet-5-5), and the other steps in sendTurn are local. The only awaited call that talks to the CLI before the message is queued is query.setPermissionMode("default"), which runs on every send that carries interactionMode: "default", whether or not the mode changed. So with a busy or stalled CLI, every ordinary message waits on a control response it does not need.

Fix in progress: skip setPermissionMode and setModel when the value is unchanged, and bound them with a timeout that fails fast with an accurate error instead of hanging.

Separate, not fixed here: the CLI itself going silent for 16 minutes while subagents run. That needs stalled-turn detection (nothing in the server detects a running turn with no events).

[nohat]

Fix deployed to production in release 14a1420da5 (branch fix/send-turn-control-stall, commit 68194b3): sendTurn now waits at most 3 s for the Claude CLI's answer to a control request (permission mode, model), logs claude.control_request.unanswered, and queues the message anyway. Leaving this open until it has run for a few days on the scaffold orchestrator thread without a recurrence. Still unbuilt: stalled-turn detection for a CLI that goes silent for minutes.

[nohat]

Triage 2026-10-03T23:46Z: real. The body's trace spans (~/.t3/userdata/logs/server.trace.ndjson.2) plus the code path in apps/server/src/provider/Layers/ClaudeAdapter.ts confirm it: sendTurn awaited setModel/setPermissionMode with no bound, so a silent CLI held the message until Stop closed the query.

Fix 68194b330a (fix/send-turn-control-stall) merged to fork/prod and deployed: release 83376e22e0, local.t3code.prod running, /.well-known/t3/environment 200 on 13774 (server 0.0.44). Regression test added in ClaudeAdapter.test.ts.

Status: fixed, awaiting soak (original symptom never reproduced on demand, so this stays open until a soak window passes with no recurrence).