Repository navigation
[Bug]: Stopping a coordinator thread walks its whole delegated-task tree (no status filter or de-dupe), blocking the next turn for minutes per interrupt #20342
Description
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
- Use a long-lived coordinator thread that has delegated many tasks over time, some of which delegated further (in my case: 307 app-owned subagent rows on the coordinator, 688 distinct threads in the whole task tree).
- While a run is active, press Stop (or send a message that interrupts the run with the queue held).
- Send a follow-up message.
- Optionally interrupt or resend a couple more times when nothing happens.
Expected behavior
Stop finishes in seconds and the follow-up turn starts. Already-finished delegated tasks are skipped, and each child thread is visited once.
Actual behavior
The follow-up run sits in starting and the UI shows "Run interrupted · Run interrupted before provider start" and "Context handoff pending". Each further interrupt makes the wait longer.
Cause, from the server state DB and source:
- A
run.interruptwithholdQueueenqueues adelegated-tasks.stopeffect for the thread. stopDelegatedTasks(apps/server/src/orchestration-v2/ThreadManagementService.ts, around line 1093 on main @ e58a485) iterates every app-owned subagent row of the thread, dispatchesthread.stopto its child, then recurses into that child. It has no status filter (completed/failed/cancelled tasks are walked too) and no visited set. Duplicate subagent rows pointing at the same child thread (19 rows → 3 child threads in my case) cause whole subtrees to be walked again for each row.- For my coordinator one stop expands to ~352,000 sequential
thread.stopdispatches +getThreadRecordscalls (computed fromorchestration_v2_projection_subagents, following the same traversal). Eachdelegated-tasks.stopeffect took 3.5–5.5 minutes, with the server process pinned at ~93% of one core. claimableCandidatePredicateinEffectOutbox.tsonly lets a thread's effect be claimed when no other non-title effect on the same thread isrunningor older andpending. So the follow-upprovider-turn.startwaits behind every queued stop.- Every new interrupt cancels the waiting
provider-turn.start("was interrupted before its provider turn started") and adds anotherdelegated-tasks.stop. Users naturally press Stop/resend when nothing happens, which piles them up. I had 1 running + 3 pending stops ahead of the new run. The three child leads had all been stopped minutes earlier, so the remaining work was entirely redundant.
Outbox for the thread at the time (trimmed):
delegated-tasks.stop running attempt 2 claimed 17:26:20, lease_expires 17:26:50, still running at 17:31:53
delegated-tasks.stop pending
delegated-tasks.stop pending
delegated-tasks.stop pending
provider-turn.start pending (run 2311, requested 17:27:48)
Possible fixes:
- Skip subagents whose task/child thread is already terminal, and keep a visited set of child thread ids per stop.
- Coalesce
delegated-tasks.stop: if one is already pending/running for the thread, don't enqueue another. - Don't block
provider-turn.startondelegated-tasks.stop(it doesn't need ordering with the new turn), or run the stop sweep in the background. - Renew the lease while the stop is running. The running effect's lease expired after 30 s and was neither renewed nor reclaimed.
Impact
Blocks work completely
Version or commit
0.1.1-nightly.20261016.3203 (also seen on 0.0.46-nightly.20261015.3171); source references from main @ e58a485
Environment
Headless t3 serve on Linux x64 (Ubuntu), desktop client on macOS connected through T3 Connect. Coordinator on Claude (claude-opus-5-5), children mixed Claude/Codex. State DB statev2.sqlite is ~8 GB.
Logs or stack traces
# effect_outbox rows for the coordinator thread, showing each interrupt cancelling the waiting start and queueing another stop
provider-turn.start cancelled 'Run …:ordinal:2303 was interrupted before its provider turn started.'
provider-turn.start cancelled 'Run …:ordinal:2304 was interrupted before its provider turn started.'
delegated-tasks.stop succeeded created 17:12:44 completed 17:18:04
delegated-tasks.stop succeeded created 17:12:47 completed 17:21:44
delegated-tasks.stop succeeded created 17:16:59 completed 17:25:26
provider-turn.start cancelled 'Run …:ordinal:2306 was interrupted before its provider turn started.'
provider-turn.start cancelled 'Run …:ordinal:2307 was interrupted before its provider turn started.'
provider-turn.start cancelled 'Cancelled because the server shut down before the provider work completed.'
provider-turn.start cancelled 'Run …:ordinal:2310 was interrupted before its provider turn started.'The stop sweep itself logs nothing while it runs, so from the UI and logs it just looks hung.
Workaround
Don't press Stop or resend; wait for every queued delegated-tasks.stop to drain (≈4 min each). Alternatively, mark the redundant pending/running delegated-tasks.stop rows for the thread as cancelled in orchestration_v2_effect_outbox.
Activity
juliusmarminge commentedon Oct 16, 2026
Note
Grok responding on behalf of Julius.
Tracking this under open #18826, which de-dupes
stopDelegatedTasksso each child thread (and its subtree) is visited once when follow-up rows reuse the samechildThreadId. That matches the combinatorial walk you measured. Leaving this open until that PR merges.