Skip to content

[Bug]: Stopping a coordinator thread walks its whole delegated-task tree (no status filter or de-dupe), blocking the next turn for minutes per interrupt #20342

Closed

Description

@samdickson22

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. Use a long-lived coordinator thread that has delegated many tasks over time, some of which delegated further (in my case: 307 app-owned subagent rows on the coordinator, 688 distinct threads in the whole task tree).
  2. While a run is active, press Stop (or send a message that interrupts the run with the queue held).
  3. Send a follow-up message.
  4. Optionally interrupt or resend a couple more times when nothing happens.

Expected behavior

Stop finishes in seconds and the follow-up turn starts. Already-finished delegated tasks are skipped, and each child thread is visited once.

Actual behavior

The follow-up run sits in starting and the UI shows "Run interrupted · Run interrupted before provider start" and "Context handoff pending". Each further interrupt makes the wait longer.

Cause, from the server state DB and source:

  1. A run.interrupt with holdQueue enqueues a delegated-tasks.stop effect for the thread.
  2. stopDelegatedTasks (apps/server/src/orchestration-v2/ThreadManagementService.ts, around line 1093 on main @ e58a485) iterates every app-owned subagent row of the thread, dispatches thread.stop to its child, then recurses into that child. It has no status filter (completed/failed/cancelled tasks are walked too) and no visited set. Duplicate subagent rows pointing at the same child thread (19 rows → 3 child threads in my case) cause whole subtrees to be walked again for each row.
  3. For my coordinator one stop expands to ~352,000 sequential thread.stop dispatches + getThreadRecords calls (computed from orchestration_v2_projection_subagents, following the same traversal). Each delegated-tasks.stop effect took 3.5–5.5 minutes, with the server process pinned at ~93% of one core.
  4. claimableCandidatePredicate in EffectOutbox.ts only lets a thread's effect be claimed when no other non-title effect on the same thread is running or older and pending. So the follow-up provider-turn.start waits behind every queued stop.
  5. Every new interrupt cancels the waiting provider-turn.start ("was interrupted before its provider turn started") and adds another delegated-tasks.stop. Users naturally press Stop/resend when nothing happens, which piles them up. I had 1 running + 3 pending stops ahead of the new run. The three child leads had all been stopped minutes earlier, so the remaining work was entirely redundant.

Outbox for the thread at the time (trimmed):

delegated-tasks.stop  running  attempt 2  claimed 17:26:20, lease_expires 17:26:50, still running at 17:31:53
delegated-tasks.stop  pending
delegated-tasks.stop  pending
delegated-tasks.stop  pending
provider-turn.start   pending  (run 2311, requested 17:27:48)

Possible fixes:

  • Skip subagents whose task/child thread is already terminal, and keep a visited set of child thread ids per stop.
  • Coalesce delegated-tasks.stop: if one is already pending/running for the thread, don't enqueue another.
  • Don't block provider-turn.start on delegated-tasks.stop (it doesn't need ordering with the new turn), or run the stop sweep in the background.
  • Renew the lease while the stop is running. The running effect's lease expired after 30 s and was neither renewed nor reclaimed.

Impact

Blocks work completely

Version or commit

0.1.1-nightly.20261016.3203 (also seen on 0.0.46-nightly.20261015.3171); source references from main @ e58a485

Environment

Headless t3 serve on Linux x64 (Ubuntu), desktop client on macOS connected through T3 Connect. Coordinator on Claude (claude-opus-5-5), children mixed Claude/Codex. State DB statev2.sqlite is ~8 GB.

Logs or stack traces

# effect_outbox rows for the coordinator thread, showing each interrupt cancelling the waiting start and queueing another stop
provider-turn.start   cancelled  'Run …:ordinal:2303 was interrupted before its provider turn started.'
provider-turn.start   cancelled  'Run …:ordinal:2304 was interrupted before its provider turn started.'
delegated-tasks.stop  succeeded  created 17:12:44  completed 17:18:04
delegated-tasks.stop  succeeded  created 17:12:47  completed 17:21:44
delegated-tasks.stop  succeeded  created 17:16:59  completed 17:25:26
provider-turn.start   cancelled  'Run …:ordinal:2306 was interrupted before its provider turn started.'
provider-turn.start   cancelled  'Run …:ordinal:2307 was interrupted before its provider turn started.'
provider-turn.start   cancelled  'Cancelled because the server shut down before the provider work completed.'
provider-turn.start   cancelled  'Run …:ordinal:2310 was interrupted before its provider turn started.'

The stop sweep itself logs nothing while it runs, so from the UI and logs it just looks hung.

Workaround

Don't press Stop or resend; wait for every queued delegated-tasks.stop to drain (≈4 min each). Alternatively, mark the redundant pending/running delegated-tasks.stop rows for the thread as cancelled in orchestration_v2_effect_outbox.

Activity

  1. juliusmarminge commented on Oct 16, 2026

    @juliusmarminge
    Member

    Note

    Grok responding on behalf of Julius.

    Tracking this under open #18826, which de-dupes stopDelegatedTasks so each child thread (and its subtree) is visited once when follow-up rows reuse the same childThreadId. That matches the combinatorial walk you measured. Leaving this open until that PR merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions