Skip to content

fix(server): background jobs stay stoppable after a turn fails on a usage limit - #193

Merged
andrewcai8 merged 6 commits into
mainfrom
fix/background-jobs-stoppable-after-failed-turn
Oct 7, 2026
Merged

andrewcai8 merged 6 commits into
mainfrom
fix/background-jobs-stoppable-after-failed-turn

Conversation

@andrewcai8

@andrewcai8 andrewcai8 commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

Problem

A Claude chat that hits its usage limit while it runs background shells (run_in_background Bash) loses track of them. The CLI process and its shells keep running. The thread no longer lists them. Stop can't reach them either: the banner flips to "Stopping..." and back to "Stop". In practice only an account switch, which releases the session, ended them.

Root cause

T3 stops tracking the shells in two places once the CLI reports a failed turn:

  • apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts (finalizeActiveTurn). Every non-completed terminal called clearPendingBackgroundTasksForNativeThread. The roster written before the terminal was therefore empty, even though the process still runs the shells.
  • apps/server/src/orchestration-v2/RunExecutionService.ts (shouldStopProviderEventIngestion). Any non-completed terminal returned true straight away, so the run stopped reading provider events. Keeping the roster in the adapter alone is not enough. When a shell ends later, or Stop closes the CLI, the adapter emits a roster clear, and nothing ingests it. The banner then stays up for good.

Fix

  • Adapter. A failure the CLI reports in its own result leaves the roster in place. The roster is still dropped when the turn was interrupted (T3 closes the CLI) or when the CLI process ended (no result).
  • Ingestion. A failed or interrupted turn still drops child-turn, child-subagent and background-item tracking, the same as before. It now keeps reading only while the adapter's per-thread probe (hasPendingBackgroundWorkForThread) reports work. This is the same probe that already holds completed runs open. The adapter clears it when the work ends, so a late roster clear reaches the thread. Root provider_thread.updated writes after the terminal are still owner-gated (writeIfProviderThreadOwner).
  • The CLI ends while no turn runs. The adapter clears that native thread's roster and wake eligibility when its live CLI ends, and emits the empty roster. Buffered wake output stays, so a continuation queued behind a stopped or crashed user turn still delivers the background result. That output is marked as coming from an ended CLI. If it began a wake turn that never reached its result, the continuation replays it and then completes, instead of waiting on a fresh CLI that was never prompted. It skips the clear when openQuery closed the CLI to open a replacement, or when a queued turn already opened a new CLI. Both reset that state themselves. Before, a CLI that ended between turns left its dead shells listed, kept the session from idle-releasing, and kept the run's event reader open.
  • The session is released. Releasing a session (account switch, idle timeout, runtime error) now clears the shell lists of the provider threads it owned. The session manager fails its event subscribers before it closes the adapter, so the adapter's own clear cannot reach the thread there. Server shutdown is excluded, because startup recovery settles those rosters and records the cancelled work. The release writes its records under the session's open lock, so a session reopened with the same id while the old process still closes keeps its own status and list. Before, an account switch after a limit left the dead shell listed, which also kept a cloud machine "busy", until the user pressed Stop.
  • Stop clears a stale list. A run can stop reading just before the adapter's roster clear lands, which left the old list on the thread. Stop now clears a provider thread's list when its live session reports no pending work for that thread (hasPendingBackgroundWorkForThread). A session without that probe still owns its list, as before.
  • Queue holding is unchanged. Run finalization still happens at the terminal, so queued runs and Stop's hold behave as before. Wake eligibility is still cleared on failure, so a shell that ends after a limit clears the banner without waking the agent into the limit.

Per-adapter decision

The ingestion rule applies to every adapter that implements hasPendingBackgroundWorkForThread:

  • Claude. Fixed by the changes above.
  • Codex. No change in practice. A failed or interrupted turn is never kept in settledTurns, and its running items are terminalized before turn.terminal, so the probe reads false right after its own failed terminal. It reads true only when an earlier completed turn on the same thread still has retained work. That already holds a completed later run's stream open today.
  • OpenCode2. A failed turn settles the background calls it started. An interrupted turn keeps background subagent calls on purpose, so its run now keeps reading until they report. Before this change, their late completions were dropped. No test covers this yet, since it needs a new replay transcript with an aborted parent execution.
  • Cursor, Grok, ACP, Pi, OpenCode v1. No probe, so the behavior is unchanged: a non-completed terminal still stops ingestion.

The release-time clear applies to every provider. A released session's process is gone for all of them, so no adapter can report its roster ended afterwards.

Accepted tradeoffs. An OpenCode2 session on an external server, and a release whose scope close times out, now hide background work whose process may still run. T3 can't stop that work in either case.

Tests

apps/server/src/orchestration-v2/ClaudeBackgroundWorkAfterLimit.test.ts runs the real orchestrator and Claude adapter against a fake CLI. In it, a turn starts a background shell (background_tasks_changed) and then fails on blocking_limit.

  • The shell is still listed on the thread shell. Stop (run.interrupt with holdQueue, as the banner sends it) closes the CLI and clears the list.
  • When the shell ends on its own after the limit, the list clears.
  • When an account switch releases the session, the CLI closes, the session is gone, and the list clears. This fails without the release-time clear.

ClaudeAdapterV2.test.ts "keeps the roster after a failed turn until the CLI exits" checks that the roster survives the failure, and that once the CLI exits with no turn running, the adapter emits an empty idle roster and both pending-work probes read false. The reader stops on that probe. This fails without the adapter change.

Review follow-ups, each failing before its fix:

  • ClaudeAdapterV2.test.ts "delivers a buffered wake after Stop ends the user turn queued ahead of it". Before, the continuation settled with an empty buffer and never showed the wake result.
  • ClaudeBackgroundWorkAfterLimit.test.ts "Stop clears a shell list the thread kept after missing the shell's end". Before, the list stayed after Stop.
  • ProviderSessionManager.test.ts "leaves a replacement's background work to it when a slow release records". Before, the old release wiped the replacement's list. "releases a session whose attachment fails to persist" guards the open path, which releases while it holds the lock. Without the holdsOpenLock flag it deadlocks. A release whose caller is interrupted while it waits for that lock hands its records write to the existing background retry (no new test; it reuses retryReleaseRecords).
  • ClaudeAdapterV2.test.ts "settles a wake continuation whose CLI died before the wake turn's result". Before, the continuation waited for output from a fresh CLI and timed out.

The first two orchestrator tests fail on main: the thread lists [] right after the limit. Without the ingestion change they fail on a 10 second wait, because the roster clear never reaches the thread. Targeted suites pass: RunExecutionService, ProviderSessionManager, the Claude, Codex and OpenCode2 adapters, BackgroundWorkStop, OpenCode2 orchestrator, ThreadStop, the Claude, Codex and orchestrator replay fixtures, ProviderAccountSwitch, ProviderSwitch, and runtime recovery.

Claude Opus 5.5 (1M context), Claude Code

🤖 Generated with Claude Code

andrewcai8 and others added 2 commits October 7, 2026 13:55
…a usage limit

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…sage limit

When the Claude CLI reports a failed turn, such as a usage limit, its process
and background shells keep running. The adapter dropped the shell list at that
terminal, and the run stopped reading provider events after any non-completed
turn, so the shells disappeared from the thread and Stop could not clear them.

The adapter now drops the list only when the turn was interrupted or the CLI
process ended. After a failed or interrupted turn, the run keeps reading
provider events while the adapter's per-thread background probe reports work,
so a shell that ends later, or Stop's clear, reaches the thread.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:S labels Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 20.9 KiB 20.9 KiB 0 B (0.0%) 29.3 KiB ✅
Codex Live turn messages 2 2 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB −24 B (−0.5%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB −24 B (−2.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.2 KiB 21.2 KiB −41 B (−0.2%) 29.3 KiB ✅
Claude Live turn messages 2 1 −1 (−50.0%) 8 ✅

Baseline: 57cd3ae · PR result: 97b3b0f · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

…or its session is released

A shell kept listed after a usage limit stayed listed when the CLI ended with
no turn running, and when an account switch released the session. Both kill
the shell, so the list showed a dead shell, kept Stop up, held the session
and its run's event reader open, and kept a cloud machine busy.

- The Claude adapter clears a native thread's shells and wake state when its
  live CLI ends, unless openQuery closed it to open a replacement.
- Releasing a session, except at server shutdown, clears the shell lists of
  the provider threads it owned. Subscribers are failed before the scope
  closes, so the adapter's own clear can no longer reach the thread there.
- The new limit test file provides FileSystem, Path and IdAllocator to the
  adapter registry layer, and its waits fail after 10 seconds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added size:M and removed size:S labels Oct 7, 2026
… a replacement's shell list

- A Claude CLI that exits keeps its buffered wake output, so a continuation
  queued behind a stopped user turn still delivers the background result.
  The exit reset also skips a process a queued turn opened meanwhile.
- Stop clears a shell list the thread still shows when the live session
  reports no work for that thread, so a list the run stopped reading
  before its clear landed no longer sticks.
- Session release records its session status and shell clears under the
  session's open lock, so a session reopened with the same id while the
  old process was closing keeps its own state.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added size:L and removed size:M labels Oct 7, 2026
andrewcai8 and others added 2 commits October 7, 2026 15:25
…sult

The CLI begins a wake turn (task_notification, init, assistant output),
a continuation is requested, and the CLI exits before the turn's result.
The continuation opens a fresh CLI that was never prompted, so on
97b3b0f it waits for output that never comes and times out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…turn exited

The CLI exit handler keeps buffered wake output so a continuation queued
behind a stopped user turn still delivers it. When that output began a
native wake turn (init or assistant) and the CLI then died before its
result, the continuation replayed it and waited for more output from a
fresh CLI that was never prompted, so the turn ran until the user pressed
Stop.

The exit handler now marks the thread's wake buffer as belonging to an
ended process. A drain of such a buffer replays it and, with no result in
it, finishes the continuation as completed. Frames a live process buffers
later clear the mark. A replaced process does not mark the buffer, as
before.

Session release also retries its records write in the background when the
caller is interrupted while it waits for the session's open lock.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@andrewcai8
andrewcai8 enabled auto-merge (squash) October 7, 2026 14:26
@andrewcai8
andrewcai8 merged commit bc068c7 into main Oct 7, 2026
25 of 26 checks passed
andrewcai8 added a commit that referenced this pull request Oct 7, 2026
Brings in #193 and #194, which landed on main after the sync branched.

- ProviderSessionManager.ts: upstream's release-on-interrupt for a failed
  attach, with #193's holdsOpenLock on both branches, since both run
  under the session's open lock.
- ProviderSessionManager.test.ts: both sides' test options and sinks.
- mcp/toolkits/home/handlers.ts: #194's t3_fork_cancel, declared through
  McpToolAccess like t3_fork_run.
- #193's new Claude adapter tests pass the Crypto service upstream's
  makeClaudeAdapterV2 now needs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
andrewcai8 added a commit that referenced this pull request Oct 19, 2026
…ssion is live

Upstream now reconciles the stopped turn's own provider thread through
stillReportsBackgroundWork, which covers the failed-turn case #193 fixed.
For every other provider thread a live session owns its roster again, as
upstream's BackgroundWorkStop tests expect, so the fork's probe-based
liveSessionRunsWork goes away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant