Repository navigation
fix(server): background jobs stay stoppable after a turn fails on a usage limit - #193
Merged
Merged
Conversation
…a usage limit Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…sage limit When the Claude CLI reports a failed turn, such as a usage limit, its process and background shells keep running. The adapter dropped the shell list at that terminal, and the run stopped reading provider events after any non-completed turn, so the shells disappeared from the thread and Stop could not clear them. The adapter now drops the list only when the turn was interrupted or the CLI process ended. After a failed or interrupted turn, the run keeps reading provider events while the adapter's per-thread background probe reports work, so a shell that ends later, or Stop's clear, reaches the thread. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Thread transfer impact✅ Thread transfer remains within every enforced ceiling.
Baseline: Scenario and decoded snapshot size10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed. |
…or its session is released A shell kept listed after a usage limit stayed listed when the CLI ended with no turn running, and when an account switch released the session. Both kill the shell, so the list showed a dead shell, kept Stop up, held the session and its run's event reader open, and kept a cloud machine busy. - The Claude adapter clears a native thread's shells and wake state when its live CLI ends, unless openQuery closed it to open a replacement. - Releasing a session, except at server shutdown, clears the shell lists of the provider threads it owned. Subscribers are failed before the scope closes, so the adapter's own clear can no longer reach the thread there. - The new limit test file provides FileSystem, Path and IdAllocator to the adapter registry layer, and its waits fail after 10 seconds. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… a replacement's shell list - A Claude CLI that exits keeps its buffered wake output, so a continuation queued behind a stopped user turn still delivers the background result. The exit reset also skips a process a queued turn opened meanwhile. - Stop clears a shell list the thread still shows when the live session reports no work for that thread, so a list the run stopped reading before its clear landed no longer sticks. - Session release records its session status and shell clears under the session's open lock, so a session reopened with the same id while the old process was closing keeps its own state. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…sult The CLI begins a wake turn (task_notification, init, assistant output), a continuation is requested, and the CLI exits before the turn's result. The continuation opens a fresh CLI that was never prompted, so on 97b3b0f it waits for output that never comes and times out. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…turn exited The CLI exit handler keeps buffered wake output so a continuation queued behind a stopped user turn still delivers it. When that output began a native wake turn (init or assistant) and the CLI then died before its result, the continuation replayed it and waited for more output from a fresh CLI that was never prompted, so the turn ran until the user pressed Stop. The exit handler now marks the thread's wake buffer as belonging to an ended process. A drain of such a buffer replays it and, with no result in it, finishes the continuation as completed. Frames a live process buffers later clear the mark. A replaced process does not mark the buffer, as before. Session release also retries its records write in the background when the caller is interrupted while it waits for the session's open lock. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
andrewcai8
enabled auto-merge (squash)
October 7, 2026 14:26
andrewcai8
added a commit
that referenced
this pull request
Oct 7, 2026
Brings in #193 and #194, which landed on main after the sync branched. - ProviderSessionManager.ts: upstream's release-on-interrupt for a failed attach, with #193's holdsOpenLock on both branches, since both run under the session's open lock. - ProviderSessionManager.test.ts: both sides' test options and sinks. - mcp/toolkits/home/handlers.ts: #194's t3_fork_cancel, declared through McpToolAccess like t3_fork_run. - #193's new Claude adapter tests pass the Crypto service upstream's makeClaudeAdapterV2 now needs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This was referenced Oct 13, 2026
andrewcai8
added a commit
that referenced
this pull request
Oct 19, 2026
…ssion is live Upstream now reconciles the stopped turn's own provider thread through stillReportsBackgroundWork, which covers the failed-turn case #193 fixed. For every other provider thread a live session owns its roster again, as upstream's BackgroundWorkStop tests expect, so the fork's probe-based liveSessionRunsWork goes away. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A Claude chat that hits its usage limit while it runs background shells (
run_in_backgroundBash) loses track of them. The CLI process and its shells keep running. The thread no longer lists them. Stop can't reach them either: the banner flips to "Stopping..." and back to "Stop". In practice only an account switch, which releases the session, ended them.Root cause
T3 stops tracking the shells in two places once the CLI reports a failed turn:
apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts(finalizeActiveTurn). Every non-completed terminal calledclearPendingBackgroundTasksForNativeThread. The roster written before the terminal was therefore empty, even though the process still runs the shells.apps/server/src/orchestration-v2/RunExecutionService.ts(shouldStopProviderEventIngestion). Any non-completed terminal returnedtruestraight away, so the run stopped reading provider events. Keeping the roster in the adapter alone is not enough. When a shell ends later, or Stop closes the CLI, the adapter emits a roster clear, and nothing ingests it. The banner then stays up for good.Fix
resultleaves the roster in place. The roster is still dropped when the turn was interrupted (T3 closes the CLI) or when the CLI process ended (noresult).hasPendingBackgroundWorkForThread) reports work. This is the same probe that already holds completed runs open. The adapter clears it when the work ends, so a late roster clear reaches the thread. Rootprovider_thread.updatedwrites after the terminal are still owner-gated (writeIfProviderThreadOwner).openQueryclosed the CLI to open a replacement, or when a queued turn already opened a new CLI. Both reset that state themselves. Before, a CLI that ended between turns left its dead shells listed, kept the session from idle-releasing, and kept the run's event reader open.hasPendingBackgroundWorkForThread). A session without that probe still owns its list, as before.Per-adapter decision
The ingestion rule applies to every adapter that implements
hasPendingBackgroundWorkForThread:settledTurns, and its running items are terminalized beforeturn.terminal, so the probe reads false right after its own failed terminal. It reads true only when an earlier completed turn on the same thread still has retained work. That already holds a completed later run's stream open today.The release-time clear applies to every provider. A released session's process is gone for all of them, so no adapter can report its roster ended afterwards.
Accepted tradeoffs. An OpenCode2 session on an external server, and a release whose scope close times out, now hide background work whose process may still run. T3 can't stop that work in either case.
Tests
apps/server/src/orchestration-v2/ClaudeBackgroundWorkAfterLimit.test.tsruns the real orchestrator and Claude adapter against a fake CLI. In it, a turn starts a background shell (background_tasks_changed) and then fails onblocking_limit.run.interruptwithholdQueue, as the banner sends it) closes the CLI and clears the list.ClaudeAdapterV2.test.ts"keeps the roster after a failed turn until the CLI exits" checks that the roster survives the failure, and that once the CLI exits with no turn running, the adapter emits an empty idle roster and both pending-work probes read false. The reader stops on that probe. This fails without the adapter change.Review follow-ups, each failing before its fix:
ClaudeAdapterV2.test.ts"delivers a buffered wake after Stop ends the user turn queued ahead of it". Before, the continuation settled with an empty buffer and never showed the wake result.ClaudeBackgroundWorkAfterLimit.test.ts"Stop clears a shell list the thread kept after missing the shell's end". Before, the list stayed after Stop.ProviderSessionManager.test.ts"leaves a replacement's background work to it when a slow release records". Before, the old release wiped the replacement's list. "releases a session whose attachment fails to persist" guards the open path, which releases while it holds the lock. Without theholdsOpenLockflag it deadlocks. A release whose caller is interrupted while it waits for that lock hands its records write to the existing background retry (no new test; it reusesretryReleaseRecords).ClaudeAdapterV2.test.ts"settles a wake continuation whose CLI died before the wake turn's result". Before, the continuation waited for output from a fresh CLI and timed out.The first two orchestrator tests fail on main: the thread lists
[]right after the limit. Without the ingestion change they fail on a 10 second wait, because the roster clear never reaches the thread. Targeted suites pass: RunExecutionService, ProviderSessionManager, the Claude, Codex and OpenCode2 adapters, BackgroundWorkStop, OpenCode2 orchestrator, ThreadStop, the Claude, Codex and orchestrator replay fixtures, ProviderAccountSwitch, ProviderSwitch, and runtime recovery.Claude Opus 5.5 (1M context), Claude Code
🤖 Generated with Claude Code