fix(server): a stuck provider call no longer freezes its thread until restart

[shivamhwp]

If a provider call never answers, for example a session open that hangs, the v2 run stays "starting" forever. Worse, V2 runs only one outbox effect per thread at a time and never reclaims a running row, so that thread's interrupt, detach and cleanup effects all queue behind the hung one until the server restarts. The 30-second lease on each row is written but never checked.

I reproduced it in a real client with a fake provider that never answers ACP session/new: the run stayed starting and its provider-turn.start effect stayed running indefinitely.

What changes:

• Starting a run is bounded. Each provider call made while a run starts (session open, thread load and resume, and handing over the turn) now has a 5-minute bound. A hang fails the run with a provider_error item through the existing start-failure path, the same one fix(server): a turn that fails to start after its session opens now fails the run #14183 uses, so the user sees a failed run instead of a spinner. The slowest legitimate provider calls in the codebase are capped at 180 seconds.
• Every outbox effect has a backstop. Every effect gets a 15-minute backstop that goes through the worker's normal retry and fail path, so a hung interrupt, detach, cleanup or rollback can't hold the lane forever. The handler runs in its own fiber, so the backstop settles the claim even when a handler is stuck somewhere it can't be interrupted, such as an ACP response waiting for its native acknowledgement. A cancellation still waits for the handler to stop. Nothing reclaims a still-running row, which keeps the decision recorded in FoundationPersistence.test.ts.
• An interrupted session open closes its scope. Before, a timed-out or cancelled open left the spawned provider process behind. The close uses the same 30-second bound as a session release, so a provider that never lets go of its message stream can't turn the timeout itself into a new hang.

A turn start doesn't cover the whole turn: RunExecutionService.startRootRun forks event ingestion and returns once the provider accepts the turn. I checked every adapter (Codex, Claude, the ACP adapters, Cursor, OpenCode, Pi), and each returns on acceptance, so these bounds can't cut off a long turn. The one exception is OpenCode's /compact, which waits for session.summarize to finish, so compaction is left out of the start bound and keeps only the 15-minute backstop.

Tests (TestClock, no sleeps) cover a hung open, thread load and startTurn each failing the run visibly, and a hung start freeing the lane so a later terminal.cleanup runs through the real outbox and worker. They also cover a hung non-start effect timing out, including one that can't be interrupted, a cancellation still stopping the running handler, unchanged retries up to maxAttempts, a start under the bound being unaffected, an interrupted open closing its scope, and an interrupted open whose scope close hangs still finishing. All seven new timeout tests fail with the bounds disabled.

Decision for reviewers: a start timeout fails the run on the first attempt instead of retrying, which trades retries for a visible failure after 5 minutes.

Works with #14201, which lets users press Stop on a run that's still starting.

Model: Claude Sonnet 5, Claude Fable 5.1 and Claude Opus 5.5 (Claude Code)

🤖 Generated with Claude Code

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This production change alters existing orchestration timeout, retry, provider fallback, session cleanup, credential handling, and thread-lane behavior across multiple components, with timed-out work continuing in detached fibers. An unresolved high-severity concern also questions whether a non-interruptible ACP handler can still prevent claim settlement, so the lifecycle behavior merits human review.

No code changes detected at 887ad60. Prior analysis still applies.

^{Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more.}

[github-actions]

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	—	4.9 KiB	—	6.8 KiB	✅
Codex	Thread snapshot wire	—	3.7 KiB	—	4.9 KiB	✅
Codex	Live turn WebSocket wire	—	1.2 KiB	—	2.0 KiB	✅
Codex	Live turn WebSocket decoded	—	20.4 KiB	—	29.3 KiB	✅
Codex	Live turn messages	—	2	—	8	✅
Claude	Total thread wire	—	4.9 KiB	—	6.8 KiB	✅
Claude	Thread snapshot wire	—	3.7 KiB	—	4.9 KiB	✅
Claude	Live turn WebSocket wire	—	1.2 KiB	—	2.0 KiB	✅
Claude	Live turn WebSocket decoded	—	20.8 KiB	—	29.3 KiB	✅
Claude	Live turn messages	—	2	—	8	✅

Baseline: unavailable · PR result: 887ad60 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

• Codex decoded thread snapshot: 106.1 KiB
• Claude decoded thread snapshot: 106.4 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

[juliusmarminge]

Note

This comment is posted by Julius' dot

The hang and TestClock regressions are well described, but this introduces a five-minute startup deadline, failure on the first timeout, and a fifteen-minute backstop for every outbox effect. The maintainer direction in #13392 and #14183 covers settling terminal startup failures; it does not establish those broader timeout/retry choices. This is a substantial lifecycle-policy change outside the small obvious-bug exception in prior approval. Please get maintainer agreement on the deadlines, retry behavior, and all-effect scope before resubmitting; closing this submission while preserving the reproduced hung-provider problem.