Skip to content

feat(runner): detect stalled agent runs via event inactivity - #6595

Open
guyoron1 wants to merge 21 commits into
fullsend-ai:mainfrom
guyoron1:feat/runner-stall-watchdog
Open

guyoron1 wants to merge 21 commits into
fullsend-ai:mainfrom
guyoron1:feat/runner-stall-watchdog

Conversation

@guyoron1

@guyoron1 guyoron1 commented Aug 25, 2026 •

Copy link
Copy Markdown
Contributor

Heyaa : )

This one came out of watching a wedged run burn its entire global timeout while the heartbeat cheerfully printed "agent running" the whole time — the process was dead and nothing could tell.

The runtime now watches the runtime output stream instead: every well-formed line the stream parser reads is proof of life (including lines that map to no AgentEvent — pi's tool_execution_update, Claude Code's user tool results, codex's item.started/item.completed), half a timeout of silence warns once per stall episode (::warning:: in CI, printer everywhere), and a full timeout kills the run and fails it distinctly with ErrStalled + "stalled": true in metrics.json.

Config: FULLSEND_STALL_TIMEOUT (Go duration, default 15m, 0 disables), resolved by the CLI (runStallTimeout in internal/cli/run_overrides.go) and handed to the runtime in RunParams.StallTimeout — runtimes don't read env themselves (#6526). A malformed value is reported and the default applies. A non-zero value is floored at 1m (a pi sub-agent's 30s liveness tick), and at 6m on codex (one poll of a running command can be silent for 5 minutes); the run logs the raise. The value is allowlisted in setup-agent-env.sh, so a CI repo variable can set it. The watchdog is disarmed — and the run says so — when the stall timeout plus its detection interval (a twentieth of the timeout, capped at 30s) is not below the harness timeout_minutes, since the global deadline would always win: at the default that means harnesses at 15 minutes or less get no stall protection.

Coverage: claude, pi, codex and opencode — every runtime that drains a stream through sandbox.ExecStreamReader — with the same call shape at each call site (startStallWatchdog, a per-line stall.note hook, stall.stalledErr() ahead of the Wait error); the scripted dummy runtimes ignore the field. On claude and pi the watchdog arms per model-fallback attempt, and a stall is final, never retried on the next model. A source-shape test finds the streaming runtimes from their ExecStreamReader call sites and pins all four.

What still reads as silence: codex writes nothing while a command runs (its polls of a running command produce no JSONL line), opencode writes a tool line only when the call ends, and pi's bash tool writes progress only when output arrives. On those runtimes a quiet command longer than the stall timeout is killed as stalled. The 15m default clears Claude Code's bash ceiling only; it is not yet backed by measured inter-line gaps from real runs.

Design points:

  • The kill reaches the agent. The cancel from sandbox.ExecStreamReader only SIGKILLs the local openshell sandbox exec client; OpenShell has no signal API. So the kill first runs the same TERM-then-KILL stray-process sweep that clears the sandbox between iterations, over a second exec channel, then cancels — the wedged agent stops writing the workspace and spending tokens, under --keep-sandbox too. ErrStalled is reported only once that sweep has finished (bounded by its exec timeout), so its result line prints before the failure and the extracted artifacts are final. The sweep is not serialized against the credential refreshers (a kill path must not wait on a lock a wedged upload may hold); the only file it can truncate is the five-minute OIDC token, and the docs tell operators to re-seed before reusing a kept sandbox.
  • Default is 15m, not Cloudflare-style 60s: it must sit above Claude Code's BASH_MAX_TIMEOUT_MS ceiling (600000ms), which the model routinely requests for test suites. Repos that raise the bash ceiling, or run long quiet commands on pi, codex or opencode, raise the stall timeout with it.
  • pi sub-agents: a child's stdout stays inside the Agent tool, so the tool emits a progress update every 30s while its child (and any fallback attempt) runs; a long persona never reads as a stall, and a wedged child is bounded by its own timeoutSeconds (900s). On a stall, completed children's usage is folded into the metrics and the parent's own agent_end result is kept.
  • The watchdog derives no context and never rebinds the caller's ctx (a body-scope rebind once made every successful run report "cancelled"); the stop/fire race is closed with a CAS and silence is measured on the monotonic clock.

Tests: the watchdog matrix (flowing lines never kill / silence kills exactly once / 0 disables / warn-once-per-episode, rearmed by the next line / no annotations outside CI / no warning after disarm / ErrStalled waits for a slow sweep), resolveStallTimeout, effectiveStallTimeout and runStallTimeout cases (including the codex floor), the sweep-then-cancel order and its failure path, TestClaudeRuntime_Run_StallSweepsTheSandboxOnce driving the real Run path, the line hooks of all four parsers, the source-shape guard, pi's usage fold and parent result on a stall, stalled,omitempty marshalling, and the Agent tool's liveness updates (node --test). go build, go vet, and internal/runtime, internal/cli, internal/scaffold pass.

Non-goals: per-dimension timeouts (dimensions run inside one CLI process the runner can't see into), and no change to the heartbeat or the global timeout — the heartbeat's future is a separate discussion after merge.

@github-actions

github-actions Bot commented Aug 25, 2026 •

Copy link
Copy Markdown

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer with write access must add the ok-to-test label after the latest push.

See E2E testing guide for details.

@github-actions

Copy link
Copy Markdown

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

@github-actions

github-actions Bot commented Aug 25, 2026 •

Copy link
Copy Markdown

Site preview

Preview: https://0cf71631-site.fullsend-ai.workers.dev

Commit: 2a9dd18891c9bb3a0ee7e592a078db35c37c2cf7

@codecov

codecov Bot commented Aug 25, 2026 •

Copy link
Copy Markdown

@guyoron1
guyoron1 force-pushed the feat/runner-stall-watchdog branch from ffe01bf to 0339139 Compare September 1, 2026 11:53
@guyoron1
guyoron1 marked this pull request as ready for review September 1, 2026 11:54
@guyoron1
guyoron1 requested a review from a team September 1, 2026 11:54
@qodo-code-review

qodo-code-review Bot commented Sep 1, 2026 •

Copy link
Copy Markdown

PR Summary by Qodo

Detect stalled agent runs from event-stream inactivity

✨ Enhancement 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

Grey Divider

AI Description

• Detects stalled Claude and Pi runs from normalized event-stream inactivity.
• Warns midway, then cancels wedged sandbox commands through the existing termination path.
• Adds configurable timeouts, distinct errors, metrics, documentation, and regression coverage.
Diagram

graph TD
  CONFIG["Stall configuration"] --> PARAMS["RunParams"] --> RUNTIME["Claude / Pi"] --> EVENTS["Agent events"] --> WATCH{"Silence threshold?"}
  WATCH -- "Half timeout" --> WARN["Warn once"]
  WATCH -- "Full timeout" --> CANCEL["Sandbox cancel"] --> RESULT["Error and metrics"]
Loading
High-Level Assessment

The shared event-inactivity watchdog is the appropriate design because normalized events provide stronger liveness evidence than the wall-clock heartbeat, while reusing ExecStreamReader's cancellation avoids competing termination mechanisms. Runtime-specific watchdogs would duplicate behavior, and deriving another context would risk altering caller-owned lifecycle semantics.

Files changed (10) +462 / -2

Enhancement (5) +209 / -2
run.goPropagate stall settings and report stalled runs +18/-0

Propagate stall settings and report stalled runs

• Resolves the stall timeout, warns when configuration is invalid, and passes it through RunParams. Maps ErrStalled to a specific failure message and records stalled=true in aggregate metrics.

internal/cli/run.go

claude.goAttach inactivity watchdog to Claude event streaming +15/-0

Attach inactivity watchdog to Claude event streaming

• Starts the shared watchdog using the sandbox command's existing cancel function and resets it for every normalized event. Disarms it before process reaping and prioritizes ErrStalled over the resulting wait error.

internal/runtime/claude.go

pi_run.goAttach inactivity watchdog to Pi event streaming +14/-0

Attach inactivity watchdog to Pi event streaming

• Integrates the shared watchdog with Pi's normalized event handler and existing sandbox cancellation path. Stops monitoring when the stream closes and returns the distinct stall error before generic process errors.

internal/runtime/pi_run.go

runtime.goExpose stall timeout through RunParams +9/-2

Expose stall timeout through RunParams

• Adds StallTimeout to the runtime invocation contract so the CLI owns environment resolution and streaming runtimes receive a duration directly.

internal/runtime/runtime.go

stall.goImplement shared event-inactivity watchdog +153/-0

Implement shared event-inactivity watchdog

• Introduces ErrStalled and a concurrency-safe watchdog that warns once after half the configured silence window and cancels once at the full timeout. Supports CI annotations, printer warnings, event-based rearming, idempotent shutdown, and disabled operation.

internal/runtime/stall.go

Tests (2) +214 / -0
run_overrides_test.goTest stall timeout resolution and metric serialization +43/-0

Test stall timeout resolution and metric serialization

• Covers unset, valid, trimmed, disabled, malformed, bare-number, and negative timeout values. Verifies stalled is omitted when false and serialized when true.

internal/cli/run_overrides_test.go

stall_test.goCover watchdog timing, warnings, and cancellation semantics +171/-0

Cover watchdog timing, warnings, and cancellation semantics

• Tests active streams, silent-stream termination, disabled operation, warning rearming, CI annotation behavior, and single-fire cancellation. Adds a regression guard ensuring Claude and Pi reuse the sandbox cancel function without rebinding the caller context.

internal/runtime/stall_test.go

Documentation (2) +8 / -0
run.mdDocument stall watchdog behavior and metrics +7/-0

Document stall watchdog behavior and metrics

• Explains warning and termination thresholds, FULLSEND_STALL_TIMEOUT configuration, and the rationale for the 10-minute default. Documents the conditional stalled field in metrics.json.

docs/cli/run.md

runtimes.mdAdd stall timeout to runtime configuration matrix +1/-0

Add stall timeout to runtime configuration matrix

• Documents FULLSEND_STALL_TIMEOUT as an environment-only runtime setting, including its default and disable value.

docs/runtimes.md

Other (1) +31 / -0
run_overrides.goResolve configurable stall timeout +31/-0

Resolve configurable stall timeout

• Adds FULLSEND_STALL_TIMEOUT parsing with a 10-minute default and support for disabling via zero. Invalid or negative values return the default alongside a descriptive error.

internal/cli/run_overrides.go

@qodo-code-review

qodo-code-review Bot commented Sep 1, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (1) 📜 Skill insights (0)

Grey Divider


Action required

1. Large events trigger false stalls ✓ Resolved
Description
Both stream parsers discard JSON lines exceeding 1 MiB before invoking the new liveness hook. A
runtime producing valid oversized events as its only output can therefore remain active while the
watchdog observes silence and terminates the run as stalled.

2. Watchdog fires after disarm ✓ Resolved
Description
stop() and note() do not synchronize with the ticker's load-and-fire sequence, so a pending tick
can set fired and kill the command after the stream was disarmed or after a new event was
recorded. Claude and Pi then trust stalledErr() and can report a normally completed or active run
as stalled.


Remediation recommended

3. Valid durations are dropped ✓ Resolved
Description
Adding FULLSEND_STALL_TIMEOUT to the scaffold override list subjects it to an ASCII allowlist that
rejects valid Go durations such as +5m, 1µs, and 1μs. Those repository-variable overrides
never reach the CLI, so it silently resolves the timeout from another source or uses the 15-minute
default.

4. Events cannot suppress warnings ✓ Resolved
Description
The warning path locks only after calculating silence, while note() updates lastEvent without that
mutex. An event arriving between those operations ends the stall episode but can still be followed
by a misleading inactivity warning.

5. Warning survives watchdog disarm ✓ Resolved
Description
After stop() transitions the watchdog to stopped and closes w.stopped, a queued ticker event may
still win the select and emit a stall warning because the warning branch never verifies the armed
state. A normally completed run can therefore produce a misleading local warning or GitHub Actions
annotation after its event stream has ended.

View medium (3)
6. Wall clock skews silence ✓ Resolved
Description
The watchdog stores time.Now().UnixNano(), reconstructs it with time.Unix, and therefore
discards Go's monotonic timestamp before measuring silence. A host wall-clock adjustment can
prematurely kill an active run or postpone stall detection beyond the configured timeout.

7. Runtime guide not consulted ✓ Resolved

Resolving commit: 6169e74

Description
The PR changes the runtime.Runtime execution contract by adding RunParams.StallTimeout and
watchdog integration, but neither updates docs/contributing/runtime-implementation.md nor states
in the PR description that the guide was consulted. The new event-inactivity and cancellation
requirements are therefore undocumented for runtime implementers.

8. stallTimeout guard lacks tests 📘 Rule violation ▣ Testability
Description
The new branch that disables the watchdog when its timeout meets or exceeds the global run timeout
has no corresponding behavioral test. Regressions could silently leave runs unprotected or disable
valid watchdog configurations.
Code

internal/cli/run.go[1856]

+	if stallTimeout >= timeout {
Relevance

●● Moderate

Dedicated branch tests are often accepted, but recent coverage-only findings were also rejected.

PR-#5961
PR-#6753

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
PR Compliance ID 1062049 requires assertions covering new or modified Go logic. The newly added
comparison at internal/cli/run.go:1856 introduces a distinct deactivation path, but repository
tests do not exercise or assert this condition.

Rule 1062049: Require tests for new or modified Go logic
internal/cli/run.go[1856-1864]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Add behavioral tests for the new stall-timeout deactivation condition.

## Issue Context
Verify that a stall timeout equal to or greater than the run timeout is disabled, while a smaller positive timeout remains enabled. Assert the resulting runtime parameters or observable behavior rather than only the log text.

## Fix Focus Areas
- internal/cli/run.go[1852-1864]
- internal/cli/run_test.go[1-1]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


Grey Divider

Context sources
✅ Compliance rules (platform): 72 rules
Review mode: ⚖️ Balanced

Grey Divider

Tip of the day
💡 Did you know, you can reply 'qodo' on any finding to push back, ask questions, or dig deeper

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Previous reviews

Review updated until commit 2a9dd18

Results up to commit 0339139 ⚖️ Balanced


🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)


Action required
1. Watchdog fires after disarm ✓ Resolved
Description
stop() and note() do not synchronize with the ticker's load-and-fire sequence, so a pending tick
can set fired and kill the command after the stream was disarmed or after a new event was
recorded. Claude and Pi then trust stalledErr() and can report a normally completed or active run
as stalled.


Remediation recommended
2. Wall clock skews silence ✓ Resolved
Description
The watchdog stores time.Now().UnixNano(), reconstructs it with time.Unix, and therefore
discards Go's monotonic timestamp before measuring silence. A host wall-clock adjustment can
prematurely kill an active run or postpone stall detection beyond the configured timeout.

3. Runtime guide not consulted ✓ Resolved

Resolving commit: 6169e74

Description
The PR changes the runtime.Runtime execution contract by adding RunParams.StallTimeout and
watchdog integration, but neither updates docs/contributing/runtime-implementation.md nor states
in the PR description that the guide was consulted. The new event-inactivity and cancellation
requirements are therefore undocumented for runtime implementers.

Results up to commit 2b1e60b ⚖️ Balanced


🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)


Remediation recommended
1. Warning survives watchdog disarm ✓ Resolved
Description
After stop() transitions the watchdog to stopped and closes w.stopped, a queued ticker event may
still win the select and emit a stall warning because the warning branch never verifies the armed
state. A normally completed run can therefore produce a misleading local warning or GitHub Actions
annotation after its event stream has ended.

Results up to commit 957481a ⚖️ Balanced


🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)


Remediation recommended
1. Events cannot suppress warnings ✓ Resolved
Description
The warning path locks only after calculating silence, while note() updates lastEvent without that
mutex. An event arriving between those operations ends the stall episode but can still be followed
by a misleading inactivity warning.

Results up to commit 815582c ⚖️ Balanced


🐞 Bugs (0) 📘 Rule violations (1) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)


Action required
1. Large events trigger false stalls ✓ Resolved
Description
Both stream parsers discard JSON lines exceeding 1 MiB before invoking the new liveness hook. A
runtime producing valid oversized events as its only output can therefore remain active while the
watchdog observes silence and terminates the run as stalled.


Remediation recommended
2. Valid durations are dropped ✓ Resolved
Description
Adding FULLSEND_STALL_TIMEOUT to the scaffold override list subjects it to an ASCII allowlist that
rejects valid Go durations such as +5m, 1µs, and 1μs. Those repository-variable overrides
never reach the CLI, so it silently resolves the timeout from another source or uses the 15-minute
default.

3. stallTimeout guard lacks tests 📘 Rule violation ▣ Testability
Description
The new branch that disables the watchdog when its timeout meets or exceeds the global run timeout
has no corresponding behavioral test. Regressions could silently leave runs unprotected or disable
valid watchdog configurations.
Code

internal/cli/run.go[1856]

+	if stallTimeout >= timeout {
Relevance

●● Moderate

Dedicated branch tests are often accepted, but recent coverage-only findings were also rejected.

PR-#5961
PR-#6753

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
PR Compliance ID 1062049 requires assertions covering new or modified Go logic. The newly added
comparison at internal/cli/run.go:1856 introduces a distinct deactivation path, but repository
tests do not exercise or assert this condition.

Rule 1062049: Require tests for new or modified Go logic
internal/cli/run.go[1856-1864]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Add behavioral tests for the new stall-timeout deactivation condition.

## Issue Context
Verify that a stall timeout equal to or greater than the run timeout is disabled, while a smaller positive timeout remains enabled. Assert the resulting runtime parameters or observable behavior rather than only the log text.

## Fix Focus Areas
- internal/cli/run.go[1852-1864]
- internal/cli/run_test.go[1-1]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


Grey Divider

Qodo Logo

Comment thread internal/runtime/runtime.go Resolved
Comment thread internal/runtime/stall.go Resolved
Comment thread internal/runtime/stall.go Resolved
@guyoron1
guyoron1 marked this pull request as draft September 1, 2026 12:15
@guyoron1
guyoron1 marked this pull request as ready for review September 1, 2026 12:48
@qodo-code-review

Copy link
Copy Markdown

Code review by qodo was updated up to the latest commit 0339139

@guyoron1
guyoron1 force-pushed the feat/runner-stall-watchdog branch from d81237a to 2b1e60b Compare September 2, 2026 06:04
@guyoron1

guyoron1 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

/review

@qodo-code-review

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Warning

/review is deprecated. Use /agentic_review instead (removal date not yet scheduled).

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Misclassified Timeout

The watchdog has no awareness that the global timeout or another cancellation source may have already terminated the command. If command shutdown or stream draining overlaps the stall threshold, the watchdog can transition to fired and cause Run to report ErrStalled even though the global timeout was the actual cause. Consider disarming it when the command context is canceled, or otherwise preserving which cancellation source won.

case silence >= w.timeout:
	if w.state.CompareAndSwap(watchdogArmed, watchdogFired) {
		w.kill()
	}
	return

@guyoron1

guyoron1 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

/agentic_review

Comment thread internal/runtime/stall.go Resolved
@qodo-code-review

Copy link
Copy Markdown

Code review by qodo was updated up to the latest commit 2b1e60b

@guyoron1

guyoron1 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

/agentic_review

Comment thread internal/runtime/stall.go Resolved
@qodo-code-review

Copy link
Copy Markdown

Code review by qodo was updated up to the latest commit 957481a

@rh-hemartin

Copy link
Copy Markdown
Member

Any way we can fold this with the heartbeat? Or the other way around: any way to fold the heart beat into this? They serve similar purposes and the watchdog could be reporting "agent working: x seconds since last event" each 30 seconds (as the heartbeat interval is 30 seconds).

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review sweep — 5 findings posted inline (no approval/request-changes; comment only):

  • HIGH internal/runtime/stall.go:46 — the stall kill releases the local openshell client; nothing signals the agent inside the sandbox (verified against OpenShell v0.0.116 source)
  • HIGH internal/runtime/pi_run.go:528 — liveness counted at the AgentEvent level, so streaming tool output (pi tool_execution_update, Claude user tool_result) reads as silence
  • HIGH internal/cli/run_overrides.go:115 — default 10m equals Claude Code's BASH_MAX_TIMEOUT_MS ceiling; previously-successful runs now fail; not marked breaking
  • MEDIUM docs/runtimes.md:88 — FULLSEND_STALL_TIMEOUT documented as a CI repo-variable override but not in the setup-agent-env.sh allowlist
  • MEDIUM internal/cli/run.go:1852 — stall timeout not bounded by the run timeout; a no-op for timeout_minutes <= 10 harnesses (triage, prioritize)

Comment thread internal/runtime/stall.go Resolved
Comment thread internal/runtime/pi_run.go Resolved
Comment thread internal/cli/run_overrides.go Resolved
Comment thread docs/runtimes.md Resolved
Comment thread internal/cli/run.go Resolved
@guyoron1

guyoron1 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

/agentic_review

Comment thread internal/cli/run.go Resolved
Comment thread internal/runtime/claude_progress.go Resolved
@qodo-code-review

Copy link
Copy Markdown

Code review by qodo was updated up to the latest commit 815582c

- re-check lastEvent under mu before the half-timeout warning, so an
  event arriving after the tick computed its silence suppresses the
  stale warning
- count liveness per well-formed stream line (parseClaudeStreamLines /
  parsePiStreamLines feed stall.note), so streaming tool output and
  lifecycle lines with no AgentEvent mapping keep the watchdog quiet
- raise the default stall timeout to 15m: Claude Code's bash ceiling
  (BASH_MAX_TIMEOUT_MS, 600000ms) makes a 10m tool call legitimate, so
  the default needs headroom above it
- allowlist FULLSEND_STALL_TIMEOUT in setup-agent-env.sh so the
  documented CI repository variable actually reaches the runner
- skip arming the watchdog (with a log line) when the stall timeout is
  not below the run timeout, where the global deadline always fires
  first and the watchdog could never act
- document the real kill chain: the cancel kills the local openshell
  exec client; the in-sandbox agent dies when the deferred sandbox
  teardown deletes the sandbox, and survives under --keep-sandbox

Signed-off-by: guy oron <goron@redhat.com>
- count a fully consumed oversized stream line (> streamBufSize) as
  liveness in both parsers: it is excluded from semantic parsing, but a
  runtime writing megabytes is alive and must not be killed as stalled
- extract the stall-vs-run-timeout disable decision into
  effectiveStallTimeout and cover it with a behavioral test (disabled at
  or above the run timeout, untouched below, 0 stays 0)
- validate FULLSEND_STALL_TIMEOUT repository variables with a
  duration-shaped pattern in setup-agent-env.sh: the shared charset
  rejected valid Go durations (+5m, 1µs, 1μs), silently dropping the
  override; µ/μ are matched as literal alternations so the check stays
  byte-safe in any locale, and injection protection is preserved

Signed-off-by: guy oron <goron@redhat.com>
- the stall kill only released the local `openshell sandbox exec`
  client, which is all the global timeout does and signals nothing
  inside the sandbox: the wedged agent kept writing the workspace and
  spending tokens until teardown, and indefinitely under
  --keep-sandbox. stallKill now runs the stray-process sweep over a
  second exec channel first (OpenShell exposes no signal API), then
  cancels; a failed sweep is reported and still releases the client
- arm the watchdog in CodexRuntime.Run: it has the same
  ExecStreamReader -> handler -> parse shape as the other two streaming
  runtimes but never read params.StallTimeout, so a wedged codex run
  burned its whole global timeout with no ErrStalled.
  parseCodexStreamLines gives it the same per-line liveness hook, so
  item.started/item.updated progress lines are not mistaken for silence
- derive the file list in the wiring tests from the ExecStreamReader
  call sites, so a fourth streaming runtime cannot ship unguarded the
  way codex did
- export StallDetectionLatency (the poll interval watch() ticks at) as
  the one source of truth for how late the kill can land
- correct the RunParams doc comment: it said runtimes that stream no
  events ignore StallTimeout, which read as "no streaming runtime is
  missing"

Signed-off-by: guy oron <goron@redhat.com>
- effectiveStallTimeout compared the threshold, not the kill: the
  watchdog polls, so a stall just under the run timeout (14m50s against
  15m) was reported as armed while the global deadline usually won the
  race. Compare against stall + StallDetectionLatency instead, and name
  the interval in the "inactive" line
- cover the run.go wiring: fold the resolve/warn/disarm decision into
  runStallTimeout and the ErrStalled verdict into noteStalledRun, both
  exercised directly — the branches sat inside runAgent, which no test
  reaches past sandbox creation
- test the case the fleet actually hits: harnesses with
  timeout_minutes: 10 get no watchdog at the 15m default

Signed-off-by: guy oron <goron@redhat.com>
The kill was documented as leaving the in-sandbox agent running until
teardown; it now terminates it. Name codex as a covered runtime, and
say that the stall timeout must clear timeout_minutes by more than the
poll interval — so a harness at 10 minutes or less has no stall
protection at the default.

Signed-off-by: guy oron <goron@redhat.com>
The unit tests cover the watchdog, stallKill and the line hooks in
isolation; nothing showed they compose. Drive the real ClaudeRuntime.Run
against a fake openshell on PATH -- the stub shape claude_test.go
already uses -- and assert that a stream which goes quiet ends with the
sandbox swept exactly once and ErrStalled returned. Deleting the sweep
from stallKill, or the stalledErr check from claude.go, fails it.

TestStreamingRuntimesArmTheWatchdog stays, now labelled for what it is:
a source-shape guard covering all three runtimes at once, not a
behavioural test. Its cost is a rename or a reflowed call breaking it;
what it buys is catching a fourth streaming runtime shipped unguarded,
which is how codex shipped unguarded here.

Also: the malformed-value warning goes through the printer, which run.go
points at stdout, so run.md saying "reported on stderr" was wrong.

Signed-off-by: guy oron <goron@redhat.com>
effectiveStallTimeout disarms when timeout_minutes is not above the stall
timeout plus its detection interval: at the default that is 15m + 30s,
so harnesses at 15 minutes or less get no stall protection, not 10 or
less as the prose still said from before the interval was accounted for.

Signed-off-by: guy oron <goron@redhat.com>
…ntials

stallKill's sweep is not serialized against the credential refreshers,
and the comment justified that with the sandbox being torn down — which
is not true under --keep-sandbox. State the actual reason: a kill path
must not wait on a lock a wedged upload may hold, the only file it can
truncate is the OIDC token, and that token is dead within five minutes
of the run failing regardless. docs/cli/run.md now tells operators to
re-seed credentials before reusing a stall-killed sandbox they kept.

Signed-off-by: guy oron <goron@redhat.com>
The Agent tool consumes its child's stdout instead of forwarding it, so
the parent pi's JSON stream is silent for the whole child — up to the
child's timeoutSeconds (900s), which is exactly the runner's default
FULLSEND_STALL_TIMEOUT. A legitimately long persona therefore raced the
stall kill, and lost by the watchdog's poll latency.

run() now emits a progress update through pi's onUpdate every 30s while
a child is running; each one is a tool_execution_update line on the
parent's stream, which the watchdog already counts as liveness. The
interval is unref'd and cleared when the child is gone, and a child that
is itself wedged stays bounded by its own timeout, as before.

Signed-off-by: guy oron <goron@redhat.com>
Two stall-watchdog passages were inaccurate, verified against the pinned
codex rust-v0.152.1 and the run loop:

- codex `item.updated` is a plan/todo event, not a command stream, so it
  was never the liveness source the docs claimed. A long codex command
  emits nothing between item.started and item.completed; what keeps it
  live is unified-exec yielding to the model at least every 30s
  (default-on at the pin). Record that assumption in the CODEX_VERSION
  re-check table.
- the global-timeout path already TERM/KILLs the sandbox processes when
  an iteration ends at its budget, so "cancel is all the global timeout
  does" was stale. What is unique to the stall path is that it sweeps
  from the runtime layer, because the run loop's failure branch returns
  before the budget-end sweep runs.

Comment- and doc-only; no behaviour change.

Signed-off-by: guy oron <goron@redhat.com>
A stall kill reaches run.go's `runErr != nil` branch, which returned
before step 9b/9c extraction, so a wedged agent that had done useful
work lost its output files, transcripts and debug log -- the only
evidence of what it was doing. The global-timeout path preserves them.

Factor the extraction into a closure and run it on the stall branch too
(stallKill already swept the sandbox from the runtime layer, so the
artifacts are final). Other runtime failures keep the prior no-extract
behaviour, since their sandbox was not swept.

Signed-off-by: guy oron <goron@redhat.com>
A stalled pi run returned at stall.stalledErr() before folding the
extension's sub-agent usage file, so a parent that stalled after its
children finished dropped their token/cost from metrics.json and left
per_model_usage no longer summing to the totals. Sub-agent stalls are a
real path -- the 30s liveness tick exists because parents wait on
children.

Move the fold into a closure and run it before returning ErrStalled,
the same best-effort read as the clean path. Add a stall test that pins
the completed child's spend surviving the kill.

Signed-off-by: guy oron <goron@redhat.com>
While a pi sub-agent runs, the Agent tool feeds the parent stream a
liveness line only every 30s (DEFAULT_LIVENESS_MS), and setInterval does
not fire at once. resolveStallTimeout accepted any non-negative value,
so a timeout at or below that tick would kill a parent waiting on a
healthy child as stalled.

Floor a non-zero FULLSEND_STALL_TIMEOUT at two ticks (1m) in
runStallTimeout, logging the raise; 0 (disabled) is untouched. Document
the floor next to the advice to lower the timeout for short harnesses.

Signed-off-by: guy oron <goron@redhat.com>
OpenCodeRuntime.Run landed on main (unbound-force#510) after this
branch said its Run was a stub. It drains an NDJSON stream through
sandbox.ExecStreamReader like the other three, so
TestStreamingRuntimesArmTheWatchdog now finds it and requires the
watchdog.

Run arms it with the same stallKill, feeds it from a new
parseOpenCodeStreamLines line hook, disarms before Wait and reports
ErrStalled ahead of the Wait error. The docs and the RunParams comment
name opencode, and say that a single long opencode tool call is
silent, as on codex.

Signed-off-by: Guy Oron <goron@redhat.com>
piExecModel's stall return dropped lastResult, while the clean return
carries it. pi's parent tokens and cost come only from the agent_end
ResultEvent, so a pi process that wrote agent_end and then hung (a
lingering child or an open handle) lost the parent's whole spend from
metrics.json, while the sub-agent fold still added the children.

The stall return now carries lastResult, which Run already replays
into the metrics ahead of the stall branch. A test streams a full run
and then goes silent, and asserts the parent's cost and tokens next to
ErrStalled.

Signed-off-by: Guy Oron <goron@redhat.com>
kill() ran inline in watch() and nothing waited for it. The sweep's
TERM ends the agent, the stream reaches EOF, Wait returns and Run
returned ErrStalled while killStrayProcesses could still be in its KILL
pass or blocked on its exec timeout. Its "Terminated N stalled sandbox
process(es)" or could-not-terminate line then printed after the
failure, or not at all when the CLI exited first, and run.go's "the
sweep is done, the artifacts are final" premise held only by luck.

watch() now closes a killed channel once kill returns, and stalledErr,
which every streaming runtime calls before returning ErrStalled, waits
on it, bounded by killStrayProcessesTimeout plus a 5s margin. A test
with a slow sweep asserts the sweep's line is written before
stalledErr returns.

Signed-off-by: Guy Oron <goron@redhat.com>
The codex liveness claim did not hold. MAX_YIELD_TIME_MS caps only the
first exec_command yield; an empty write_stdin poll of a running
command waits up to DEFAULT_MAX_BACKGROUND_TERMINAL_TIMEOUT_MS (5m) and
emits EventMsg::TerminalInteraction, which the exec JSONL processor
drops in its catch-all arm. The only other line a poll turn can add is
a reasoning item, dropped when its summary is empty, and the runner
requests none. Checked at rust-v0.152.1 and at the current pin,
rust-v0.166.0.

So a codex command is one silent stretch: the comments, docs/cli/run.md
and the CODEX_VERSION re-check row now say that, and that a codex
command longer than FULLSEND_STALL_TIMEOUT is killed as stalled. The
re-check row points at the poll ceiling and the catch-all arm.

A single poll is 5m of silence on its own, so any setting below that
could kill a healthy codex run. runStallTimeout now takes the runtime
and floors codex at the poll ceiling plus the existing 1m floor (6m),
logging the raise as it does for the 1m floor.

Signed-off-by: Guy Oron <goron@redhat.com>
defaultStallTimeout was justified only by Claude Code's bash ceiling
(BASH_MAX_TIMEOUT_MS, 10m), which bounds Claude alone. pi's bash tool
has no default timeout and writes progress only when output arrives,
codex writes nothing while a command runs, and opencode writes a tool
line only when the call ends, so on those runtimes a quiet command
longer than the stall timeout is killed as stalled.

The constant's comment and docs/cli/run.md now call 10m a Claude-only
lower bound, say plainly which quiet commands are killed, and note that
15m is not yet backed by measured inter-line gaps from real runs.

Signed-off-by: Guy Oron <goron@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 12:11 AM UTC · Completed 12:24 AM UTC

Commit: 78dff50 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $3.23

@guyoron1
guyoron1 force-pushed the feat/runner-stall-watchdog branch from 78dff50 to 2a9dd18 Compare November 15, 2026 00:24
@fullsend-ai-review fullsend-ai-review Bot added the risk/moderate PR risk: moderate label Nov 15, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Risk Assessment: moderate (2/5)

Details

Large feature adding runtime stall detection with elevated churn and coupling in modified CLI and runtime files, balanced by zero protected or security-sensitive paths, no dependency or CI workflow changes, and an established contributor.

Previous run

Risk Assessment: moderate (2/5)

Details

Substantial additions across runtime files and elevated historical churn are mitigated by zero protected-path or dependency changes, established contributor authorship, and absence of CI workflow modifications.

@fullsend-ai-review

fullsend-ai-review Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Review

Findings

Medium

  • [logic-error] internal/runtime/codex_run.go:690 — Returning immediately on ErrStalled skips the child-rollout discovery and foldCodexChildUsage below. If a Codex parent stalls after its sub-agents have completed work, their tokens and cost are missing from the partial metrics.json, although the Pi stall path preserves completed children's usage.
    Remediation: Preserve the stall verdict, run child-rollout discovery and the best-effort usage fold after the sweep finishes, then return ErrStalled. Add a Codex stall test with completed child usage.
  • [incomplete-doc] docs/contributing/config-source-inventory.md:126 — FULLSEND_STALL_TIMEOUT (no YAML field, copied by setup-agent-env.sh) is missing from the "Values with no file field" table that lists the other such variables.
    Remediation: Add a FULLSEND_STALL_TIMEOUT entry to that table.
  • [backward-incompatible] internal/cli/run_overrides.go:150 — The watchdog is on by default (15m) for existing consumers; a pi/codex/opencode run silent for longer than that (e.g. a long quiet command) is now killed with ErrStalled, and the default is not backed by measured inter-line gaps. FULLSEND_STALL_TIMEOUT=0 opts out and the PR body discloses the default. Advisory.
    Remediation: Consider opt-in (0) until real gap data exists, or a higher per-runtime default for pi/codex/opencode; call out the default-on change in release notes.
  • [missing-authorization] internal/runtime/stall.go — Non-trivial feature with no linked authorizing issue (Surface the selected runtime/model where users look, then make the runtime selectable at setup and per run #6526 is referenced only as prior work). Process gap; the PR body documents motivation, scope and non-goals. Advisory.
    Remediation: Link the authorizing tracking issue or have a maintainer confirm scope.

Low

  • [doc-style] internal/runtime/stall_test.go:381 — The doc comment for TestStallKill_TerminatesTheAgentThenReleasesTheClient sits above TestStallWatchdog_StalledErrWaitsForTheSweep, leaving the former (line 409) without a comment.
    Remediation: Move lines 381-384 directly above the TestStallKill_... func.
  • [pattern-inconsistency] internal/runtime/stall.go:191 — stallKillWait is a package-level var never reassigned; sibling timeouts are consts.
    Remediation: Change it to const.

Labels: PR changes agent runner stall-detection behavior in Go.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR
Previous run

Review

Reason: stale-head

The review agent reviewed commit 78dff506981bc24d9e0d0a15ca1a1500d158207f but the PR HEAD is now 2a9dd18891c9bb3a0ee7e592a078db35c37c2cf7. This review was discarded to avoid approving unreviewed code.

@fullsend-ai-review

Copy link
Copy Markdown

/fs-review

@guyoron1

Copy link
Copy Markdown
Contributor Author

@waynesun09 thanks, all 4 addressed. Rebased onto current main, new head 2a9dd1889.

  • Codex yield claim (HIGH): you're right, and it still holds at the current pin (0.166.0): a write_stdin poll can wait 5m and the JSONL processor writes no line for it. The comments, docs/cli/run.md and the CODEX_VERSION re-check row now say a codex command is one silent stretch and that one longer than FULLSEND_STALL_TIMEOUT is killed; the re-check row points at DEFAULT_MAX_BACKGROUND_TERMINAL_TIMEOUT_MS and the catch-all arm. Codex is also floored at 6m (one poll plus the 1m floor), logged like the existing floor. e6ac533c5
  • 15m default rests on Claude's bash ceiling (MEDIUM): the constant's comment and docs/cli/run.md now call 10m a Claude-only lower bound and say that quiet pi, codex and opencode commands longer than the timeout are killed. I haven't measured inter-line gaps from real output.jsonl artifacts yet, and the comment says the number is not backed by that. 2a9dd1889
  • Pi stall return drops lastResult (MEDIUM): the stall return now carries it, and a new test streams a full run, goes silent, and checks the parent's cost and tokens next to ErrStalled. e690f2953
  • Nothing waits for the kill (MEDIUM): watch() closes a killed channel once the kill returns, and stalledErr waits on it (bounded by killStrayProcessesTimeout + 5s), so the sweep's line prints before the failure and the extracted artifacts are final. A test with a slow sweep covers it. 4b9375686

Since September main added two things this touches:

  • OpenCodeRuntime.Run is no longer a stub, so the source-shape guard found it: opencode now arms the watchdog like the other three. be75907e3
  • Claude got a model-fallback loop like pi's: the watchdog arms per attempt in runAttempt, and a stall comes back as the attempt's error, which claudeShouldFallBack never retries. The sub-agent liveness tick now spans the whole child fallback chain.

@fullsend-ai-review

fullsend-ai-review Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 12:26 AM UTC · Completed 12:40 AM UTC

Commit: 2a9dd18 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $3.63

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the review comment for full details.


// A stall is the cause of whatever Wait reports, so it is checked first.
if stallErr := stall.stalledErr(); stallErr != nil {
return exitCode, stallErr

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] logic-error

Returning immediately on ErrStalled skips the child-rollout discovery and foldCodexChildUsage below. If a Codex parent stalls after its sub-agents have completed work, their recorded tokens and cost are omitted from the partial metrics.json, even though those rollouts remain available after the sandbox sweep. The equivalent Pi stall path explicitly preserves completed children's usage.

Suggested fix: Preserve the stall verdict, perform child-rollout discovery and the best-effort usage fold after the sweep finishes, then return ErrStalled. Add a Codex stall test with completed child usage to verify that it survives in the returned metrics.

// which 15m is assumed, not measured, to cover. 15m is not yet backed by
// measured inter-line gaps from real output.jsonl artifacts; repos that know
// their cadence, or run longer quiet commands, tune FULLSEND_STALL_TIMEOUT.
const defaultStallTimeout = 15 * time.Minute

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] backward-incompatible

The stall watchdog is on by default (defaultStallTimeout = 15m) for existing consumers. A pi, codex or opencode run that is silent for more than 15m (e.g. a long quiet command) will now be killed with ErrStalled even though it previously worked; the code comments acknowledge the default is not backed by measured inter-line gaps. FULLSEND_STALL_TIMEOUT=0 opts out, and the PR body discloses the default, but downstream harnesses get no opt-in period. The new stalled metrics field and ErrStalled are additive, and a zero RunParams.StallTimeout leaves external Runtime callers unaffected.

Suggested fix: Consider shipping the default as opt-in (0) until real inter-line gap data exists, or a higher per-runtime default for pi/codex/opencode; at minimum flag the default-on behavior change and the opt-out in release notes.

}
}

// TestStallKill_TerminatesTheAgentThenReleasesTheClient: cancelling the exec

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] doc-style

The doc comment for TestStallKill_TerminatesTheAgentThenReleasesTheClient (lines 381-384) sits above TestStallWatchdog_StalledErrWaitsForTheSweep (which then has two consecutive comment blocks), leaving TestStallKill_TerminatesTheAgentThenReleasesTheClient at line 409 with no doc comment.

Suggested fix: Move the comment block at lines 381-384 directly above func TestStallKill_TerminatesTheAgentThenReleasesTheClient.

Comment thread internal/runtime/stall.go
// stallKillWait bounds how long stalledErr waits for the kill to return. The
// kill is stallKill, whose sweep exec is bounded by killStrayProcessesTimeout;
// the margin covers the local openshell client around it.
var stallKillWait = killStrayProcessesTimeout + 5*time.Second

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] pattern-inconsistency

stallKillWait is a package-level var built from constants and is never reassigned (only used at stall.go:191 and 206), unlike the sibling timeouts killStrayProcessesTimeout and checkoutBranchRunnerTimeout, which are consts.

Suggested fix: Change var stallKillWait = killStrayProcessesTimeout + 5*time.Second to const.

@fullsend-ai-review fullsend-ai-review Bot added the component/runner Agent runner behavior and lifecycle label Nov 15, 2026

This branch was successfully deployed

1 active deployment
site-preview — 2a9dd188 Deployed Nov 15, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/runner Agent runner behavior and lifecycle Review effort 3/5 risk/moderate PR risk: moderate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants