Repository navigation
Conversation
E2E tests did not runE2E tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer with write access must add the See E2E testing guide for details. |
E2E tests did not runE2E tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer must add the See E2E testing guide for details. |
Site previewPreview: https://0cf71631-site.fullsend-ai.workers.dev Commit: |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
ffe01bf to
0339139
Compare
PR Summary by QodoDetect stalled agent runs from event-stream inactivity
AI Description
Diagram
High-Level Assessment
Files changed (10)
|
Code Review by Qodo
1.
|
|
Code review by qodo was updated up to the latest commit 0339139 |
d81237a to
2b1e60b
Compare
|
/review |
PR Reviewer Guide 🔍Warning
Here are some key observations to aid the review process:
|
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit 2b1e60b |
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit 957481a |
|
Any way we can fold this with the heartbeat? Or the other way around: any way to fold the heart beat into this? They serve similar purposes and the watchdog could be reporting "agent working: x seconds since last event" each 30 seconds (as the heartbeat interval is 30 seconds). |
waynesun09
left a comment
There was a problem hiding this comment.
Review sweep — 5 findings posted inline (no approval/request-changes; comment only):
- HIGH
internal/runtime/stall.go:46— the stall kill releases the local openshell client; nothing signals the agent inside the sandbox (verified against OpenShell v0.0.116 source) - HIGH
internal/runtime/pi_run.go:528— liveness counted at the AgentEvent level, so streaming tool output (pitool_execution_update, Claudeusertool_result) reads as silence - HIGH
internal/cli/run_overrides.go:115— default 10m equals Claude Code'sBASH_MAX_TIMEOUT_MSceiling; previously-successful runs now fail; not marked breaking - MEDIUM
docs/runtimes.md:88—FULLSEND_STALL_TIMEOUTdocumented as a CI repo-variable override but not in thesetup-agent-env.shallowlist - MEDIUM
internal/cli/run.go:1852— stall timeout not bounded by the run timeout; a no-op fortimeout_minutes <= 10harnesses (triage, prioritize)
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit 815582c |
- re-check lastEvent under mu before the half-timeout warning, so an event arriving after the tick computed its silence suppresses the stale warning - count liveness per well-formed stream line (parseClaudeStreamLines / parsePiStreamLines feed stall.note), so streaming tool output and lifecycle lines with no AgentEvent mapping keep the watchdog quiet - raise the default stall timeout to 15m: Claude Code's bash ceiling (BASH_MAX_TIMEOUT_MS, 600000ms) makes a 10m tool call legitimate, so the default needs headroom above it - allowlist FULLSEND_STALL_TIMEOUT in setup-agent-env.sh so the documented CI repository variable actually reaches the runner - skip arming the watchdog (with a log line) when the stall timeout is not below the run timeout, where the global deadline always fires first and the watchdog could never act - document the real kill chain: the cancel kills the local openshell exec client; the in-sandbox agent dies when the deferred sandbox teardown deletes the sandbox, and survives under --keep-sandbox Signed-off-by: guy oron <goron@redhat.com>
- count a fully consumed oversized stream line (> streamBufSize) as liveness in both parsers: it is excluded from semantic parsing, but a runtime writing megabytes is alive and must not be killed as stalled - extract the stall-vs-run-timeout disable decision into effectiveStallTimeout and cover it with a behavioral test (disabled at or above the run timeout, untouched below, 0 stays 0) - validate FULLSEND_STALL_TIMEOUT repository variables with a duration-shaped pattern in setup-agent-env.sh: the shared charset rejected valid Go durations (+5m, 1µs, 1μs), silently dropping the override; µ/μ are matched as literal alternations so the check stays byte-safe in any locale, and injection protection is preserved Signed-off-by: guy oron <goron@redhat.com>
- the stall kill only released the local `openshell sandbox exec` client, which is all the global timeout does and signals nothing inside the sandbox: the wedged agent kept writing the workspace and spending tokens until teardown, and indefinitely under --keep-sandbox. stallKill now runs the stray-process sweep over a second exec channel first (OpenShell exposes no signal API), then cancels; a failed sweep is reported and still releases the client - arm the watchdog in CodexRuntime.Run: it has the same ExecStreamReader -> handler -> parse shape as the other two streaming runtimes but never read params.StallTimeout, so a wedged codex run burned its whole global timeout with no ErrStalled. parseCodexStreamLines gives it the same per-line liveness hook, so item.started/item.updated progress lines are not mistaken for silence - derive the file list in the wiring tests from the ExecStreamReader call sites, so a fourth streaming runtime cannot ship unguarded the way codex did - export StallDetectionLatency (the poll interval watch() ticks at) as the one source of truth for how late the kill can land - correct the RunParams doc comment: it said runtimes that stream no events ignore StallTimeout, which read as "no streaming runtime is missing" Signed-off-by: guy oron <goron@redhat.com>
- effectiveStallTimeout compared the threshold, not the kill: the watchdog polls, so a stall just under the run timeout (14m50s against 15m) was reported as armed while the global deadline usually won the race. Compare against stall + StallDetectionLatency instead, and name the interval in the "inactive" line - cover the run.go wiring: fold the resolve/warn/disarm decision into runStallTimeout and the ErrStalled verdict into noteStalledRun, both exercised directly — the branches sat inside runAgent, which no test reaches past sandbox creation - test the case the fleet actually hits: harnesses with timeout_minutes: 10 get no watchdog at the 15m default Signed-off-by: guy oron <goron@redhat.com>
The kill was documented as leaving the in-sandbox agent running until teardown; it now terminates it. Name codex as a covered runtime, and say that the stall timeout must clear timeout_minutes by more than the poll interval — so a harness at 10 minutes or less has no stall protection at the default. Signed-off-by: guy oron <goron@redhat.com>
The unit tests cover the watchdog, stallKill and the line hooks in isolation; nothing showed they compose. Drive the real ClaudeRuntime.Run against a fake openshell on PATH -- the stub shape claude_test.go already uses -- and assert that a stream which goes quiet ends with the sandbox swept exactly once and ErrStalled returned. Deleting the sweep from stallKill, or the stalledErr check from claude.go, fails it. TestStreamingRuntimesArmTheWatchdog stays, now labelled for what it is: a source-shape guard covering all three runtimes at once, not a behavioural test. Its cost is a rename or a reflowed call breaking it; what it buys is catching a fourth streaming runtime shipped unguarded, which is how codex shipped unguarded here. Also: the malformed-value warning goes through the printer, which run.go points at stdout, so run.md saying "reported on stderr" was wrong. Signed-off-by: guy oron <goron@redhat.com>
effectiveStallTimeout disarms when timeout_minutes is not above the stall timeout plus its detection interval: at the default that is 15m + 30s, so harnesses at 15 minutes or less get no stall protection, not 10 or less as the prose still said from before the interval was accounted for. Signed-off-by: guy oron <goron@redhat.com>
…ntials stallKill's sweep is not serialized against the credential refreshers, and the comment justified that with the sandbox being torn down — which is not true under --keep-sandbox. State the actual reason: a kill path must not wait on a lock a wedged upload may hold, the only file it can truncate is the OIDC token, and that token is dead within five minutes of the run failing regardless. docs/cli/run.md now tells operators to re-seed credentials before reusing a stall-killed sandbox they kept. Signed-off-by: guy oron <goron@redhat.com>
The Agent tool consumes its child's stdout instead of forwarding it, so the parent pi's JSON stream is silent for the whole child — up to the child's timeoutSeconds (900s), which is exactly the runner's default FULLSEND_STALL_TIMEOUT. A legitimately long persona therefore raced the stall kill, and lost by the watchdog's poll latency. run() now emits a progress update through pi's onUpdate every 30s while a child is running; each one is a tool_execution_update line on the parent's stream, which the watchdog already counts as liveness. The interval is unref'd and cleared when the child is gone, and a child that is itself wedged stays bounded by its own timeout, as before. Signed-off-by: guy oron <goron@redhat.com>
Two stall-watchdog passages were inaccurate, verified against the pinned codex rust-v0.152.1 and the run loop: - codex `item.updated` is a plan/todo event, not a command stream, so it was never the liveness source the docs claimed. A long codex command emits nothing between item.started and item.completed; what keeps it live is unified-exec yielding to the model at least every 30s (default-on at the pin). Record that assumption in the CODEX_VERSION re-check table. - the global-timeout path already TERM/KILLs the sandbox processes when an iteration ends at its budget, so "cancel is all the global timeout does" was stale. What is unique to the stall path is that it sweeps from the runtime layer, because the run loop's failure branch returns before the budget-end sweep runs. Comment- and doc-only; no behaviour change. Signed-off-by: guy oron <goron@redhat.com>
A stall kill reaches run.go's `runErr != nil` branch, which returned before step 9b/9c extraction, so a wedged agent that had done useful work lost its output files, transcripts and debug log -- the only evidence of what it was doing. The global-timeout path preserves them. Factor the extraction into a closure and run it on the stall branch too (stallKill already swept the sandbox from the runtime layer, so the artifacts are final). Other runtime failures keep the prior no-extract behaviour, since their sandbox was not swept. Signed-off-by: guy oron <goron@redhat.com>
A stalled pi run returned at stall.stalledErr() before folding the extension's sub-agent usage file, so a parent that stalled after its children finished dropped their token/cost from metrics.json and left per_model_usage no longer summing to the totals. Sub-agent stalls are a real path -- the 30s liveness tick exists because parents wait on children. Move the fold into a closure and run it before returning ErrStalled, the same best-effort read as the clean path. Add a stall test that pins the completed child's spend surviving the kill. Signed-off-by: guy oron <goron@redhat.com>
While a pi sub-agent runs, the Agent tool feeds the parent stream a liveness line only every 30s (DEFAULT_LIVENESS_MS), and setInterval does not fire at once. resolveStallTimeout accepted any non-negative value, so a timeout at or below that tick would kill a parent waiting on a healthy child as stalled. Floor a non-zero FULLSEND_STALL_TIMEOUT at two ticks (1m) in runStallTimeout, logging the raise; 0 (disabled) is untouched. Document the floor next to the advice to lower the timeout for short harnesses. Signed-off-by: guy oron <goron@redhat.com>
OpenCodeRuntime.Run landed on main (unbound-force#510) after this branch said its Run was a stub. It drains an NDJSON stream through sandbox.ExecStreamReader like the other three, so TestStreamingRuntimesArmTheWatchdog now finds it and requires the watchdog. Run arms it with the same stallKill, feeds it from a new parseOpenCodeStreamLines line hook, disarms before Wait and reports ErrStalled ahead of the Wait error. The docs and the RunParams comment name opencode, and say that a single long opencode tool call is silent, as on codex. Signed-off-by: Guy Oron <goron@redhat.com>
piExecModel's stall return dropped lastResult, while the clean return carries it. pi's parent tokens and cost come only from the agent_end ResultEvent, so a pi process that wrote agent_end and then hung (a lingering child or an open handle) lost the parent's whole spend from metrics.json, while the sub-agent fold still added the children. The stall return now carries lastResult, which Run already replays into the metrics ahead of the stall branch. A test streams a full run and then goes silent, and asserts the parent's cost and tokens next to ErrStalled. Signed-off-by: Guy Oron <goron@redhat.com>
kill() ran inline in watch() and nothing waited for it. The sweep's TERM ends the agent, the stream reaches EOF, Wait returns and Run returned ErrStalled while killStrayProcesses could still be in its KILL pass or blocked on its exec timeout. Its "Terminated N stalled sandbox process(es)" or could-not-terminate line then printed after the failure, or not at all when the CLI exited first, and run.go's "the sweep is done, the artifacts are final" premise held only by luck. watch() now closes a killed channel once kill returns, and stalledErr, which every streaming runtime calls before returning ErrStalled, waits on it, bounded by killStrayProcessesTimeout plus a 5s margin. A test with a slow sweep asserts the sweep's line is written before stalledErr returns. Signed-off-by: Guy Oron <goron@redhat.com>
The codex liveness claim did not hold. MAX_YIELD_TIME_MS caps only the first exec_command yield; an empty write_stdin poll of a running command waits up to DEFAULT_MAX_BACKGROUND_TERMINAL_TIMEOUT_MS (5m) and emits EventMsg::TerminalInteraction, which the exec JSONL processor drops in its catch-all arm. The only other line a poll turn can add is a reasoning item, dropped when its summary is empty, and the runner requests none. Checked at rust-v0.152.1 and at the current pin, rust-v0.166.0. So a codex command is one silent stretch: the comments, docs/cli/run.md and the CODEX_VERSION re-check row now say that, and that a codex command longer than FULLSEND_STALL_TIMEOUT is killed as stalled. The re-check row points at the poll ceiling and the catch-all arm. A single poll is 5m of silence on its own, so any setting below that could kill a healthy codex run. runStallTimeout now takes the runtime and floors codex at the poll ceiling plus the existing 1m floor (6m), logging the raise as it does for the 1m floor. Signed-off-by: Guy Oron <goron@redhat.com>
defaultStallTimeout was justified only by Claude Code's bash ceiling (BASH_MAX_TIMEOUT_MS, 10m), which bounds Claude alone. pi's bash tool has no default timeout and writes progress only when output arrives, codex writes nothing while a command runs, and opencode writes a tool line only when the call ends, so on those runtimes a quiet command longer than the stall timeout is killed as stalled. The constant's comment and docs/cli/run.md now call 10m a Claude-only lower bound, say plainly which quiet commands are killed, and note that 15m is not yet backed by measured inter-line gaps from real runs. Signed-off-by: Guy Oron <goron@redhat.com>
|
🤖 Finished Review · ✅ Success · Started 12:11 AM UTC · Completed 12:24 AM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $3.23 |
78dff50 to
2a9dd18
Compare
|
Risk Assessment: moderate (2/5) DetailsLarge feature adding runtime stall detection with elevated churn and coupling in modified CLI and runtime files, balanced by zero protected or security-sensitive paths, no dependency or CI workflow changes, and an established contributor. Previous runRisk Assessment: moderate (2/5) DetailsSubstantial additions across runtime files and elevated historical churn are mitigated by zero protected-path or dependency changes, established contributor authorship, and absence of CI workflow modifications. |
ReviewFindingsMedium
Low
Labels: PR changes agent runner stall-detection behavior in Go. Next steps:
Previous runReviewReason: stale-head The review agent reviewed commit |
|
/fs-review |
|
@waynesun09 thanks, all 4 addressed. Rebased onto current
Since September main added two things this touches:
|
|
🤖 Finished Review · ✅ Success · Started 12:26 AM UTC · Completed 12:40 AM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $3.63 |
|
|
||
| // A stall is the cause of whatever Wait reports, so it is checked first. | ||
| if stallErr := stall.stalledErr(); stallErr != nil { | ||
| return exitCode, stallErr |
There was a problem hiding this comment.
[medium] logic-error
Returning immediately on ErrStalled skips the child-rollout discovery and foldCodexChildUsage below. If a Codex parent stalls after its sub-agents have completed work, their recorded tokens and cost are omitted from the partial metrics.json, even though those rollouts remain available after the sandbox sweep. The equivalent Pi stall path explicitly preserves completed children's usage.
Suggested fix: Preserve the stall verdict, perform child-rollout discovery and the best-effort usage fold after the sweep finishes, then return ErrStalled. Add a Codex stall test with completed child usage to verify that it survives in the returned metrics.
| // which 15m is assumed, not measured, to cover. 15m is not yet backed by | ||
| // measured inter-line gaps from real output.jsonl artifacts; repos that know | ||
| // their cadence, or run longer quiet commands, tune FULLSEND_STALL_TIMEOUT. | ||
| const defaultStallTimeout = 15 * time.Minute |
There was a problem hiding this comment.
[medium] backward-incompatible
The stall watchdog is on by default (defaultStallTimeout = 15m) for existing consumers. A pi, codex or opencode run that is silent for more than 15m (e.g. a long quiet command) will now be killed with ErrStalled even though it previously worked; the code comments acknowledge the default is not backed by measured inter-line gaps. FULLSEND_STALL_TIMEOUT=0 opts out, and the PR body discloses the default, but downstream harnesses get no opt-in period. The new stalled metrics field and ErrStalled are additive, and a zero RunParams.StallTimeout leaves external Runtime callers unaffected.
Suggested fix: Consider shipping the default as opt-in (0) until real inter-line gap data exists, or a higher per-runtime default for pi/codex/opencode; at minimum flag the default-on behavior change and the opt-out in release notes.
| } | ||
| } | ||
|
|
||
| // TestStallKill_TerminatesTheAgentThenReleasesTheClient: cancelling the exec |
There was a problem hiding this comment.
[low] doc-style
The doc comment for TestStallKill_TerminatesTheAgentThenReleasesTheClient (lines 381-384) sits above TestStallWatchdog_StalledErrWaitsForTheSweep (which then has two consecutive comment blocks), leaving TestStallKill_TerminatesTheAgentThenReleasesTheClient at line 409 with no doc comment.
Suggested fix: Move the comment block at lines 381-384 directly above func TestStallKill_TerminatesTheAgentThenReleasesTheClient.
| // stallKillWait bounds how long stalledErr waits for the kill to return. The | ||
| // kill is stallKill, whose sweep exec is bounded by killStrayProcessesTimeout; | ||
| // the margin covers the local openshell client around it. | ||
| var stallKillWait = killStrayProcessesTimeout + 5*time.Second |
There was a problem hiding this comment.
[low] pattern-inconsistency
stallKillWait is a package-level var built from constants and is never reassigned (only used at stall.go:191 and 206), unlike the sibling timeouts killStrayProcessesTimeout and checkoutBranchRunnerTimeout, which are consts.
Suggested fix: Change var stallKillWait = killStrayProcessesTimeout + 5*time.Second to const.
Heyaa : )
This one came out of watching a wedged run burn its entire global timeout while the heartbeat cheerfully printed "agent running" the whole time — the process was dead and nothing could tell.
The runtime now watches the runtime output stream instead: every well-formed line the stream parser reads is proof of life (including lines that map to no
AgentEvent— pi'stool_execution_update, Claude Code'susertool results, codex'sitem.started/item.completed), half a timeout of silence warns once per stall episode (::warning::in CI, printer everywhere), and a full timeout kills the run and fails it distinctly withErrStalled+"stalled": trueinmetrics.json.Config:
FULLSEND_STALL_TIMEOUT(Go duration, default15m,0disables), resolved by the CLI (runStallTimeoutininternal/cli/run_overrides.go) and handed to the runtime inRunParams.StallTimeout— runtimes don't read env themselves (#6526). A malformed value is reported and the default applies. A non-zero value is floored at1m(a pi sub-agent's 30s liveness tick), and at6mon codex (one poll of a running command can be silent for 5 minutes); the run logs the raise. The value is allowlisted insetup-agent-env.sh, so a CI repo variable can set it. The watchdog is disarmed — and the run says so — when the stall timeout plus its detection interval (a twentieth of the timeout, capped at 30s) is not below the harnesstimeout_minutes, since the global deadline would always win: at the default that means harnesses at 15 minutes or less get no stall protection.Coverage: claude, pi, codex and opencode — every runtime that drains a stream through
sandbox.ExecStreamReader— with the same call shape at each call site (startStallWatchdog, a per-linestall.notehook,stall.stalledErr()ahead of the Wait error); the scripteddummyruntimes ignore the field. On claude and pi the watchdog arms per model-fallback attempt, and a stall is final, never retried on the next model. A source-shape test finds the streaming runtimes from theirExecStreamReadercall sites and pins all four.What still reads as silence: codex writes nothing while a command runs (its polls of a running command produce no JSONL line), opencode writes a tool line only when the call ends, and pi's bash tool writes progress only when output arrives. On those runtimes a quiet command longer than the stall timeout is killed as stalled. The 15m default clears Claude Code's bash ceiling only; it is not yet backed by measured inter-line gaps from real runs.
Design points:
cancelfromsandbox.ExecStreamReaderonly SIGKILLs the localopenshell sandbox execclient; OpenShell has no signal API. So the kill first runs the same TERM-then-KILL stray-process sweep that clears the sandbox between iterations, over a second exec channel, then cancels — the wedged agent stops writing the workspace and spending tokens, under--keep-sandboxtoo.ErrStalledis reported only once that sweep has finished (bounded by its exec timeout), so its result line prints before the failure and the extracted artifacts are final. The sweep is not serialized against the credential refreshers (a kill path must not wait on a lock a wedged upload may hold); the only file it can truncate is the five-minute OIDC token, and the docs tell operators to re-seed before reusing a kept sandbox.BASH_MAX_TIMEOUT_MSceiling (600000ms), which the model routinely requests for test suites. Repos that raise the bash ceiling, or run long quiet commands on pi, codex or opencode, raise the stall timeout with it.timeoutSeconds(900s). On a stall, completed children's usage is folded into the metrics and the parent's ownagent_endresult is kept.ctx(a body-scope rebind once made every successful run report "cancelled"); the stop/fire race is closed with a CAS and silence is measured on the monotonic clock.Tests: the watchdog matrix (flowing lines never kill / silence kills exactly once /
0disables / warn-once-per-episode, rearmed by the next line / no annotations outside CI / no warning after disarm /ErrStalledwaits for a slow sweep),resolveStallTimeout,effectiveStallTimeoutandrunStallTimeoutcases (including the codex floor), the sweep-then-cancel order and its failure path,TestClaudeRuntime_Run_StallSweepsTheSandboxOncedriving the realRunpath, the line hooks of all four parsers, the source-shape guard, pi's usage fold and parent result on a stall,stalled,omitemptymarshalling, and the Agent tool's liveness updates (node --test).go build,go vet, andinternal/runtime,internal/cli,internal/scaffoldpass.Non-goals: per-dimension timeouts (dimensions run inside one CLI process the runner can't see into), and no change to the heartbeat or the global timeout — the heartbeat's future is a separate discussion after merge.