fix(agy-acp): surface swallowed backend errors (quota 429) instead of "(no response)" - #1320
Conversation
…response)" `agy --print` exits 0 with empty stdout/stderr when the model backend fails (e.g. quota 429 / RESOURCE_EXHAUSTED), recording the cause only in its own cli.log and never in the trajectory DB's error_details. agy-acp keyed failure detection solely off agy's exit code, so an exhausted-quota turn produced a blank `end_turn` and OpenAB rendered "(no response)" with no indication of why. When agy exits 0 but streams nothing this turn, scan the cli-*.log written during the turn for a known error signature (agent executor error / model unreachable / RESOURCE_EXHAUSTED), de-wrap glog's self-duplicated tail, and return a JSON-RPC -32603 error carrying the human-readable cause. This mirrors claude-agent-acp, which already surfaces backend limit errors to the user. Genuinely-empty successful turns (no matching log line) are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018QWocnhY9XoT7TjPK7JVaj
`cargo build`/`cargo test` from `working-directory: agy-acp` (what .github/workflows/ci-agy-acp.yml runs) fails with "current package believes it's in a workspace when it's not" because agy-acp is not a member of the root openab workspace and the parent Cargo.toml doesn't exclude it. Add an empty [workspace] table to agy-acp/Cargo.toml so the crate is its own workspace root and builds standalone, matching how CI and the release image build it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018QWocnhY9XoT7TjPK7JVaj
The PR adds [workspace] to agy-acp/Cargo.toml directly, but Dockerfile.unified unconditionally appends another [workspace] via printf, creating a duplicate TOML table header that causes cargo to fail (exit 101). Guard with grep -q so the printf is skipped when [workspace] already exists. Applied to both openab-agent and agy-acp lines for consistency.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Address review findings F1/F2/F5/F6 on the swallowed-error detector. Root change: snapshot cli-*.log sizes (name -> byte length) instead of just filenames, so detection scans only bytes appended during this turn. - F1 (stale-error false positive): remove the "fall back to all logs sorted by mtime" branch entirely; a candidate now qualifies only when its size grew past the pre-turn snapshot offset, so an error logged in an earlier turn is never re-surfaced. - F2 (cross-session leakage): reading from the per-file snapshot offset isolates this turn's output from a concurrent session appending to the same shared log directory. - F5 (unbounded read): replace read_to_string with a seek-to-offset tail read capped at MAX_LOG_SCAN_BYTES (256 KiB), decoded via from_utf8_lossy; errors live at the tail so the cap is safe. - F6 (test gaps): add tests for char-boundary truncation of a multi-byte message, stale pre-existing error isolation, and appended-only reads. F3 (-32603 vs -32000) and F4 (log path coupling) left as-is per the review's own note that they are acceptable / sufficient for now. cargo test: 22 passed (was 19). No new clippy warnings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ANHjz7TXsqjhoDk6SiJiGd
|
Addressed the review in
No new clippy warnings (remaining ones are pre-existing: |
Final Aggregated Review — PR #1320Modegroup-review (3/3 voices active: Claude, Codex, Gemini/agy) Review scope: Consensus Critical (must fix)
Consensus Important (must fix)
Actionable NIT (must fix — user requires all NITs cleaned up)
DisputedNone — full 3/3 convergence after R2 cross-debate. Voices unavailableNone — all 3 voices (Claude, Codex, Gemini/agy) completed R1 and R2. |
Code review — Claude Code + Codex (consolidated)Reviewed the diff with two independent passes (Claude Code and Codex). Both agree: the single-session happy path is sound and this is a genuine UX improvement, but the code comments claim a cross-session isolation guarantee that the implementation does not actually provide. No merge blockers; the items below are worth addressing (or explicitly documenting as accepted limitations) before/after merge. Both reviewers independently verified the two things that look like traps are actually correct — no change needed:
Findings (most severe first)1. [major] Cross-session error mis-attribution — the isolation guarantee is overstated. 2. [minor→major] 3. [minor] Only three hard-coded error anchors. 4. [minor] Premature de-wrap on the first 5. [minor] Log rotation/truncation blind spot. 6. [minor] Test coverage gaps. Bottom lineShip-worthy for the 429 UX win. Findings 1 and 2 are the ones to resolve or consciously accept — mainly by confirming whether concurrent Reviewed by Claude Code (Opus) + Codex. Neither pass compiled the crate; findings are from control-flow tracing of the diff and PR-head source. |
…nce, CI regression)
3-voice mob review (Claude + Codex + Gemini) converged on two Critical and
three Important findings across R1+R2 cross-debate; this addresses all of them.
Critical:
- Narrow (not fully close, given closed-source agy) the concurrent-session
log-misattribution window: detect_swallowed_agy_error now additionally
requires a candidate log's mtime to be at/after this turn's own agy child
spawn_time, excluding logs that finished growing before this turn started.
Doc comments on snapshot_agy_logs/detect_swallowed_agy_error now describe
the residual limitation honestly instead of overclaiming "never".
- .github/workflows/pre-beta-build.yml unconditionally appended [workspace]
to agy-acp/Cargo.toml, which this PR already declares permanently -- every
pre-beta run would produce a duplicate table and fail to parse. Dropped the
now-redundant injection for agy-acp there and in Dockerfile.unified (which
had the same dead branch after the Cargo.toml change).
Important:
- Dockerfile.unified's openab-agent workspace-injection line: && and ||
share equal left-to-right precedence, so a failed `cd` still let printf
and cargo build run in the wrong directory. Grouped with `{ ...; }` so a
failed cd now aborts the RUN as before.
- Log-scanner silent-failure paths (read_dir/metadata errors, no-anchor-match)
now eprintln instead of converging to identical silence -- distinguishes
"detection ran and found nothing" from "detection couldn't run".
- Extracted decide_turn_error as a pure, directly-unit-tested function so the
execute_prompt branch wiring (the cancelled/had_updates/status gate and
-32000 vs -32603 code selection) has coverage beyond the underlying helpers.
Also: read_log_tail takes the caller's already-known file length instead of
re-stat'ing; is_agy_cli_log/truncate_to_byte_boundary dedupe repeated inline
logic; new tests for the take(3) boundary and the 256KB tail-read cap.
cargo test: 31 passed (was 22). No new clippy warnings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ANHjz7TXsqjhoDk6SiJiGd
…, fix fake tests Round 2 of mob review (Claude + Codex + Gemini) verified the prior fix commit and found it incomplete in three ways; this addresses all of them. 1. detect_swallowed_agy_error still had two silent `.ok()?` drops (metadata()/modified()) inside its per-candidate closure, unlike its sibling snapshot_agy_logs which already logged the equivalent failure. Mirrored the explicit match+eprintln pattern there too, and added a dir-entry-iteration-error counter to both functions. 2. read_log_tail collapsed open/seek/read_to_end into a single silent `None`, so a genuine I/O failure on the log that actually recorded the swallowed error was indistinguishable from "read fine, no anchor matched" -- the caller's diagnostic message would confidently claim a clean scan when the content was never read at all. Each I/O step now eprintlns on failure, and the caller emits a distinct message when a candidate could not be read vs. genuinely didn't match. 3. `.take(3)` on the candidate scan is a genuine correctness gap (Codex + Claude code-reviewer both found it independently): with 4+ logs grown in one turn, the 4th-oldest candidate's error would never be scanned and the turn would silently fall back to "(no response)". Removed the cap entirely -- every candidate that grew this turn is now scanned (newest-first order preserved), since read cost per file is already bounded by MAX_LOG_SCAN_BYTES. Also fixed two tests that mutation testing (3 independent reviewers: Claude code-reviewer, Codex, Gemini) proved were not exercising what their own docstrings claimed: - test_detect_swallowed_agy_error_scans_beyond_first_candidate wrote the error-bearing log *last*, making it mtime-newest and never touching the take(3) boundary it claimed to test. Renamed and rewritten so the error log is oldest-by-mtime, now genuinely proving there's no cap. - test_read_log_tail_respects_offset_and_cap_on_large_file's single assertion couldn't distinguish "started at offset" from "started at len-cap" (both scenarios pass through the same content). Split into two tests, each isolating one branch of `offset.max(len - cap)`. Verified via mutation testing myself: reintroducing take(3) now fails the rewritten test; reverted after confirming. Hardened two timestamp-ordering tests' sleep durations to 1.1s (was 5-20ms) so ordering doesn't depend on sub-second mtime resolution on unusual filesystems (Gemini's flakiness NIT). cargo test: 32 passed (was 31). No new clippy warnings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ANHjz7TXsqjhoDk6SiJiGd
Round 3 codex spot-check found: on filesystems that truncate mtime to whole seconds, this turn's own cli-*.log (written a few hundred ms after spawn_time) could appear to predate spawn_time and get wrongly excluded by the `mtime < spawn_time` filter -- silently reintroducing the exact "(no response)" bug this PR exists to fix, on every affected turn rather than only under concurrency. A false negative here is worse than the already-acknowledged, narrower concurrent-session misattribution risk, so the comparison now only excludes a candidate that is unambiguously more than 1s stale (`mtime + 1s < spawn_time`). Added a test for the tolerance window itself (log written 300ms before spawn_time is still detected) alongside the existing exclusion test (log written 1.1s before spawn_time is still excluded). cargo test: 33 passed (was 32). No new clippy warnings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ANHjz7TXsqjhoDk6SiJiGd
Mob review — 3 rounds completeScope: Round 1 (Claude + Codex + Gemini, full R1+R2 debate) —
|
|
LGTM ✅ — Well-engineered adapter-boundary error surfacing; all prior review findings addressed. What This PR DoesWhen the How It Works
Findings
Addressing External Reviewer Feedback@dogzzdogzz (Claude Code + Codex consolidated review)
✅ Addressed in
✅ Addressed in
ℹ️ Accepted: Intentionally quota-focused for v1. Other backend failure classes (
ℹ️ Accepted: For the known glog format,
ℹ️ Accepted: agy uses timestamped log files (
✅ Addressed in Mob review (3 rounds by @howie)All Critical and Important findings from Rounds 1-3 verified addressed in commits Baseline Check
What's Good (🟢)
|
What problem does this solve?
When the Antigravity backend fails for a whole turn (most commonly an individual
quota exhaustion:
RESOURCE_EXHAUSTED/ HTTP 429), the user sees a bare"(no response)" with no indication of what went wrong or that it will recover.
This sends operators down the wrong path (suspecting auth / re-login) when the real
cause is a quota limit that resets on its own.
Root cause:
agy --printexits 0 with empty stdout/stderr on a backend failure,recording the cause only in its own
cli.log(and not even in the trajectory DB'serror_details).agy-acpkeyed failure detection solely off agy's exit code, so anexhausted-quota turn looked like a successful-but-empty
end_turn.By contrast
claude-agent-acpsurfaces backend limit errors to the user (e.g."You've hit your org's monthly spend limit ..." as a
-32603). This PR brings agy toparity.
At a Glance
Prior Art & Industry Research
claude-agent-acpalready surfaces backend spend/quota limits as a JSON-RPC-32603with the human-readable cause in the message; this PR makes agy behave thesame way, so the two adapters render consistently in OpenAB.
src/acp/error-normalization.ts: the canonical peer ACP clientnormalizes opaque agent failures into typed, user-facing errors rather than passing
through empty turns. The approach here (detect an empty successful turn, recover the
real cause from the agent's own diagnostics, emit a normalized error) is the same
pattern applied at the adapter boundary.
-32603+ human-readablemessageconvention matches howcodex-acpalreadyreports its distinct
-32603causes (model deprecation, missing peer dep, auth) inthis project.
Proposed Solution
Single file:
agy-acp/src/main.rs.snapshot_agy_logs(): before spawning agy, record the existingcli-*.logfilenames under
<conversations_dir>/../log, so a log written during this turncan be attributed to it.
execute_promptbranch: when agy exits 0 but streamed nothing(
status.success() && !had_updates), calldetect_swallowed_agy_error().detect_swallowed_agy_error(): scan this turn's newcli-*.log(fallback: themost recently modified one), read up to the 3 newest candidates, and extract the
cause via
extract_agy_error_message().extract_agy_error_message(): anchor on the most specific terminal error(
agent executor error:thenmodel unreachable:thenRESOURCE_EXHAUSTED), stripglog's self-wrapped duplicate tail (
<msg>.: <msg>), cap length on a char boundary.{"code": -32603, "message": <cause>}instead of theempty
end_turn. If no matching log line is found, behavior is unchanged (agenuinely-empty successful turn still returns
end_turn), so there are no falsepositives.
Why This Approach
cli.log; the process exit code,stdout/stderr, and the trajectory DB
error_detailsare all empty on a 429(verified on a live pod). So log-scraping is the sole viable signal without an
upstream change to Google's closed
agyCLI.success && !had_updateskeeps the change conservative: it only convertsa turn that already produced nothing for the user, and only when a known error
signature is present.
turn.
Alternatives Considered
error_detailsBLOB: rejected, verified empty on quotafailures; agy doesn't populate it for this class of error.
AGY_EXTRA_ARGSto make agy print / exit non-zero on error:rejected,
agyis a closed-source Google CLI; no such flag is documented andprint-mode swallowing is its behavior.
legitimately-empty responses; requiring a matching log signature avoids false
positives.
agy: not in this project's control; this adapter-levelnormalization is the acpx-style boundary fix.
Validation
Built/tested standalone (the crate is built from its own directory, per
.github/workflows/ci-agy-acp.ymlworking-directory: agy-acp):cargo check: clean (only pre-existingdead_codewarnings, none from this change).cargo test -- --include-ignored --skip e2e: 19 passed; 0 failed; 1 filtered,including 4 new tests:
test_extract_agy_error_message_dewraps_quota_errortest_extract_agy_error_message_none_for_clean_logtest_detect_swallowed_agy_error_reads_new_turn_logtest_detect_swallowed_agy_error_none_when_no_logscargo clippy: no new warnings from the added code (remaining warnings arepre-existing:
adapter.rscomplex type / arg count,db.rs&PathBuf, protobuftest blob builders,
main()collapsible-if).Manual evidence (live OrbStack pod, image
0.9.0-beta.6-antigravity, agy1.0.16):reproduced
agy --print "say hi"on an exhausted quota, observed exit 0, emptystdout/stderr; the
RESOURCE_EXHAUSTED (429) ... Resets in 40h...cause was presentonly in
~/.gemini/antigravity-cli/log/cli-*.log, which is exactly what this changenow recovers and surfaces.
Discord Discussion URL: https://discord.com/channels/1491295327620169908/1491969620754567270/1523983902849630268