Skip to content

[CI] Review guard hard-fails on an unguarded lookup 404 — a lookup failure becoming the verdict its own comments forbid #2309

Closed

Description

@erikdarlingdata

The review guard (claude-review-guard.yml) hard-failed PR #2307 twice on a raw lookup 404 — the exact failure mode its own comments forbid ("a lookup failure must not become a review verdict"), in a call the || true audit missed.

Evidence

  • Run 32038137165 (verify, PR Give the analysis pass a stopping token and classify shutdown residue (#2299) #2307), attempt 1: after the poll loop, the step's runtime output is exactly gh: Not Found (HTTP 404) → Process completed with exit code 1 at 14:17:46Z. Attempt 2 (re-run after everything had settled): identical gh: Not Found (HTTP 404) → exit 1, five seconds in.
  • No ::error title=… annotation exists on either attempt — the only annotation is the bare "Process completed with exit code 1". So none of the script's deliberate verdict branches fired; the step died raw under bash -e inside an unguarded $(gh api …).
  • The review it was guarding demonstrably posted: claude[bot] left an inline comment at 13:51:29Z (attempt-1 window) and an issue comment at 14:17:26Z (head SHA 58b6f96's review, 20 seconds before attempt 1 died). Had the tally been reached, total >= 1 → exit 0.
  • Every lookup the script makes succeeds when replayed with an ordinary token: the workflows/claude-review.yml/runs?head_sha= listing returns 32038137167 completed success, the step-conclusion jq returns success, and all three artifact endpoints return the artifacts above. So the 404 is specific to the job's GITHUB_TOKEN + one endpoint, and the script currently cannot say which.

Why this is worth fixing rather than re-running

A deterministic red on a PR whose review is posted and clean is the "mark people learn to ignore" the guard's own header warns about — it spends the guard's credibility on its weakest link (unguarded plumbing), not its verdict. And because the failing call is unidentified in the output, every occurrence costs a manual log-dig that ends at one context-free line.

Fix direction

  1. Audit every $(gh api …) in the step for the -e assignment-kill: the three artifact tallies (issues/$PR/comments, pulls/$PR/comments, pulls/$PR/reviews), the jobs-id listing feeding the annotations loop, and the pulls/$PR/files lookup are all unguarded today. The contents lookup at the bottom is the only guarded one.
  2. On lookup failure, print WHICH endpoint failed and exit with the script's own philosophy: warn UNCONFIRMED (like the sentinel-absent branch) rather than fail — the verdict branches already reserve exit 1 for evidence-backed conclusions.
  3. Optional: curl -sw '%{http_code}'-style capture or gh api --include on first failure so the log names the endpoint and status without a re-run.

Both attempts' logs: https://github.com/erikdarlingdata/PerformanceMonitor/actions/runs/32038137165

Related lineage: #2229 (review output vanishing — the guard's reason to exist), #2243 (review prompt pointing at a gitignored file). This one is the guard itself.

Activity

  1. erikdarlingdata commented on Aug 17, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed by #2311 (77aaebe) plus plumbing follow-up #2313 (cbdf957). Every catalog lookup now runs through a helper that names the failing endpoint and downgrades to an UNCONFIRMED warning + exit 0 — exit 1 stays reserved for evidence-backed verdicts; the poll loop tolerates transient misses as ordinary not-yet states; a successful call's stderr can no longer pollute captured values (the review's catch on the fix itself); and the helper's own mktemp/cat plumbing degrades instead of killing the step (the review's second catch, #2313). Validated three ways: a stub-gh matrix reproducing the original raw-404 death byte-for-byte and proving every failure combination now warns-and-exits-0 or reaches the verdict; and the fixed guard ran itself green on both PRs' own verify checks — including polling through a live review re-run and counting its artifacts correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions