CI fails when the external reviewer did not actually review

[blooop]

Why

gate's own comment already argues this case:

a check nobody ran reads exactly like a check that passed, and a gate that covers nothing is the purest form of that

That is what happened to the external reviewer, and nobody noticed for a day and a half. Sourcery answers a quota refusal as a review:

Sorry @blooop, you have reached your weekly rate limit of 500000 diff characters.

A review is what a reader looking for one finds — so this arrives in the same shape as a review that found nothing, satisfies anything checking that a review exists, and says nothing to anyone.

Twenty-six consecutive pull requests merged behind that sentence, from #343 (Aug 22, 15:53) to #376. Among them the largest changes in this repo: #369 (1,631 lines, breaking CLI grammar, shipped as 0.9.0), #367 (1,940 lines, shipped as 0.8.0 and withdrawn again in #376 twenty-two hours later), #371, #370, #362. Across the three days, 29 of 38 pull requests got no external review; three got a review with findings.

The second refusal is worse targeted than the first:

Sorry @blooop, your pull request is larger than the review limit of 150000 diff characters

That one fires on size, so it exempts precisely the changes that least tolerate going unread. #369 is the pull request that got it.

What this adds

An external-review job inside gate's needs. It reads the pull request's reviews and refuses three things: a refusal on quota, a refusal on diff size, and no review at all after two minutes of polling. Sourcery normally answers inside a minute — well inside the shortest job here — but "well inside" is not "before", and a race lost would fail a pull request for the reviewer's latency rather than for anything about the change.

It fails rather than warns, because a warning is precisely what already did not work. The way past it is the no-external-review label (created on the repo), so merging unreviewed is a decision recorded on a named pull request rather than a default that quietly stopped applying.

Where the classification lives

scripts/external_review_verdict.sh, called by the workflow and executed by the test — the same reason scripts/public-api-snapshots.sh is a script and not a run: block. A case statement that exists only inside a workflow is testable only by copying it, and the copy is the half that goes stale.

How it was verified

test/test_external_review_guard.py runs the real script against the real bodies those pull requests received — both refusals verbatim, plus the verbatim "reviewed your changes and they look great" that #330/#331/#333/#334/#337/#342 got, which is the case the guard must not fail and the one a naive "did the review say anything useful" check gets wrong.

Red before green, both directions:

• Neutering the script to cat >/dev/null; echo ok turns the four refusal tests red and leaves the four pass-through tests green.
• Removing external-review from gate's needs turns test_the_gate_requires_the_job red — the same shape of nothing-gating-anything the job exists to catch.

Also replayed the classifier over the archived review bodies of all 38 pull requests from the window: it passes the 6 that were genuine reviews and fails exactly the 26 refusals, distinguishing #369's diff-cap message from the weekly-quota one. pixi run style clean (pylint 10.00/10), yaml.safe_load on the workflow clean.

Note on this pull request's own CI

The external-review job will fail on this pull request, because Sourcery's weekly quota is still exhausted. That is the guard working, on the first change it covers. Apply no-external-review to merge it — which is the intended flow, and makes this PR its own demonstration.

What this does not do

It does not judge review quality, and it does not gate on a human having read anything — only on the external reviewer having answered about the code rather than about its own billing.

🤖 Generated with Claude Code

Summary by Sourcery

Require a substantive external reviewer response before CI can pass, while providing an explicit override for unreviewed merges.

New Features:

• Add an external-review CI job that verifies the external reviewer produced a genuine review before allowing the gate to pass.
• Provide a no-external-review label override for explicitly merging without external review.

Bug Fixes:

• Prevent pull requests from appearing reviewed when the external reviewer returns quota, diff-size, or no-review responses.

Enhancements:

• Move external-review response classification into a reusable script with actionable CI error messages.

CI:

• Require the external-review job as part of the main CI gate and poll for reviewer responses before evaluating them.

Documentation:

• Document the external-review guard, its failure conditions, and the label-based override.

Tests:

• Add coverage for quota refusals, diff-cap refusals, missing reviews, valid reviews, mixed review histories, workflow wiring, and script usage.

[sourcery-ai]

Sorry @blooop, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[sourcery-ai]

Reviewer's Guide

Adds a CI guard that ensures the external Sourcery reviewer actually reviewed the pull request, centralizing the classification logic in a testable script and wiring it into the gate job’s dependencies.

Sequence diagram for external review CI gating

sequenceDiagram
    participant PR as PullRequest
    participant Job as external-review
    participant GitHub as GitHubAPI
    participant Sourcery as Sourcery
    participant Verdict as external_review_verdict.sh
    participant Gate as gate

    Job->>GitHub: gh pr view(PR, labels)
    alt no-external-review label
        Job-->>Gate: exit 0
    else review required
        loop up to 12 polls
            Job->>GitHub: gh api pulls/PR/reviews
            GitHub-->>Job: Sourcery review bodies
            opt no review yet
                Job->>Job: sleep 10
            end
        end
        Job->>Verdict: review bodies on stdin
        alt quota refusal, diff-size refusal, or no review
            Verdict-->>Job: exit 1
            Job-->>Gate: external-review failed
        else genuine review
            Verdict-->>Job: exit 0
            Job-->>Gate: external-review passed
        end
    end
    Gate->>Gate: evaluate external-review in needs

Loading

File-Level Changes

Change	Details	Files
Introduce an external-review CI job that polls Sourcery reviews, honors an override label, and feeds review bodies to a classifier script.	Add external-review job to .github/workflows/ci.yml with GitHub CLI-based polling of Sourcery reviews over ~2 minutes. Short-circuit the job for non-pull_request events and when the no-external-review override label is present, emitting a GitHub notice. Pipe concatenated Sourcery review bodies into an external_review_verdict.sh script for classification. Require the external-review job by adding it to gate.needs so merges are blocked when the guard fails.	.github/workflows/ci.yml
Implement a bash classifier script that decides whether the external reviewer actually reviewed, failing on quota/size refusals or missing reviews and emitting GitHub annotations.	Create scripts/external_review_verdict.sh to read review bodies from stdin and exit 0/1 based on content. Treat empty or whitespace-only bodies as "no external review" and emit a GitHub ::error workflow command including override instructions. Match Sourcery weekly quota and per-diff-cap refusal messages by stable substrings and fail with descriptive error output. Allow any non-refusal body (including "clean" and "with findings" reviews) to pass with a success message.	scripts/external_review_verdict.sh
Add tests that exercise the classifier script against real Sourcery review bodies and assert CI wiring for the external-review job.	Add test_external_review_guard.py to run the bash script via subprocess with real refusal and success message samples. Test behavior for weekly quota refusals, diff-cap refusals, empty bodies, genuine clean reviews, and reviews with findings, including mixed-body cases. Assert that the external-review job is present in ci.yml and included in gate.needs to ensure the guard is required. Assert that the workflow calls ./scripts/external_review_verdict.sh rather than embedding duplicated logic in a run block.	test/test_external_review_guard.py
Document the new external-review guard behavior and override mechanism in the changelog and README.	Update CHANGELOG.md under Unreleased to describe the external-review job, its failure cases, and the no-external-review label escape hatch. Expand README.md CI section to explain why the external-review job exists, what it checks, how the label works, and how the classification script is tested against real refusal bodies.	CHANGELOG.md README.md

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.