Skip to content

Re-review only what changed since the last review, and say when it's done - #77

Merged
polewskm merged 14 commits into
mainfrom
issue-24
Oct 31, 2026
Merged

polewskm merged 14 commits into
mainfrom
issue-24

Conversation

@polewskm

Copy link
Copy Markdown
Contributor

A review of a pull request the bot has reviewed before is now a re-review. It compares the pull request's merge-base diff now with the same diff at the last reviewed head (from the summary marker), and reviews only the hunks that differ, plus what the author took out. Merging main in or rebasing changes nothing by itself. If the old head can't be fetched (a force-push), the run does a full review instead. A re-review runs only correctness on Sonnet. It sees the earlier bot findings and the replies in their threads as JSON data, and it reports a status for each unresolved finding. It posts only blockers and should-fixes on lines changed since the last review, never nits. Its summary says "No further findings above nits" when nothing at or above should-fix is left. The first review is unchanged.

All the logic is in the new src/rereview.ts. Other files only gain small hooks: exported helpers in review.ts, gather.ts and threads.ts, fetchCommit in fetch-head.ts, and the wiring in main.ts. Hermetic tests in test/rereview.test.ts use local git repos: a merge of main, a rebase plus force-push with the old head still fetchable or pruned, the posting filter, the roster (with a fake agent), the summary and verdict, and the thread reader. Two eval cases replay #29 and #30 round 8, the latter with main merged in. Their checks are worked out apart from the engine. I didn't run the paid eval (node evals/run.ts --only re-review-declined-trigger,re-review-merge-of-main). npm run typecheck and npm test (101 tests) pass.

Decisions:

  • "Changed" works at the line level inside a hunk that differs, not hunk-wide. A file the PR adds is one hunk, so a one-line edit to it would otherwise mark the whole file. Lines the earlier diff already had count as reviewed. A hunk that only shrank counts at its first line. Lines the author took out are listed to the reviewer as <taken_back>, so a fix made by deleting a line still gets its finding checked.
  • Earlier findings come from GraphQL reviewThreads, the only API that says which threads are resolved. All earlier findings are context. Only unresolved ones get a status. Replies are included only from OWNER, MEMBER or COLLABORATOR, as for the description. If the threads can't be read, the re-review runs without them and logs a warning.
  • A /review of the head already reviewed does a full review, since the command means review it again.
  • A run whose reviewers all failed now records the previously reviewed head in its summary marker, or none. Otherwise the next re-review would skip changes nobody reviewed. This also applies to a full review where every reviewer failed.
  • A re-review skips context selection, rule checks, selection and synthesis. Its summary is written by the engine, not by Opus.

Closes #24

…done

A review of a pull request the bot has reviewed before compares the pull
request's merge-base diff now with its merge-base diff at the last reviewed
head, and reviews only the hunks that differ, plus what the author took back,
so a merge of the default branch or a rebase changes nothing by itself. The
last reviewed head is fetched if it isn't in the checkout; when it can't be,
the review is a full one.

Only correctness reviews them, on Sonnet, with the earlier findings and the
replies in their threads as data, and checks each unresolved finding. It posts
only blockers and should-fixes on lines changed since the last review, never
nits, and its summary says "No further findings above nits" when nothing at or
above should-fix is left. A run whose reviewers all failed records the head
reviewed before it. The first review is unchanged.

Two eval cases replay re-reviews of #29 and of #30 with main merged in.

Closes #24
@polewskm
polewskm deployed to code-review October 31, 2026 04:47 — with GitHub Actions Active

@ncode-review ncode-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 NCode Review at c06ad83: 12 new findings on the changed lines.

Comment thread src/rereview.ts Resolved
Comment thread src/rereview.ts Resolved
Comment thread src/main.ts Resolved
Comment thread src/rereview.ts Resolved
Comment thread src/rereview.ts Resolved
Comment thread src/rereview.ts Resolved
Comment thread src/rereview.ts Resolved
Comment thread src/rereview.ts Resolved
Comment thread src/rereview.ts Resolved
Comment thread src/rereview.ts Resolved
@ncode-review

ncode-review Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

🤖 Review complete

Verdict: 💬 Comment — This PR adds re-reviews. When the bot has reviewed a pull request before, it compares the merge-base diff now with the one at the last reviewed head, runs only correctness on the hunks that differ, checks the earlier findings against the current code, and posts nothing below should-fix. If every reviewer fails, the summary records the previous reviewed head. The design holds together: the diff-of-diffs approach copes with merges from the default branch and with force-pushes, and every failure path falls back to a full review or to a verdict that won't approve. The real risks are format knowledge copied from threads.ts, a JSON parse that can throw past the fallback, and new wiring in main.ts that no test covers.

8 reviewers run — 8 findings (run details)

Reviewer Model Findings Top
synthesis opus-5-5 8 🟠
architecture sonnet-5-5 1 🟠
completeness sonnet-5-5 0 🟢
conventions sonnet-5-5 1 🟡
correctness sonnet-5-5 0 🟢
hygiene haiku-5-5 1 🟡
rule-check sonnet-5-5 3 🟠
security sonnet-5-5 0 🟢
tests haiku-5-5 3 🟠

Skipped: api-compat, data-model, infrastructure, intent, performance

Findings

Trial: findings are posted as review threads and in the check run. $0.80 on Amazon Bedrock, 95s, at b2d89c6. (how to interact)

… test its wiring and fallbacks

A re-review whose earlier review threads can't be read now says so and gives a comment verdict instead of approving. The choice between a re-review and a full review, the recorded head and the re-review trace move from main.ts into runReview, which is tested. Thread reads take the finding from the first comment and the replies from the newest 50. Adds tests for the comparison and thread-read fallbacks, fetchCommit with a non-SHA, the generated-files summary and the size caps; reuses severityRank, errorText and the configured default tier; unexports parseHunks.
@polewskm

Copy link
Copy Markdown
Contributor Author

/review

@ncode-review ncode-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 NCode Review at 6dce13d: 7 new findings on the changed lines.

Not on a changed line:

  • 🟡 F7 Nit: severityRank is defined twice src/results.ts:20. results.ts keeps its own copy of the severityRank map that review.ts now exports and rereview.ts imports. Fix: Import severityRank from ./review.ts in results.ts and delete the local const.

Comment thread src/rereview.ts Resolved
Comment thread src/rereview.ts Resolved
Comment thread evals/run.ts Resolved
Comment thread src/rereview.ts Resolved
Comment thread src/rereview.ts Resolved
Comment thread src/main.ts Resolved
Comment thread CHANGELOG.md Resolved
@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: answering #77 (comment) and #77 (review).

@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: Round 2 answered. No blocker or should-fix is left open; the remaining threads are nits, answered above. Recommend merging once you've resolved them.

@polewskm
polewskm marked this pull request as ready for review October 31, 2026 06:46
@polewskm
polewskm deployed to code-review October 31, 2026 06:46 — with GitHub Actions Active
@polewskm
polewskm enabled auto-merge October 31, 2026 06:46

@ncode-review ncode-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 NCode Review at b2d89c6: 4 new findings on the changed lines.
3 more were already posted on this pull request.

Not on a changed line:

  • 🟡 F6 Nit: severityRank is defined twice src/results.ts:40. results.ts keeps a local severityRank that is identical to the one review.ts now exports and rereview.ts imports. Fix: Import severityRank from review.ts in results.ts and delete the local copy.

Comment thread src/rereview.ts Resolved
Comment thread src/rereview.ts Resolved
Comment thread src/rereview.ts Resolved
Comment thread src/rereview.ts Resolved
# Conflicts:
#	CHANGELOG.md
#	evals/cases.json
#	evals/run.ts
#	src/review.ts
# Conflicts:
#	evals/run.ts
#	src/main.ts
#	src/review.ts
A re-review now sends what it would post through #41's verification stage, as a first review does. A should-fix verification can't settle becomes a nit and isn't posted, and a re-review whose findings verification refuted or made nits never approves or says it's done.
@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: merged main (with #72's verification stage) into this branch in 4f13b4e. Conflicts in src/review.ts (Charters keeps both reReviewPrompt and verificationPrompt), src/main.ts (the trace keeps both verification and the re-review trace) and evals/run.ts (the report keeps both refuted and reReview) were resolved by keeping both sides. ReReview now fills Review's new refuted/verification fields.

Decision, in 5fff0ec: with CODE_REVIEW_VERIFICATION=on, a re-review's blockers and should-fixes go through verification too, as a first review's do. Refuted findings are dropped and listed under "Refuted by verification". A blocker it can't settle is posted as a should-fix question. A should-fix it can't settle becomes a nit, so it isn't posted, because a re-review posts no nits. As with #72, verification never turns a re-review into an approval: when it refuted findings or made them nits, the verdict is at most Comment and the summary doesn't say "No further findings above nits". With verification off, nothing changes. README, CHANGELOG and two new tests in test/rereview.test.ts cover it. npm run typecheck and npm test (180 tests) pass.

README keeps both main's cost-ceiling paragraph and this branch's re-review sentence in the verification paragraph. The run-ceiling tests' charters get the reReviewPrompt this branch added to Charters.
@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: answering #77 (review).

Also merged main (#61's cost ceiling) into this branch in 33540c6. The README conflict keeps both main's cost-ceiling paragraph and this branch's re-review sentence. The run-ceiling tests' Charters gets reReviewPrompt. npm run typecheck and npm test (194 tests) pass.

@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: Round 3 answered. The review-round limit (3) is reached; no blocker or should-fix is left open. The new F1 repeats round 2's nit, and F3, F4 and F5 are nits, all declined above. F3 (response.json() outside the try/catch) and the F1 parser move could be follow-up issues. Recommend merging once you've resolved the threads.

# Conflicts:
#	CHANGELOG.md
#	src/main.ts
#	src/review.ts
…erun on Bedrock

runAgent now returns every call of a request as records, two when the credits ran out and it reran on Bedrock. The re-review's correctness call logs and keeps all of them, and a test checks that it gets the run's fallback. parsePosted now also returns the finding's id, which main's new marker test didn't expect.
# Conflicts:
#	README.md
#	evals/run.ts
#	src/review.ts
…d, as main's first review does

Main's #71 drops such findings before posting and gives Results a dropped list; the re-review now does the same, lists every changed file under the same heading, and its tests carry the dropped list.
@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: merged main (#73 eval precision, #71 dropped findings) into this branch in eac9e0a, then adapted the re-review in 8e2d2d2.

  • evals/run.ts: the re-review fields (reviewed, earlier, mergeIntoHead) moved onto EvalCase in main's new evals/cases.ts. Re-review cases now get precision judging and reviewCostUsd like every other case, and keep their reReview report and off-the-changed-lines violations.
  • src/review.ts: kept main's fileList(gathered, inDiff?, elsewhere). It stays exported for rereview.ts. completeList is exported too.
  • README.md: kept the re-review eval paragraph and main's Precision, Comparing and Growing paragraphs.
  • Main's Never post a finding that says a changed file isn't changed #71 adds a required dropped field to Review/Results and leaves out findings that say a changed file isn't changed. A re-review now does the same before it numbers its findings, so they show in its summary comment and check run like a first review's. Its prompt lists the changed files under main's "every file this pull request changes" heading. There's a new test in test/rereview.test.ts.

npm run typecheck and npm test (268 tests) pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Re-reviews don't converge: each round raises new nits on unchanged code and can reverse the round before

1 participant