docs(review): a diff never shows the prose it just falsified - #1003
Merged
lilyshen0722 merged 3 commits intoAug 18, 2026
Conversation
This was referenced Aug 18, 2026
Three times on 2026-08-18, in careful PRs by the author who knew the subject best, a change invalidated a comment it did not touch: #1001 removed the 30-minute pending delete, while models/OnboardingSilenceEpisode.ts:18-19 went on citing "pending rows are deleted at AGENT_EVENT_STALE_PENDING_MINUTES (default 30)" as the reason its event snapshot is NOT OPTIONAL. #1002 collapsed a double log emission, while the comment it ADDED in the same diff argued "each failure already prints twice" as the reason to stamp both sinks -- a behaviour its own new test now forbids. #997 the AX entry drafted the same hour, cataloguing this identical defect on two product surfaces. The third is the tell. A room that could name the failure in the product could not see it in its own diffs, because the author holds the pre-change model in their head as the thing being fixed. That makes it a reviewer's rule rather than an author's, which is why it lands here rather than in REVIEW.md. The check is mechanical and cheap: for each behaviour a PR alters, git grep the constant, env var, status value, threshold or tool name it touches, and read every prose hit. Two riders included because both changed what the fix should be -- the stale sentence is usually the stated justification for a design decision, so the next reader inherits a false model rather than a typo; and its conclusion is often still correct for a different surviving reason, so rewrite the reason rather than deleting the rule. Ran the rule against this diff: nothing states the checklist's length or enumerates its contents (ADR-019 cites rule 9 by number and is unaffected by an append), so no prose here goes stale. Rule 14 verified byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Added by @pod-architect, from the case that earned the rule. Having found the onError block asserting a premise its own commit removed, they went back and read the collapse-block comment forty lines up — and it was clean: past tense throughout, describing behaviour the code now prevents rather than claiming it still happens. Two comments written minutes apart about the same change, one aged badly and one didn't. So finding the first says nothing about the second in either direction, and a reviewer who stops at the first hit closes the file with the other still wrong. The sweep is per-comment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… stop Rule 11 already says to verify against the current head. Tonight showed the cost is asymmetric in a way worth stating, because it changes what the rule asks for. Five crossings in thirty minutes on one PR. Three were findings that turned out already-known: someone re-read them and moved on, cost a paragraph each. Two were assertions that a blocker remained outstanding when the head under review had already fixed it -- and one of those was offered as the reason to hold a live-defect fix while the defect was still shipping. Same staleness, opposite cost. The difference is not accuracy, it is force: a stale observation is noise, a stale blocker stops work, and the author on the other side cannot tell which they are looking at. So the rule is not "re-verify more often" -- nobody sustains that across a long review. It is: re-resolve the head before asserting anything that would hold a merge, and accept that non-blocking observations will sometimes arrive already-answered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
samxu01
force-pushed
the
docs/review-checklist-comment-survives-the-diff
branch
from
August 18, 2026 13:22
8c97f60 to
db34029
Compare
lilyshen0722
deleted the
docs/review-checklist-comment-survives-the-diff
branch
August 18, 2026 21:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds rule 15 to
docs/development/review-checklist.md. Docs-only, additive; rule 14 verified byte-identical.What earned it
Three instances today, all in careful PRs, all by the author who knew the subject best:
models/OnboardingSilenceEpisode.ts:18–19— "AgentEvent pending rows are deleted atAGENT_EVENT_STALE_PENDING_MINUTES(default 30)", cited as the reason its event snapshot is NOT OPTIONALThe third is the tell, and it is why this is worth a rule rather than two review comments: a room that could name the failure in the product could not see it in its own diffs.
Why it is a reviewer's rule
The author is structurally the last person who will notice, because they hold the pre-change model in their head as the thing being fixed. In #1002 that is sharper than distance — the stale sentence is inside the diff, five lines from the change, freshly written, still describing the before-state, because it was drafted while the bug was live.
Hence this file rather than
REVIEW.md, per the split that file already states: authors pre-PR there, reviewers mid-PR here.The check
Mechanical and cheap. For each behaviour the PR alters,
git grepthe constant, env var, status value, threshold or tool name it touches, and read every prose hit — comments never appear in a diff of the code they describe.Two riders are included because each changes what the fix should be:
Ran the rule on this diff
Since it would be a poor rule to break while adding it:
git grep review-checklistturns upCLAUDE.md:173,REVIEW.md:20,REVIEW.md:242,docs/development/README.md:16andADR-019. All describe the file by nature ("incident-derived rules, each naming the defect that earned it") rather than by length or contents, and ADR-019's citation of rule 9 by number is unaffected by an append. Nothing goes stale.