Skip to content

docs(review): a diff never shows the prose it just falsified - #1003

Merged
lilyshen0722 merged 3 commits into
mainfrom
docs/review-checklist-comment-survives-the-diff
Aug 18, 2026
Merged

docs(review): a diff never shows the prose it just falsified#1003
lilyshen0722 merged 3 commits into
mainfrom
docs/review-checklist-comment-survives-the-diff

Conversation

@lilyshen0722

Copy link
Copy Markdown
Contributor

Adds rule 15 to docs/development/review-checklist.md. Docs-only, additive; rule 14 verified byte-identical.

What earned it

Three instances today, all in careful PRs, all by the author who knew the subject best:

the change the sentence it falsified
#1001 removed the 30-minute pending delete models/OnboardingSilenceEpisode.ts:18–19"AgentEvent pending rows are deleted at AGENT_EVENT_STALE_PENDING_MINUTES (default 30)", cited as the reason its event snapshot is NOT OPTIONAL
#1002 collapsed a double log emission the comment it added in the same diff"each failure already prints twice", as the reason to stamp both sinks; its own new test now forbids that behaviour
#997 the AX entry drafted the same hour, cataloguing this identical defect on two product surfaces

The third is the tell, and it is why this is worth a rule rather than two review comments: a room that could name the failure in the product could not see it in its own diffs.

Why it is a reviewer's rule

The author is structurally the last person who will notice, because they hold the pre-change model in their head as the thing being fixed. In #1002 that is sharper than distance — the stale sentence is inside the diff, five lines from the change, freshly written, still describing the before-state, because it was drafted while the bug was live.

Hence this file rather than REVIEW.md, per the split that file already states: authors pre-PR there, reviewers mid-PR here.

The check

Mechanical and cheap. For each behaviour the PR alters, git grep the constant, env var, status value, threshold or tool name it touches, and read every prose hit — comments never appear in a diff of the code they describe.

Two riders are included because each changes what the fix should be:

  1. The stale sentence is usually load-bearing — the stated justification for a design decision ("WHY X IS NOT OPTIONAL"), so the next reader inherits a false model of the system, not a cosmetic inaccuracy.
  2. Its conclusion is often still correct for a different surviving reason. Both instances above are like this: both sinks do still need stamps, the snapshot may still be worth keeping. So rewrite the reason rather than deleting the rule — and say which, or the next reader removes a constraint that still holds.

Ran the rule on this diff

Since it would be a poor rule to break while adding it: git grep review-checklist turns up CLAUDE.md:173, REVIEW.md:20, REVIEW.md:242, docs/development/README.md:16 and ADR-019. All describe the file by nature ("incident-derived rules, each naming the defect that earned it") rather than by length or contents, and ADR-019's citation of rule 9 by number is unaffected by an append. Nothing goes stale.

lilyshen0722 and others added 3 commits August 18, 2026 06:22
Three times on 2026-08-18, in careful PRs by the author who knew the
subject best, a change invalidated a comment it did not touch:

  #1001  removed the 30-minute pending delete, while
         models/OnboardingSilenceEpisode.ts:18-19 went on citing "pending
         rows are deleted at AGENT_EVENT_STALE_PENDING_MINUTES (default
         30)" as the reason its event snapshot is NOT OPTIONAL.

  #1002  collapsed a double log emission, while the comment it ADDED in
         the same diff argued "each failure already prints twice" as the
         reason to stamp both sinks -- a behaviour its own new test now
         forbids.

  #997   the AX entry drafted the same hour, cataloguing this identical
         defect on two product surfaces.

The third is the tell. A room that could name the failure in the product
could not see it in its own diffs, because the author holds the
pre-change model in their head as the thing being fixed. That makes it a
reviewer's rule rather than an author's, which is why it lands here
rather than in REVIEW.md.

The check is mechanical and cheap: for each behaviour a PR alters, git
grep the constant, env var, status value, threshold or tool name it
touches, and read every prose hit. Two riders included because both
changed what the fix should be -- the stale sentence is usually the
stated justification for a design decision, so the next reader inherits
a false model rather than a typo; and its conclusion is often still
correct for a different surviving reason, so rewrite the reason rather
than deleting the rule.

Ran the rule against this diff: nothing states the checklist's length or
enumerates its contents (ADR-019 cites rule 9 by number and is unaffected
by an append), so no prose here goes stale. Rule 14 verified byte-identical.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Added by @pod-architect, from the case that earned the rule. Having found
the onError block asserting a premise its own commit removed, they went
back and read the collapse-block comment forty lines up — and it was
clean: past tense throughout, describing behaviour the code now prevents
rather than claiming it still happens.

Two comments written minutes apart about the same change, one aged badly
and one didn't. So finding the first says nothing about the second in
either direction, and a reviewer who stops at the first hit closes the
file with the other still wrong. The sweep is per-comment.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… stop

Rule 11 already says to verify against the current head. Tonight showed
the cost is asymmetric in a way worth stating, because it changes what
the rule asks for.

Five crossings in thirty minutes on one PR. Three were findings that
turned out already-known: someone re-read them and moved on, cost a
paragraph each. Two were assertions that a blocker remained outstanding
when the head under review had already fixed it -- and one of those was
offered as the reason to hold a live-defect fix while the defect was
still shipping.

Same staleness, opposite cost. The difference is not accuracy, it is
force: a stale observation is noise, a stale blocker stops work, and the
author on the other side cannot tell which they are looking at.

So the rule is not "re-verify more often" -- nobody sustains that across
a long review. It is: re-resolve the head before asserting anything that
would hold a merge, and accept that non-blocking observations will
sometimes arrive already-answered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@samxu01
samxu01 force-pushed the docs/review-checklist-comment-survives-the-diff branch from 8c97f60 to db34029 Compare August 18, 2026 13:22
@lilyshen0722
lilyshen0722 merged commit 2702ddf into main Aug 18, 2026
10 checks passed
@lilyshen0722
lilyshen0722 deleted the docs/review-checklist-comment-survives-the-diff branch August 18, 2026 21:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant