Repository navigation
Review rounds are spent debating prose in comments instead of fixing code #1631
Description
ProjectTemplate's issue count has not gone down after weeks of near constant burndown, see #1311 that is growing the count nor reducing it (yes it is entirely possible that the baseline position does have lots of defects per the current rules and we are simply discovering them and documenting them, then we need to evaluate the ROI of those rules if it enforcing them simply burns time and money vs. fixing real bugs.). Every Pr round I observe spends 90+% of the time debating prose and wording in comments vs. just fixing code. Real bugs are caught at the last moment after all prose arguments have been settled.
E.g. Nine findings across three review passes, every one against the comment I wrote, none against the key. Deleting the comment ended it. I've saved that: config changes ship bare, the measurement goes in the commit and PR body where it's reviewed once, and an issue reference never goes in a carried file. In this case the no prose in comments is a known rule, the no issue pointers in comments is also a known rule, yet this was the offered solution at round 10.
The result is that huge amounts of time is being wasted, huge amounts of tokens are being wasted, and huge amounts of github copilot and actions credits are being wasted.
Analyze all (ptr727 fleet) the recent (two weeks) PR's and determine how many review rounds violated a known rule and then determine why the rule was not followed, and calculate the time spent landing a PR, the average round count, and estimate how much time could have been saved if the rules were followed from the start.
Final step is determining what to do to prevent this, e.g. delete all prose and comment related rules so that it never violates, have a 0-agent comment rule where only humans are allowed to write comments, linter checks only and all rules are encoded in python code and PR agents only review code no prose, remove all governance rules in prose and only use skills, do periodic sweeps to keep the fleet in sync vs. enforcing conformity in real-edit-time, etc.
Activity
ptr727 commentedon Sep 15, 2026
Step 1: two weeks of fleet review rounds, measured, and the decision
Method: every PR merged 2026-09-01 to 2026-09-15 in the 25 registry repos, fetched by GraphQL with reviews, review threads, and Copilot's collapsed suppressed findings; every thread first-comment and suppressed item classified by target (what it is about) and status (why it exists relative to the rules) in 50-item batches with five hand-labelled sentinels each (sentinel agreement 93% target, 98% status; a blind 72-item re-classification agreed 94% on target, 89% on status, 97% on prose-versus-not). Rounds are head commits that received a bot review. Local review passes, token counts, and dollars come from the Claude Code session transcripts on the machine that drove 235 of the 321 feature and promotion PRs, priced at list API rates. The scripts are scratch and land later as the weekly metric under #1324.
Window 2026-09-01 to 2026-09-15, merged PRs across the 25 registry repos: 542 = 254 feature + 67 promotion (develop->main) + 221 Dependabot. 21 promotions are byte-identical re-reviews of one feature PR. Findings: 1437 (1125 threads + 312 Copilot suppressed), 1437 classified, 0 unclassified.
1. Scope by repository
2. Review rounds
A GitHub round is one head commit that received at least one bot review. A local round is one review subagent dispatched in a Claude session linked to the PR (visible only for PRs driven from this machine).
Copilot reviews vs Copilot review requests disagree on 5 of 542 PRs, each a request Copilot never answered: ESPHome-Config#146, Financial-Modeling#280, Financial-Modeling#285, Financial-Modeling#227, Financial-Modeling#228.
3. Findings, all classified (feature + promotion PRs)
Introduced by the PR: 93% yes, 7% pre-existing, 0% unknown. Disposition: fixed 609, none 538, declined 120, deferred 110, false 60. Prose targets (comment-text + doc-wording): 18%. Prose targets or prose rules: 20%. Real defects (
correctness): 78%.By bot: | bot | findings | correctness | prose target | finding-wrong |
| --- | --- | --- | --- | --- |
| copilot-pull-request-reviewer | 773 | 77% | 26% | 3% |
| coderabbitai | 401 | 87% | 6% | 4% |
| qodo-code-review | 262 | 69% | 14% | 7% |
| qodo-free-for-open-source-projects | 1 | 100% | 0% | 0% |
Classifier: 30 gold items hand-labelled, 5 seeded per 50-item batch as sentinels. Low-confidence labels: 302 of 1437.
4. The known-rule findings (
prose-rule-no-gate)Why the rule was not followed, one cause per finding:
5. Cost, and what following the known rules from the start would have saved
Extra rounds (every round after the first) on feature + promotion PRs: 495 rounds, 620 h of round-to-round wall clock, 844 bot reviews. Linked-session spend on these PRs: $15,188 (transcript tokens at list API rates, 235 PRs).
A round is saveable when every finding of the previous round was on text the PR introduced and (strict) violated a known prose rule with no gate, or (wide) was any non-defect finding on a comment, wording, or a prose rule.
For scale: every extra GitHub round interval on linked PRs, saveable or not, holds $10,124 of session spend (345 intervals), against $15,208 linked to PRs in total, so most of a PR's spend lands after its first bot review. All 108 in-window sessions on this machine, main thread and subagents, price at $15,421 at list API rates (per-model rates taken from Claude Code's own cost record, applied to transcript token counts, since that record omits subagents). The 52 sessions with a duration record spent $7,236 over 384 session-hours, about $19 per session-hour; a fan-out session running many subagents at once burns an order of magnitude faster, which is the shape a metered API key sees.
Local review rounds: every review subagent dispatched in an in-window session on this machine, identified by its dispatch description. 1172 of 1285 subagents were review passes, costing $3,328 in their own tokens. On the 235 linked feature + promotion PRs: 1123 review passes (median 2 per PR, a multi-PR session's passes split evenly) against 579 GitHub rounds, and $3,322 of the $15,188 linked spend went to the review subagents themselves, before the main thread's cost of reading and disposing of what they returned. Their per-finding classification is not recoverable from transcripts, so their prose share is not measured; the #1630 case (three passes, nine findings, all on one comment, $25 and 1.9 h for a two-line config change) is the measured instance.
6. Prevention options, scored on this window
7. Undercounts and unknowns
8. What the numbers say, and the decision
The thesis that 90% of review-round time is prose does not hold for the bot rounds. Fleet-wide, 78% of the 1,437 bot findings are real defects, 18% land on a comment or on wording, and 5% (73) are instances of a known prose rule that has no gate. Only 30 of the 495 extra GitHub rounds (6%) were caused solely by non-defect prose findings, and following every known prose rule from the start would have saved 6 to 30 rounds, 1 to 7 hours, 11 to 37 bot reviews, 18 to 87 Actions runs, and $98 to $533 over the two weeks.
Where the rules were broken, they were broken with the rule in context. 47 of the 73 known-rule findings came from sessions that had loaded the style skill or read the style doc before writing the text; 1 had not. Restating, relocating, or deleting the rules therefore changes little: the agent knows the rule and writes the comment anyway. Only a mechanical gate or removing the surface (no agent-written comments) changes that.
The cost is real, and it sits in the loop after the first review, most of it local. 82% of PR-linked session spend ($10.1k of $15.2k) lands after a PR's first bot review. 1,172 local review passes ran in the window (1.5 per push) and cost $3.3k in their own tokens before the main thread's disposal work, which is where a nine-findings-on-one-comment loop like #1630 spends its $25 and 1.9 hours. The local passes' findings are not classifiable from transcripts, so their prose share is not measured; the bots' share is, and it is not 90%.
Recommendation, in order of measured effect per unit of change:
prose_lint: a diff that adds a comment line in code or config fails, overridable by a PR label. Removes 117 findings (8%), including every Set MD024 to Sibling Scope in the Carried Markdown Config #1630-class loop, mechanically.sentence-splitandsentence-lengthon by default (22), and no re-review request on a promotion PR whose file fingerprint equals its feature PR's (20 rounds, 31 bot reviews).Not recommended: deleting the prose rules (2% of findings cite one) and moving governance into skills only (bounded at 1 finding).
Decision (ptr727, 2026-09-15): items 1 to 4, all of them. One local diff pass per push and the whole-unit canonical pass moved to a scheduled weekly sweep; a zero-agent-comment gate in
prose_lintwith a label override; a PR-title gate,sentence-splitandsentence-lengthon by default, and no re-review request on a byte-identical promotion; and Copilot and Qodo told, as CodeRabbit already is, not to report wording on Markdown. Each lands as its own PR. #1312 is done in substance (#1330 committed the stop rule and two-round budget without naming it) and #1313 is partial (#1328 named it, andreports/canonical-review.jsonis still a tracked file that 53 commits touched in this window); item 1 supersedes both.Stop Claiming a Measurement #1631 Says It Could Not Make
Retract the Same Overclaim on the Two Carried Surfaces That Kept It
Sweep a Bounded Slice of the Backlog, So New Content Has a Reader
Sweep the Carried-Content Read Weekly Instead of Gating Every Push (#…
ptr727 commentedon Sep 17, 2026
Decision item 3, the sentence rules: dropped, both stay opt-in
Decision (ptr727, 2026-09-16):
sentence-splitandsentence-lengthare not moved intoDEFAULT_RULES. Both stay opt-in. The other two parts of item 3, the PR-title gate and the no-re-review-on-a-byte-identical-promotion skip, are unaffected and still stand.The scoring above counted this option's benefit (22 findings, 2%) and not its cost. Measured on this repository at
91e0fd2:sentence-splitsentence-lengthExisting text is not the exposure, since the gate is diff-scoped. What the cost actually is: replaying the last 12 commits on
developthrough the two rules, scoped to each commit's own diff, 7 of the 12 would have failed, the largest at 159sentence-splitplus 221sentence-lengthon one change.sentence-splitsentence-lengthThe cause is structural rather than a backlog that shrinks.
sentence-splitasks for one sentence per line, and every carried skill file is hard-wrapped at about 100 columns, so nearly every sentence in one wraps. Enabling it by default is therefore a decision to reflow 123 files to semantic line breaks, which is a house-style change nobody has taken.docs/fleet-map.mdalready records the prior call that promotion intoDEFAULT_RULESwaits "if and when the corpus converges", and on this measurement it has not.Reproduce either column with a detached worktree at the commit and
python3 scripts/prose_lint.py . --diff <commit>^1 --check <rule>.Report Only a Comment That Opens Its Line, and Delete the Layer Six A…
Refuse a Comment Line a Change Adds, Unless the Pull Request Says It …