Skip to content

Fix Promotion Review Findings on Local Pass and Backlog Report - #3406

Merged
ptr727 merged 5 commits into
developfrom
fix-promotion-3405-review-findings
Oct 23, 2026
Merged

ptr727 merged 5 commits into
developfrom
fix-promotion-3405-review-findings

Conversation

@ptr727

@ptr727 ptr727 commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Answers the Copilot round on the develop -> main promotion #3405, whose findings sit on content merged into develop today.

Changes

  • scripts/pr_review.py: wait on a pull request into a branch other than the default printed "its head is covered by an attested local pass" where no attested local pass covered the head, while the verdict returned status=AWAITING_LOCAL_PASS. The note is gated on local_cover, the predicate the poll and the verdict read, so a head with no attestation and an attested head Copilot refused each get a note naming what is still owed. Tests cover both cases and the attested case.
  • .agents/skills/pr-review-conduct/SKILL.md: the heading "Copilot's absence blocks, and is answered elsewhere." contradicted the local-pass exception its own bullet states. It reads "Copilot's absence blocks where no local pass stands in, and is answered elsewhere." The two long sentences under it are split. Mirrors regenerated with build_dist.py.
  • scripts/backlog_metrics.py and scripts/README.md: the README says the report lists open issues by label, while the script counted a fixed subset. The named labels still print first, a zero included, and every other label an open non-handoff issue carries follows them, sorted. The docstring said review cost per merged pull request reads from the report, but Actions minutes and the Copilot gross cover the calendar month while the counts cover the window. The report adds a Copilot review runs per merged PR (ratio) column over the window, and the docstring and README state the month scoping.

Verification

  • python3 -m unittest discover -s tests -> OK
  • uvx ruff@latest format --check scripts tests, uvx ruff@latest check scripts tests, uvx mypy@latest -> clean
  • python3 scripts/build_dist.py --check -> current
  • python3 scripts/prose_lint.py --diff origin/develop -> no violations, and with --check sentence-length -> no violations
  • The new refused-head test fails with the note gated on attested() and passes with local_cover().

Three recorded local-strict-review passes at the frontier tier. The first raised the note's predicate (attested rather than local_cover), a docstring that said "print", and a 47-word sentence, each fixed in 34e23c1. The second raised the missing refused-head test and a 33-word sentence, fixed in 2e0d435. The third raised one finding with the edit budget spent, listed below.

Open findings from the local pass

  • introduced: tests/test_pr_review.py, test_an_attested_head_copilot_refused_is_not_called_covered asserts only a non-zero exit, so a regression from the refusal's exit 41 to AWAITING_LOCAL_PASS (49) would still pass. Fixed in b05e99a, which asserts exit 41 and fails with the note gated on attested(). A fourth recorded pass over that push raised nothing.

Review

Copilot reviewed 2e0d435 and raised three findings, each fixed and answered in its thread: the shared wait note read as sending a refused, already-attested head to attest (5a202fd), the non-zero exit assertion (b05e99a), and the README label sentence that did not say handoff issues are left out (5a202fd). A fifth recorded pass over 5a202fd raised nothing.

🤖 Generated with Claude Code

ptr727 and others added 3 commits October 23, 2026 09:49
The Copilot round on the develop -> main promotion raised these against
content merged into develop today.

- pr_review.py wait told an unattested feature pull request that an
  attested local pass covered its head. The note now says no
  attestation covers the head yet and names the attest step.
- pr-review-conduct's "Copilot's absence blocks" heading contradicted
  the local-pass exception its own bullet states.
- backlog_metrics.py counted a fixed subset of labels while the README
  said open issues by label, and its docstring said review cost per
  merged pull request reads from a report whose cost figures cover the
  month rather than the window. It now counts every carried label after
  the named ones and adds Copilot review runs per merged pull request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The local pass found the new note keyed on attestation alone, while the
poll and the verdict read local_cover, which a Copilot refusal on the
head also defeats. The note now uses local_cover. It also fixes the
count_by_label docstring, which described printing, and splits the
long sentence after the retitled pr-review-conduct heading.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The second local pass found no test for an attested head that Copilot
refused, the case the local_cover predicate exists for, and a 33-word
sentence on a line the retitle touched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings October 23, 2026 16:58
@ptr727

ptr727 commented Oct 23, 2026

Copy link
Copy Markdown
Owner Author

A recorded local strict-review pass covers head 2e0d4353b2cf8f4314c6d32f95f29b35316a7685, the content this pull request carries at that commit against develop, and it recorded 1 finding.

@coderabbitai

coderabbitai Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

A non-zero assertion also passes on AWAITING_LOCAL_PASS, which would
mean the refusal stopped outranking the hold.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ptr727

ptr727 commented Oct 23, 2026

Copy link
Copy Markdown
Owner Author

A recorded local strict-review pass covers head b05e99a48dc6b26cb7581bb6101a5b78706b6b2b, the content this pull request carries at that commit against develop, and it recorded 0 findings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved review findings remain in the refused-head messaging, refusal-specific test assertion, and handoff-label documentation.

3 open issues
What changed in this PR

This pull request fixes promotion review messaging and aligns backlog report labels and Copilot review metrics with their documented behavior.

Changes:

  • Gates the local-pass wait note on local_cover and adds coverage for unattested and refused heads.
  • Reports non-handoff issue labels and Copilot review runs per merged PR.
  • Updates documentation, canonical review guidance, generated mirrors, and the source digest.
File Summary
tests/​test_pr_review.py Tests local-pass messaging and refused-head behavior. [P2 moderate, 4 votes] Assert the refusal-specific exit code 41 rather than merely asserting a non-zero result.
tests/​test_backlog_metrics.py Tests additional labels and the Copilot review-runs-per-merged-PR ratio.
scripts/​README.md Documents label ordering and calendar-month billing scope. [P2 nit, 2 votes] Clarify that handoff issues are excluded from label output.
scripts/​pr_review.py Adjusts the promotion wait note based on local coverage. [P1 moderate, 3 votes] Do not direct an already-attested refused head to publish an attestation; distinguish refusal from a missing attestation or use neutral wording.
scripts/​backlog_metrics.py Counts additional non-handoff labels and renders the new review ratio.
.github/​skills/​pr-review-conduct/​SKILL.md Regenerated GitHub skill mirror.
.claude-plugin/​fleet-skills/​skills/​pr-review-conduct/​SKILL.md Regenerated Claude skill mirror.
.claude-plugin/​fleet-skills/​.source-digests/​pr-review-conduct Updated generated source digest.
.agents/​skills/​pr-review-conduct/​SKILL.md Clarifies the local-pass exception in the canonical review guidance.

Review effort: Lite

Comment thread scripts/pr_review.py Resolved
Comment thread tests/test_pr_review.py Resolved
Comment thread scripts/README.md Resolved
@codecov

codecov Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.61%. Comparing base (d148ffb) to head (5a202fd).

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #3406   +/-   ##
========================================
  Coverage    81.60%   81.61%           
========================================
  Files           19       19           
  Lines        10231    10235    +4     
========================================
+ Hits          8349     8353    +4     
  Misses        1882     1882           
Flag Coverage Δ
python-3.13 81.61% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot on the fix pull request read the shared note as sending an
already-attested head to attest. The note now says which of the two
defeats the local cover. The README states that the label line also
leaves out handoff issues.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ptr727

ptr727 commented Oct 23, 2026

Copy link
Copy Markdown
Owner Author

A recorded local strict-review pass covers head 5a202fd316449af15110aee8d97b68d005c121bc, the content this pull request carries at that commit against develop, and it recorded 0 findings.

@ptr727
ptr727 merged commit 4a77a1d into develop Oct 23, 2026
13 checks passed
@ptr727
ptr727 deleted the fix-promotion-3405-review-findings branch October 23, 2026 17:49
ptr727 added a commit that referenced this pull request Oct 23, 2026
## Summary

Promotes these pull requests from develop. Each carries a maintainer
decision recorded on #3391.

- [#3399](#3399): states
the write guard's threat model and documents two shapes it keeps
allowed, a quoted run-time `-C` value and an `xargs` replace string run
as a shell script (item 1).
- [#3401](#3401): lists a
pre-existing review finding in the pull request body instead of filing
an issue for it (item 2).
- [#3402](#3402): keeps
one standing handoff issue per track, edited in place each round, with a
`close` subcommand that ends a track (item 5).
- [#3400](#3400): adds a
weekly backlog and review-cost report script.
- [#3403](#3403): runs the
Copilot review once per change, at the promotion, with the attested
local pass covering a pull request into develop (item 4).
- [#3404](#3404): aligns
the remaining review rule text with the promotion-only Copilot review
(items 2 and 4).
- [#3406](#3406): fixes
the findings this promotion's first Copilot round raised on the content
above, in the `wait` note for a head no local pass covers, the
pr-review-conduct heading, and the backlog report's label line and cost
scoping.

Closes #3382
Closes #3385

## Pre-existing findings, not filed

- The [Copilot round on
4a77a1d](#3405 (review))
headline says documentation does not consistently describe current-head
Copilot refusal handling. `local_cover` in `scripts/pr_review.py`
refuses an attested head whose current Copilot round is a refusal, and
`wait` exits `41` there. The `wait` entry of the `pr_review.py`
docstring ("An attested head is covered at once.") and the
`scripts/README.md` paragraph on the defined review moments ("Where it
is attested, `wait` polls the head's checks instead.") leave that
exception out. `pr-review-conduct` Merge Gate item 2 states it ("A
refusal is not that coverage"). Both omissions are on `main` already, so
this promotion did not introduce them.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants