[ptr727]

Why

The PR bots are moving from automatic to policy-driven review, per the review-cutover track (#3877). That move needs before and after figures for what each review returns. This adds them to scripts/backlog_metrics.py, so a baseline can be captured before any repository changes its .github/pr-review.json.

What

Two new tables in the weekly report, read from the window's merged pull requests through one paged GraphQL search per repository, newest created first, capped at 500 with a reached cap named.

• Per bot: the policy trigger and the branch it was read at, review requests split into review-requested events by a bot actor and everything else, and rounds per reviewed pull request. CodeRabbit's reviewed heads are classed as automatic, explicit, or unclassified from the commit time, the ask and request times, and the first real review. Qodo's rounds are counted as the gate counts them, from its asks and its one standing comment, with no per-head classes. Threaded findings, findings a clean local attestation missed, and findings on all-green heads follow.
• Per repository: commits per merged pull request, local attestation findings, failing heads, and failing heads after a clean local attestation. Promotion pull requests are left out and counted apart.

The report now runs as a uv script, uv run --no-project --script scripts/backlog_metrics.py, with the same inline metadata as pr_review.py. That lets it import the review gate's own readers rather than copies: attestation per read commit, page_flag for cut connections, reviewer_nodes with refusal_of for reviewed heads, side_asks and side_command for asks, side_rounds for Qodo, and VOCABULARIES for the logins. The server timer's command has to change to that form, with uv on the host, before the next weekly run.

The policy is read at develop where it exists, since feature pull requests are gated there, and otherwise at the default branch. It goes through pr_review_policy.fetch, so a file the gate refuses prints error and a file it takes as absent prints the hub default's triggers marked (default). The gate's absent test, a 404 or a 422, moves from pr_review.py into gh_graphql.py as said_absent, so both share it. A change to the file inside the window is noted. A failed default-branch, develop, policy, or history read still writes the report, names that read on stderr, and exits 1. A failed review read still writes the backlog and billing tables, says the review figures were unavailable, and exits 1.

Verification

• uv run --no-project --with-requirements tests/requirements.txt python tests/test_backlog_metrics.py: 79 tests pass. Each fix was reverted on its own and its test failed.
• The full suite under uv run --no-project --with-requirements tests/requirements.txt python -m unittest discover -s tests: 3532 tests pass.
• A live run, uv run --no-project --script scripts/backlog_metrics.py --repo ProjectTemplate --days 7, exits 0 in 41 seconds. Its per-repository table is identical before and after the switch to the gate's attestation.
• ruff check and format are clean. mypy reports nothing in the changed files beyond what develop already reports.

Local-review findings, disposed

• introduced, design, left by the maintainer's decision: the reviewed-heads set in review_pr restates pr_review_rounds.reviewed_heads's filter rather than calling it, since the report needs the set and the gate returns a count. Exposing the set there would take pr_review_rounds.py past its 700-line ceiling, so Share the Attestation Marker and Reviewer Logins Between the Gate and the Metrics Report #3880 exposes it when it splits that file.
• introduced, fixed in be564a6: classify_rounds compared a committedDate keeping the committer's offset with UTC times as strings. pr_review_checks.utc_instant now converts each to UTC first, and the gate's age builds on it in 787b3b3, so one parser reads every GitHub stamp.
• introduced, fixed in be564a6: graphql duplicated the GraphQLError conversion walked already makes.
• introduced, fixed in 787b3b3: the thread loop restated the gate's thread_author, and a truncation note named the PR cap whichever cap was reached.
• introduced, fixed in 9c3240b: the README's commit-date caveat named the wrong count, its markdown-it-py list left out backlog_metrics.py, and two docstrings credited the UTC conversion rather than the parse with ordering stamps.
• introduced, fixed in e4a655b: no test built a review-request event naming CodeRabbit, so dropping events from its round triggers passed every case.

Pre-existing findings, filed

• scripts/pr_review_side.py:117, correctness: side_asked_on_head makes the same mixed-offset string comparison. Filed as Compare Side-Reviewer Ask Times With the Head's Commit Time in UTC #3887.

Dropped findings

• spec/audit.py and .github/actions/repo-gate/repo_gate.py keep their own HTTP-status parse beside said_absent, a consolidation on text this change did not touch.

Follow-ups

• Share the Attestation Marker and Reviewer Logins Between the Gate and the Metrics Report #3880 shares what remains between the gate and this report.
• The 4-week baseline, --days 28, is captured once this merges and the timer is moved.

🤖 Generated with Claude Code

[ptr727]

A recorded local strict-review pass covers head 82e8d1514cbd62e3bdef20e74ef84725593c08e3, the content this pull request carries at that commit against develop, and it recorded 7 findings.

[coderabbitai]

Important

• Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

• Source mode: Auto
• Repository: ptr727/ProjectTemplate
• Branch: feature/review-metrics
• File: .coderabbit.yaml
• Commit: e4a655b
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Advanced
• Run ID: 8cfc3d2c-8ccd-4b1b-bb45-27bae4516efa

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[Copilot]

🟡 Changes recommended

Multiple active findings show that the report can diverge from the gate and mishandle review, policy-history, and GraphQL failures, while one test is nondeterministic.

11 open issues

• Match trigger comments to the gate's request rules · New
• Share the gate's attestation parser · New
• Count Qodo standing-comment rounds · New
• Exclude refusal reviews from Copilot round counts · New
• Reuse the shared absent-response predicate · New
• Report policy-history read failures · New
• Convert cursor-walk errors to the report failure path · New
• Preserve backlog output when review collection fails · New
• Share reviewer-login lookup with the gate · New
• Reuse the per-merged formatter · New
• Inject a deterministic runner in the null-node test · New

What changed in this PR

This pull request adds review return-on-cost metrics to the weekly backlog report, including policy state, review requests, rounds, findings, attestations, failing heads, and repository-level results. It also centralizes absent-response detection and expands documentation and test coverage.

Changes:

• Adds paged GraphQL collection and aggregation for per-bot and per-repository review metrics.
• Reads review policies from the appropriate repository branch and reports policy errors or history changes.
• Adds Markdown and HTML report rendering, documentation, and extensive tests.
• Unresolved findings remain around metric accuracy, parser and gate divergence, error handling, failure isolation, and test determinism.

File	Description
tests/​test_backlog_metrics.py	Adds fixtures and coverage for review metrics, policy handling, pagination, failures, and rendering.
scripts/​README.md	Documents the new review return-on-cost tables, measurement rules, and failure behavior.
scripts/​pr_review.py	Routes absent-response detection through the shared policy helper.
scripts/​pr_review_policy.py	Adds the shared GitHub absent-response predicate.
scripts/​backlog_metrics.py	Implements GraphQL review collection, policy reads, metric aggregation, error handling, and report rendering.

_{Review effort: Lite}

[codecov]

Codecov Report

❌ Patch coverage is 99.66667% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 84.11%. Comparing base (b38e084) to head (e4a655b).
⚠️ Report is 102 commits behind head on develop.

Files with missing lines	Patch %	Lines
scripts/backlog_metrics.py	99.64%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3886      +/-   ##
===========================================
+ Coverage    81.60%   84.11%   +2.50%     
===========================================
  Files           26       32       +6     
  Lines         9070    10572    +1502     
===========================================
+ Hits          7402     8893    +1491     
- Misses        1668     1679      +11     

Flag	Coverage Δ
python-3.13	84.11% <99.66%> (+2.50%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ptr727]

A recorded local strict-review pass covers head 26c4c01f4b8ecf1be6d3a90eac853f73a799970f, the content this pull request carries at that commit against develop, and it recorded 2 findings.

[ptr727]

A recorded local strict-review pass covers head 787b3b32112529f666f32b890b481316fef10f9f, the content this pull request carries at that commit against develop, and it recorded 3 findings.

[ptr727]

A recorded local strict-review pass covers head e4a655bb80a7a16aa03fc9c9f6b6109105fa971c, the content this pull request carries at that commit against develop, and it recorded 0 findings.