Skip to content

Ask an After-Checks Side Reviewer Once CI Settles, and Compare Ask Times in UTC - #3888

Merged
ptr727 merged 4 commits into
developfrom
fix/side-ask-after-checks-and-utc
Nov 5, 2026
Merged

ptr727 merged 4 commits into
developfrom
fix/side-ask-after-checks-and-utc

Conversation

@ptr727

@ptr727 ptr727 commented Nov 5, 2026

Copy link
Copy Markdown
Owner

Why

#3881 left two defects in scripts/pr_review_side.py, and #3885 adds two design items and a documentation gap. Each sits on the review-cutover track (#3877), and the pilot needs the first one fixed before any repository gives CodeRabbit or Qodo when: after-checks.

What

scripts/pr_review.py sits at 2199 of its 2200-line ceiling after this change. The next change that adds logic there has to move some first, as #3880 plans.

Refs #3885, #3887, #3877. The closing keywords belong on the develop -> main promotion PR.

Verification

  • The full suite, uv run --no-project --with-requirements tests/requirements.txt python -m unittest discover -s tests, ran 3539 tests with 11 skipped, all passing, at 6bd9a5c. The test_pr_review* suites pass at the head, 709 tests.
  • The new TestSideAskTiming cases in tests/test_pr_review_policy.py were each run against the pre-change scripts or with their fix reverted, and each one failed. Those are the outlasting-CI ask, the poll hand-off per wait path, the merged-during-poll ask, the fresh digest read, and both UTC offsets.
  • ruff format and check are clean. The pre-commit hook's mypy, prose and EOL gates pass.

Local-review findings, disposed

Three recorded local-strict-review passes at the frontier tier covered the branch. The third, at 08cb68a, raised nothing.

  • introduced, fixed in bc8e6ae: the digest and exit code came from a payload read before the side poll, so a check that went green during it could read as NOT_PICKED_UP and exit 44.
  • introduced, fixed in bc8e6ae: a held head handed the ask a poll, which could delay exit 46 by up to --timeout.
  • introduced, fixed in bc8e6ae: a pull request merged while its checks settled could still be asked.
  • introduced, fixed in bc8e6ae: the usage text and the README 0 row still said a covered head's exit 0 never polls the checks.
  • introduced, fixed in 08cb68a: the usage text and README still said any polling wait awaits the checks after the held-head change.

🤖 Generated with Claude Code

ptr727 and others added 3 commits November 5, 2026 10:08
…mes in UTC

An explicit after-checks CodeRabbit or Qodo read the head's checks once,
after the Copilot poll, so a head whose CI outlasted Copilot's round was
never asked. Where the wait itself polls, the ask now polls the checks with
await_green on the wait's own clock, bounded by --timeout. A wait that stops
without polling still reads them once.

side_asked_on_head compared a comment's UTC createdAt with a commit's
offset-bearing committedDate as text. Both are now parsed with utc_instant.

Also documents the DIRTY and unreadable-rollup holds at exit 44, drops the
PAYLOAD_KEY alias for pr_review_payload.POLICY_KEY, and moves the
post-and-confirm of a conversation comment into one post_comment helper
that both the gate's comment command and the side asks call.

Closes #3885
Closes #3887
Refs #3877

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The local pass over the first commit found four defects in it.

- The digest and exit code came from a payload read before the after-checks
  poll, so a check that went green during it could still read as stuck.
  A wait that polled now re-reads the payload for the digest.
- A held head's own poll already waits out the checks, so it no longer hands
  the side ask a poll. Exit 46 stays immediate.
- A pull request merged while its checks settled could still be asked. The
  re-read after the check wait now checks the state again.
- The exit 0 text still said a covered head never polls the checks.

Refs #3885

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit stopped a held head from handing the side ask a poll,
but the usage text and the README still said any polling wait awaits the
checks. They now name a poll for a Copilot round, and say every other wait,
a held head's included, reads the checks once.

Refs #3885

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings November 5, 2026 18:17
@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@ptr727

ptr727 commented Nov 5, 2026

Copy link
Copy Markdown
Owner Author

A recorded local strict-review pass covers head 08cb68a3f317a4e526b0897d9405a9454b6a48e2, the content this pull request carries at that commit against develop, and it recorded 0 findings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An active P2 finding remains because a sentence in scripts/pr_review_side.py is ungrammatical and obscures the meaning about preserving the committer’s offset.

1 open issue
What changed in this PR

This pull request updates policy-driven side-review timing, UTC timestamp comparisons, comment confirmation, documentation, and related test coverage.

Changes:

  • Polls checks after Copilot waits before requesting eligible side reviewers.
  • Compares commit and comment timestamps as UTC instants.
  • Centralizes comment posting and confirmation, removes the duplicate policy-key alias, and documents exit-44 conditions.
  • Adds coverage for polling, closure, timestamp offsets, holds, and comment confirmation.
  • Review findings include one unresolved polling issue and two documentation nits.

This pull request requires changes.

File Summary
tests/​test_pr_review.py Updates tests to use the canonical policy payload key.
tests/​test_pr_review_policy.py Adds coverage for timing, polling, closure, UTC comparisons, holds, and comment confirmation.
scripts/​README.md Documents after-checks polling and exit-44 conditions.
scripts/​pr_review.py Wires side-review polling, fresh payload reads, and shared comment posting into the wait flow.
scripts/​pr_review_side.py Implements check polling, UTC timestamp handling, and shared comment confirmation.
scripts/​pr_review_policy.py Removes the duplicate payload-key alias.
scripts/​pr_review_payload.py Clarifies ownership of the canonical policy-key constant.

Review effort: Lite

Comment thread scripts/pr_review_side.py Resolved
The side_asked_on_head docstring dropped the noun its possessive stood for.

Refs #3885

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ptr727

ptr727 commented Nov 5, 2026

Copy link
Copy Markdown
Owner Author

A recorded local strict-review pass covers head ea11a9e3af0141ab81283744249cd668f24a88e2, the content this pull request carries at that commit against develop, and it recorded 0 findings.

@codecov

codecov Bot commented Nov 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.13%. Comparing base (b38e084) to head (ea11a9e).
⚠️ Report is 103 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #3888      +/-   ##
===========================================
+ Coverage    81.60%   84.13%   +2.52%     
===========================================
  Files           26       32       +6     
  Lines         9070    10580    +1510     
===========================================
+ Hits          7402     8901    +1499     
- Misses        1668     1679      +11     
Flag Coverage Δ
python-3.13 84.13% <100.00%> (+2.52%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ptr727
ptr727 merged commit 4cc7d92 into develop Nov 5, 2026
13 checks passed
@ptr727
ptr727 deleted the fix/side-ask-after-checks-and-utc branch November 5, 2026 18:37
ptr727 added a commit that referenced this pull request Nov 6, 2026
…t Policy (#3892)

## Summary

Promotes #3881, #3884, #3886, #3888, #3889, and #3891 to `main`.

Closes #3200.
Closes #3885.
Closes #3887.

- **Per-repository review policy (#3881).** `scripts/pr_review.py` reads
which bots review a repository, on which bases, when, and how often,
from the repository's own policy, with the hub default reproducing
today's behavior.
- **Review return on cost (#3886).** `scripts/backlog_metrics.py`
measures reviewer cost per useful finding. It is now a uv script, so the
metrics server's weekly timer must run it as `uv run --no-project
--script scripts/backlog_metrics.py` from this release on.
- **After-checks side reviewer (#3888).** An explicit after-checks
CodeRabbit or Qodo is asked once CI settles, and ask times compare in
UTC.
- **Commit in place for an operational repository (#3889).** The policy
file became `.github/agent-policy.json`, with its schema under `review`
and a `guard` section that lets `gh-write-guard` allow an in-place
commit on a listed branch.
- **Agent policy rendered from the registry (#3891).**
`repo-config/configure.sh` renders `.github/agent-policy.json` from
`registry/repos.json` and applies the bot settings it implies: the
Copilot ruleset rule, CodeRabbit's auto-review keys, and Qodo's
auto-feedback key. Once this is on `main`, `configure.sh check` reports
agent-policy drift on every repository until `apply --checkout` has been
run there.
- **Dependabot (#3884).** The actions-deps group bump.

Each feature pull request carried recorded local strict review passes,
with every finding fixed, declined with evidence, or filed (#3890).

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants