Skip to content

Session Handoff [review-cutover]: Step-2 Reports In, Step 3 Left to Peer Sessions, Check-In 1 Next #3877

New issue
New issue
Open
Open
Session Handoff [review-cutover]: Step-2 Reports In, Step 3 Left to Peer Sessions, Check-In 1 Next#3877
Labels
handoffA link in the session handoff chain, one open issue per track

Description

@ptr727
ptr727
opened on Nov 4, 2026

This track moves the PR bots (Copilot, CodeRabbit and Qodo) from automatic to policy-driven review, to cut reviewer cost per useful finding. The maintainer revised the plan on 2026-11-04:

  • Account level: automatic review is off for all three bots.
  • Per repository: .github/agent-policy.json says which bots review, on which bases, when, and how often. scripts/pr_review.py obeys it, and repo-config/configure.sh applies the bot-side settings.
  • Skills and rule text: they stay agnostic to whether reviews are automatic.
  • Measurement: scripts/backlog_metrics.py measures the effect.

Each repository's policy file is rendered from registry/repos.json. scripts/agent_policy.py write writes the files into a checkout. configure.sh apply writes the live settings and rulesets. check diffs against the render, reading the files from the default branch. The central settings repositories are written by scripts/central_config.py.

Four repositories review by explicit request with the same values (#3924): the hub, PlexCleaner, Financial-Modeling and HomeAutomation-Config. The hub's override names Qodo too, the other three leave Qodo on the fleet default:

Bot Trigger Bases When Rounds
Copilot explicit, gates default branch after-checks per push, capped at 3
CodeRabbit explicit main, develop after-checks once
Qodo auto on the three, explicit on the hub main, develop on-open per push

Next steps

  1. Values check-in 1, on or after 2026-11-13. Review each bot's values against the weekly scripts/backlog_metrics.py output and the 28-day baseline, which is on the maintainer's machine, so ask for it. Fold in the test reports on this issue (the round-18 and round-19 comments). Qodo answered on its own on PlexCleaner#1220, the first repository where it does, so weigh its findings there. Put each change to the maintainer as a question with a recommendation. A change is a registry override PR, then each repository writes its files, then configure.sh apply with the maintainer's go-ahead.
  2. Collect the one report still owed: the Copilot 3-round cap, which needs a main-base PR with more than one push. Read it from the repositories' own PRs. Also read the outcome of PlexCleaner#1220's CodeRabbit and Qodo rounds once it closes.
  3. Promote Financial-Modeling and PlexCleaner to main. The maintainer chose on 2026-11-08 to leave this to the peer sessions working those repositories, after their open PRs land. A next round checks whether it happened rather than doing it. Done for each when configure.sh check exits 0. Before any promotion, list its unverified commits (0 on both at round 19), and rewrite an unsigned one rather than bypassing. EspDinIoT is held on develop as a work in progress.
  4. Values check-in 2, on or after 2026-11-20, or one week after step 6 turns the toggle off, whichever is later.
  5. Add the central config files with central_config.py write in a clone of each settings repository once central_config.py precondition exits 0, then a PR there, each on the maintainer's go-ahead.
  6. Turn the account toggles off and verify (maintainer, after step 5). Copilot's personal "Automatic Copilot code review" toggle is still on, and wait notes it on every PR. Then move other repositories by their registry entry, each with its two check-ins.
  7. Close Stop pr_review.py Requesting Copilot on Feature Pull Requests into Develop #3758 with the pilot's verification. Comment on Decide: How the Coordinated Cutover to Explicit Bot-Review Requests Is Triggered, Scheduled, and Gated #3826 that the per-repo design superseded its single window. Close this track with close.

External blockers

  • The peer sessions in Financial-Modeling and PlexCleaner own step 3. The maintainer's word still gates each promotion.
  • The maintainer's word for each settings-repository write.
  • Copilot's personal auto-review toggle is UI-only and still on.
  • The metrics baseline is on the maintainer's machine.
  • CodeRabbit's plan rate limit binds explicit rounds on every repository, private ones included.

Internal dependencies

  • Step 3 before step 5. central_config.py withholds the config file until every live registry repository carries its own file on its default branch.
  • Check-in 1 before other repositories move to explicit. Its answers decide which values they inherit.

State

  • Hub: main at 24535332, release 2.0.1366. develop at 0ed3fe89 holds nothing main lacks from this track. No hub PR is open.
  • HomeAutomation-Config is on main with its policy file. Financial-Modeling (9 ahead) and PlexCleaner (31 ahead) still have no policy file on main. Financial-Modeling's only open main PR is a Dependabot bump (repo-worktree's hook step installs into the git hooks dir every worktree shares #1816).
  • Open peer PRs at round 19: PlexCleaner#1220 and Financial-Modeling#1884, both into develop.
  • Settings repositories: both config files are still withheld, since the step-3 precondition is unmet.
  • This machine: the round-18 hub clone is deleted. This round's clone, a detached main checkout named ProjectTemplate-hub-<epoch> under the fleet worktree directory, is left for the maintainer to delete, or for the next session to remove. Claude Code loads the primary checkout, still on develop.
  • Peer sessions on this host are working PlexCleaner, Financial-Modeling and HomeAutomation-Config. Leave their repositories to them.

Parked decision queue

0 open decision issues (count: 0).

What the last round did

  • Read the step-2 evidence from PlexCleaner#1220 and HomeAutomation-Config#1787 and recorded it as a comment on this issue. PlexCleaner's report is no longer owed. A rate-limited CodeRabbit released the merge live after Read CodeRabbit's Rate-Limit Reply With Markup Around Its Sentence #3930, 8 minutes against about an hour before. Qodo answered on its own for the first time, on PlexCleaner.
  • Read step-3 readiness: no policy files on either main, 0 unverified commits on either develop.
  • The maintainer chose to leave step 3 to the peer sessions. Deleted the round-18 hub clone after checking it was clean.

What not to repeat

  • Don't promote Financial-Modeling or PlexCleaner from this track while their peer sessions are live. The maintainer chose to wait for them.
  • Don't read a wait exit 0 with threads open as a contract breach. In the usage text, "with no thread open" applies only to the attested held-head branch. Worth rewording the next time that usage text changes.
  • Don't expect an explicit Copilot round on a develop PR. Copilot's bases is @default.
  • Don't add a docstring line to scripts/pr_review.py casually. It sits at the 2200-line size ceiling.
  • Don't set CodeRabbit off for a private repository. It reviews them, and the maintainer wants it kept, explicit and once.
  • Don't trust docs/pr-reviewer-reference.md on which bots review private repositories until Reviewer Reference Says CodeRabbit Skips Private Repositories, but It Reviews Them #3925 lands.
  • Don't run central_config.py or agent_policy.py with bare python3, or uv run without --no-project in a hub worktree. Use uv run --no-project --script.

New learnings

  • Recorded on this issue: the round-19 evidence comment.
  • A pre-existing edge case, not filed: a check run with no startedAt gets an empty since, so a rate-limit comment of any age clears it. Nothing in the fleet posts such a check today.

Activity

  1. ptr727
    added
    handoffA link in the session handoff chain, one open issue per track
    on Nov 4, 2026

  2. ptr727 commented on Nov 5, 2026

    @ptr727
    ptr727
    on Nov 5, 2026
    OwnerAuthor

    Round 1, closed 2026-11-04: Session Handoff [review-cutover]: Coordinated Cutover to Explicit Bot Reviews, Preparation PR Next

    This track carries the coordinated cutover from automatic to explicit bot reviews for Copilot, CodeRabbit and Qodo. The work is #3758, held until the cutover, with the plan decided on #3826 (Option 1). The maintainer chose it as the next work and is pausing other agents for it.

    Next steps

    1. Land the behavior-neutral preparation in one feature PR into develop. Done when it merges with the switch defaulting to "automatic review on", so no agent's behavior changes. It holds four parts.
      • One hub-hosted switch that says whether automatic review is still on. Decide: How the Coordinated Cutover to Explicit Bot-Review Requests Is Triggered, Scheduled, and Gated #3826 suggests a field in scripts/reviewers/*.json or spec/. Pick one place, so every reader agrees.
      • An explicit request path for CodeRabbit and Qodo in scripts/pr_review.py, beside Copilot's request_copilot_review and the wait --request flag. Today Copilot is the only reviewer the script can request.
      • A merge gate that refuses a head with no explicit request for each configured reviewer, active only while the switch says automatic review is off. Tests cover both switch states.
      • Skill and rule text in drive-pr, pr-review-conduct and GOVERNANCE.md "PR Review Etiquette" naming both states, keyed on the switch. Regenerate the generated trees per skill-lifecycle.
    2. Promote and release the preparation. Drive it with drive-pr to a mergeable develop -> main promotion PR. The merge and the release each need the maintainer's go-ahead per merge-and-release.
    3. Write the cutover runbook into this handoff before the window. List each setting, where it lives, the exact action, and how to confirm it took. It covers:
      • the switch-flip PR;
      • Copilot's account-level automatic review, which the maintainer turns off;
      • reviews.auto_review.enabled: false in each repository carrying a .coderabbit.yaml;
      • Qodo's automatic review.
        Confirm each repository from the live tree, not from memory.
    4. Run the cutover in a window the maintainer names, with no session mid-drive on the old workflow. Done when the switch is flipped and released, all three automatic reviews are off, and a fresh feature PR into develop shows no bot review_requested event that pr_review.py did not send.
    5. Close Stop pr_review.py Requesting Copilot on Feature Pull Requests into Develop #3758 with that check as evidence, then close this track with close.

    External blockers

    • The cutover window, the switch flip, and every settings change are the maintainer's. Steps 1 through 3 are safe to do meanwhile, since the switch defaults to today's behavior.
    • Copilot's account-level automatic review cannot be read through the API. Only the step 4 check, a fresh PR's timeline, confirms it is off.

    Internal dependencies

    • Step 1 before step 4: agents fetch scripts/pr_review.py from the hub when they use it. They need the switch and the explicit request paths already on main when the switch flips, or a drive stalls with no review requested.
    • Step 2 before step 4, for the same reason: agents read main, not develop.
    • Step 3 before step 4, so the window is spent acting rather than discovering.

    State

    Parked decision queue

    No open decision issues at the time of writing (count: 0).

    What the last round did

    What not to repeat

    • Do not try to fix this in run_wait. The 2026-11-01 burndown worker on feature/burn-3758 showed run_wait already requests nothing on a PR into a non-default branch. The Copilot request lands from GitHub one to two seconds after the PR is created, before any script runs (Stop pr_review.py Requesting Copilot on Feature Pull Requests into Develop #3758 comments).
    • Do not turn off any bot's automatic review on its own, or before the window. Agents on the current workflow depend on it, and the maintainer requires all three to change together.

    New learnings

    None beyond what #3758 and #3826 already record.

  3. ptr727
    changed the title [-]Session Handoff [review-cutover]: Coordinated Cutover to Explicit Bot Reviews, Preparation PR Next[/-] [+]Session Handoff [review-cutover]: #3881 Open With Its First Round Answered, Metrics Branch Waiting On It[/+] on Nov 5, 2026

  4. ptr727 commented on Nov 5, 2026

    @ptr727
    ptr727
    on Nov 5, 2026
    OwnerAuthor

    Round 2, closed 2026-11-05: Session Handoff [review-cutover]: #3881 Open With Its First Round Answered, Metrics Branch Waiting On It

    This track carries the move of the PR bots, Copilot, CodeRabbit and Qodo, from automatic to policy-driven review, to cut reviewer cost per useful finding. The maintainer revised the plan on 2026-11-04:

    • Account level: automatic review is off for all three bots.
    • Per repository: a policy file .github/pr-review.json says which bots review, on which bases, when, and how often. scripts/pr_review.py obeys it, and repo-config/configure.sh applies its bot-side settings.
    • Skills and rule text: they stay agnostic to whether reviews are automatic.
    • Measurement: the effect is measured in scripts/backlog_metrics.py.

    This revision supersedes round 1's single hub switch and single coordinated window (#3826 Option 1).

    Next steps

    1. Drive Read a Per-Repository Review Policy Before Asking Any Bot for a Review #3881 to merge into develop. It holds the policy file, the reader, Copilot fully governed, CodeRabbit and Qodo asked once per head when explicit (never gating, by the maintainer's decision), and the trigger-agnostic text. Done when its five open introduced findings each have an outcome and it is merged.
      • The findings are listed in its body, in the section "Open findings from the local strict pass".
      • The two Copilot policy edge cases, explicit with gate: false and off with a partial history, are the substantive ones. Refusing explicit with gate: false in the reader and schema is the likely answer for the first. For the second, check the policy before holds() forces a round.
      • Each fix commit needs a recorded local-strict-review pass, then a push, then attest.
    2. Push feature/review-metrics and open its PR, after Read a Per-Repository Review Policy Before Asking Any Bot for a Review #3881 merges. Its README cites spec/pr-review.schema.json and spec/pr-review.default.json, which Read a Per-Repository Review Policy Before Asking Any Bot for a Review #3881 adds.
      • Rebase onto develop first, and redraft the body from the branch's commits.
      • The body lists four open introduced findings from its third local pass:
        • no column prints BotStats.explicit_rounds;
        • a duplicated sentence in the scripts/README.md backlog_metrics paragraph;
        • a failed history read prints "policy file unreadable" to stderr, and the docstring leaves that exit case out;
        • a failed develop probe shows the default branch in "Policy branch" instead of n/a.
      • Then capture a 4-week baseline with --days 28 before any repository changes its policy.
    3. PR B: repo-config/configure.sh applies and checks the policy.
      • Copilot's copilot_code_review rule in the develop and main rulesets comes from the policy rather than from main.json. This needs a jq edit in both apply_ruleset and check_ruleset.
      • apply --checkout <path> writes only the auto-trigger keys of .coderabbit.yaml and .pr_agent.toml into a local checkout. The maintainer chose the local-checkout form. The caller commits through the ordinary PR flow.
      • check reads both files through the contents API and reports drift.
      • Update repo-config/README.md, which still describes today's fixed rule.
    4. Probe CodeRabbit's central-config repo on a personal account, with the maintainer's go-ahead. Create a minimal ptr727/coderabbit with reviews.auto_review.enabled: false, and install CodeRabbit on it. Then open a test PR on a public repo above CodeRabbit's 10-star gate that carries no .coderabbit.yaml.
    5. PR D: generate ptr727/coderabbit and ptr727/pr-agent-settings from the hub, in the build_dist.py pattern, with an audit check. It extends Generate the Qodo Settings Repository From the Hub So Fleet Rules Are Authored Once and Auditable #1464, and the maintainer chose hub-generated settings repos.
      • Qodo's key is [config] disable_auto_feedback = true, confirmed in upstream pr_agent/settings/configuration.toml.
      • Creating either repo is an outward write needing the go-ahead at the time.
    6. Promote and release Read a Per-Repository Review Policy Before Asking Any Bot for a Review #3881, the metrics PR, and PR B together. The merge and the release each need the maintainer's go-ahead per merge-and-release.
    7. Pilot on ProjectTemplate.
      • Every repo staying automatic first gets its per-repo override applied.
      • Then ProjectTemplate's .github/pr-review.json moves to explicit, through a PR followed by configure.sh apply --checkout.
      • Then the maintainer turns off Copilot's personal "Automatic Copilot code review" toggle, and the settings repos are created.
      • Verify with a fresh PR timeline. Run a week, compare against the baseline, then move other repos by their policy file. Rollback is reverting that file.
    8. Close Stop pr_review.py Requesting Copilot on Feature Pull Requests into Develop #3758 with the pilot's verification. Comment on Decide: How the Coordinated Cutover to Explicit Bot-Review Requests Is Triggered, Scheduled, and Gated #3826 that this per-repo design superseded its single window. Close this track with close.

    External blockers

    • Settings changes need the maintainer's go-ahead at the time: each settings change, creating the two settings repos, the Copilot personal toggle, and the CodeRabbit UI fallback. Steps 1 to 3 are safe meanwhile, since the hub default policy reproduces today's behavior.
    • Copilot's personal auto-review toggle is UI-only and still on. Every PR in the hub gets an automatic Copilot request at open until step 7, which Read a Per-Repository Review Policy Before Asking Any Bot for a Review #3881 showed again.

    Internal dependencies

    State

    Parked decision queue

    No open decision issues at the time of writing (count: 0). The maintainer answered this session's questions in-session:

    • the per-repo policy is a JSON file in the repo;
    • configure.sh writes bot files into a local checkout;
    • the account defaults go in hub-generated settings repos;
    • CodeRabbit and Qodo are simplified to ask-once and never gate.

    What the last round did

    What not to repeat

    • Do not make the merge wait on CodeRabbit or Qodo, or auto-fill their missed starts. Neither leaves a reliable record of which push it reviewed, and three fix rounds on that path each produced new defects until the maintainer chose ask-once, never-gate. A future gate for them should start from a per-head record of the bot's answer, not from timestamps.
    • Do not use a check's start or a suite's creation as the "ask was for this head" stamp. Both postdate the push, so an ask made before them reads as stale and is re-sent. side_asked_on_head keys on committedDate for that reason.
    • Do not add a base filter to the Copilot auto-request. held already excludes uncovered bases. The filter skips the round wait must force on a partial or truncated history.
    • Do not patch gql out in a test without attaching the default policy. GqlCase.answer now attaches it. A test that stages payloads without it skips the path every live run takes, which is how the base-filter regression passed the suite.
    • scripts/pr_review.py sits at its 2,200-line size ceiling. New policy logic goes into scripts/pr_review_side.py or scripts/pr_review_policy.py, never into the gate.

    New learnings

    None beyond what the code, #3881's body, and "What not to repeat" above record.

  5. ptr727
    changed the title [-]Session Handoff [review-cutover]: #3881 Open With Its First Round Answered, Metrics Branch Waiting On It[/-] [+]Session Handoff [review-cutover]: #3881 Answered Two Local Passes, 13 Findings Open for a Fresh Push[/+] on Nov 5, 2026
  6. ptr727
    added a commit that references this issue on Nov 5, 2026

    Read a Per-Repository Review Policy Before Asking Any Bot for a Review (

    62f8b04

  7. ptr727 commented on Nov 5, 2026

    @ptr727
    ptr727
    on Nov 5, 2026
    OwnerAuthor

    Round 3, closed 2026-11-05: Session Handoff [review-cutover]: #3881 Answered Two Local Passes, 13 Findings Open for a Fresh Push

    This track carries the move of the PR bots, Copilot, CodeRabbit and Qodo, from automatic to policy-driven review, to cut reviewer cost per useful finding. The maintainer revised the plan on 2026-11-04:

    • Account level: automatic review is off for all three bots.
    • Per repository: a policy file .github/pr-review.json says which bots review, on which bases, when, and how often. scripts/pr_review.py obeys it, and repo-config/configure.sh applies its bot-side settings.
    • Skills and rule text: they stay agnostic to whether reviews are automatic.
    • Measurement: the effect is measured in scripts/backlog_metrics.py.

    Next steps

    1. Answer Read a Per-Repository Review Policy Before Asking Any Bot for a Review #3881's 13 open local-pass findings in a fresh push, then drive it to merge into develop. The findings are numbered in its body, in "Open findings from the local strict pass". Finding 14 is declined with the evidence stated there. Done when each has an outcome, the review gate passes on the head, and it is merged.
      • Fix the behavior defects first, 1 to 6:
        • wait exits 41 on a head refusal while status reads coverage=local.
        • The exit-49 line still says "a branch other than the default".
        • checks_hold reads green between needs: stages, so reuse runs_open the way held_checks_open does.
        • The after-checks side ask blocks the Copilot poll.
        • Re-read before the Copilot request that follows await_green.
        • The explicit-trigger note fires on wait's own earlier request.
      • Then the text: finding 7, the maxRounds description, and finding 8, the skill, Copilot-instructions, and README statements copilot_settled now waives. Regenerate the skill trees with scripts/build_dist.py.
      • Then findings 9 to 13: consolidations, GATE_SOURCES, and a ceiling for pr_review_policy.py.
      • Each fix commit needs a recorded local-strict-review pass, then a push, then attest. The budget is two edit rounds per push.
    2. Push feature/review-metrics and open its PR, after Read a Per-Repository Review Policy Before Asking Any Bot for a Review #3881 merges. Its README cites spec/pr-review.schema.json and spec/pr-review.default.json, which Read a Per-Repository Review Policy Before Asking Any Bot for a Review #3881 adds.
      • Rebase onto develop first, and redraft the body from the branch's commits.
      • The body lists four open introduced findings from its third local pass:
        • no column prints BotStats.explicit_rounds;
        • a duplicated sentence in the scripts/README.md backlog_metrics paragraph;
        • a failed history read prints "policy file unreadable" to stderr, and the docstring leaves that exit case out;
        • a failed develop probe shows the default branch in "Policy branch" instead of n/a.
      • Its code reads the policy, so check it against BotPolicy.capped, never_waited_on, and the explicit-must-gate refusal Read a Per-Repository Review Policy Before Asking Any Bot for a Review #3881 added since.
      • Then capture a 4-week baseline with --days 28 before any repository changes its policy.
    3. PR B: repo-config/configure.sh applies and checks the policy.
      • Copilot's copilot_code_review rule in the develop and main rulesets comes from the policy rather than from main.json. This needs a jq edit in both apply_ruleset and check_ruleset.
      • apply --checkout <path> writes only the auto-trigger keys of .coderabbit.yaml and .pr_agent.toml into a local checkout, the form the maintainer chose. The caller commits through the ordinary PR flow.
      • check reads both files through the contents API and reports drift.
      • Update repo-config/README.md, which still describes today's fixed rule.
    4. Probe CodeRabbit's central-config repo on a personal account, with the maintainer's go-ahead. Create a minimal ptr727/coderabbit with reviews.auto_review.enabled: false, install CodeRabbit on it, and open a test PR on a public repo above CodeRabbit's 10-star gate that carries no .coderabbit.yaml.
    5. PR D: generate ptr727/coderabbit and ptr727/pr-agent-settings from the hub, in the build_dist.py pattern, with an audit check. It extends Generate the Qodo Settings Repository From the Hub So Fleet Rules Are Authored Once and Auditable #1464, and the maintainer chose hub-generated settings repos. Qodo's key is [config] disable_auto_feedback = true. Creating either repo is an outward write needing the go-ahead at the time.
    6. Promote and release Read a Per-Repository Review Policy Before Asking Any Bot for a Review #3881, the metrics PR, and PR B together. The merge and the release each need the maintainer's go-ahead per merge-and-release.
    7. Pilot on ProjectTemplate.
      • Every repo staying automatic first gets its per-repo override applied.
      • Then ProjectTemplate's .github/pr-review.json moves to explicit, through a PR followed by configure.sh apply --checkout.
      • Then the maintainer turns off Copilot's personal "Automatic Copilot code review" toggle, and the settings repos are created.
      • Verify with a fresh PR timeline. Run a week, compare against the baseline, then move other repos by their policy file. Rollback is reverting that file.
    8. Close Stop pr_review.py Requesting Copilot on Feature Pull Requests into Develop #3758 with the pilot's verification. Comment on Decide: How the Coordinated Cutover to Explicit Bot-Review Requests Is Triggered, Scheduled, and Gated #3826 that this per-repo design superseded its single window. Close this track with close.

    External blockers

    Internal dependencies

    State

    Parked decision queue

    No open decision issues at the time of writing (count: 0). This round the maintainer chose to hand the next fix round to a fresh session.

    What the last round did

    What not to repeat

    • Do not add logic to scripts/pr_review_rounds.py. It is at 699 of its 700-line ceiling, and pr_review.py is at 2192 of 2200. This round fit only by moving a decision onto BotPolicy and cutting a docstring. New logic goes into scripts/pr_review_policy.py, scripts/pr_review_side.py, or a new module, never into a raised ceiling.
    • Do not restore a line with a sed pattern after a negative-control revert. and not reviewer_requested(pr) occurs in both held and owed, and the restore rewrote both. Revert by line number, or by git diff against the commit.
    • Make the waiver one predicate. Fixing the history waiver alone (1cd983c2) left Copilot's refusal, partial round, and pending request on the head still blocking, and the next pass found it. Each further consumer of "no round is coming" must read copilot_settled. Finding 1, wait's refusal exit, is the consumer still missing.
    • Do not make the merge wait on CodeRabbit or Qodo, or auto-fill their missed starts. Neither leaves a reliable record of which push it reviewed. Three earlier fix rounds on that path each produced new defects, until the maintainer chose ask-once, never-gate.
    • Do not add a base filter to the Copilot auto-request. held already excludes uncovered bases. The filter skips the round wait must force on a partial or truncated history.
    • Do not patch gql out in a test without attaching the default policy. GqlCase.answer attaches it, and a fake gql must also accept the policy keyword.

    New learnings

    None beyond what the code, #3881's body, and "What not to repeat" above record.

  8. ptr727
    changed the title [-]Session Handoff [review-cutover]: #3881 Answered Two Local Passes, 13 Findings Open for a Fresh Push[/-] [+]Session Handoff [review-cutover]: #3881 Merged, Metrics PR and #3885 Next[/+] on Nov 5, 2026
  9. ptr727
    added a commit that references this issue on Nov 5, 2026

    Measure PR Review Return on Cost in the Backlog Report (#3886)

    ba118f1

  10. ptr727 commented on Nov 5, 2026

    @ptr727
    ptr727
    on Nov 5, 2026
    OwnerAuthor

    Round 4, closed 2026-11-05: Session Handoff [review-cutover]: #3881 Merged, Metrics PR and #3885 Next

    This track carries the move of the PR bots, Copilot, CodeRabbit and Qodo, from automatic to policy-driven review, to cut reviewer cost per useful finding. The maintainer revised the plan on 2026-11-04:

    • Account level: automatic review is off for all three bots.
    • Per repository: a policy file .github/pr-review.json says which bots review, on which bases, when, and how often. scripts/pr_review.py obeys it, and repo-config/configure.sh applies its bot-side settings.
    • Skills and rule text: they stay agnostic to whether reviews are automatic.
    • Measurement: the effect is measured in scripts/backlog_metrics.py.

    Next steps

    1. Push feature/review-metrics and open its PR. Read a Per-Repository Review Policy Before Asking Any Bot for a Review #3881 is merged into develop, so its spec files exist there now. Done when the PR is merged into develop.
      • Re-isolate into a fresh worktree per repo-worktree, then rebase onto develop. Only scripts/README.md overlaps Read a Per-Repository Review Policy Before Asking Any Bot for a Review #3881's files. Redraft the body from the branch's commits.
      • The body lists four open introduced findings from its third local pass:
        • No column prints BotStats.explicit_rounds.
        • A duplicated sentence in the scripts/README.md backlog_metrics paragraph.
        • A failed history read prints "policy file unreadable" to stderr, and the docstring leaves that exit case out.
        • A failed develop probe shows the default branch in "Policy branch" instead of n/a.
      • Its code reads the policy. Check it against BotPolicy.capped, never_waited_on, copilot_settled, the explicit-must-gate refusal, and pr_review_payload.bot_entry. Reuse those rather than re-reading the payload.
      • Each fix commit needs a recorded local-strict-review pass, then a push, then attest. The budget is two edit rounds per push.
      • Then capture a 4-week baseline with --days 28 before any repository changes its policy.
    2. Fix Ask an After-Checks Side Reviewer Once CI Settles, and Document the Exit 44 Holds #3885, the open findings Read a Per-Repository Review Policy Before Asking Any Bot for a Review #3881 merged with. The after-checks side ask is the behavior defect. The exit-44 holds are a doc gap, and two design items go with them. It can run beside step 1, since it touches pr_review*.py and step 1 touches backlog_metrics.py. Coordinate on scripts/README.md.
    3. PR B: repo-config/configure.sh applies and checks the policy.
      • Copilot's copilot_code_review rule in the develop and main rulesets comes from the policy rather than from main.json. This needs a jq edit in both apply_ruleset and check_ruleset.
      • apply --checkout <path> writes only the auto-trigger keys of .coderabbit.yaml and .pr_agent.toml into a local checkout, the form the maintainer chose. The caller commits through the ordinary PR flow.
      • check reads both files through the contents API and reports drift.
      • Update repo-config/README.md, which still describes today's fixed rule.
    4. Probe CodeRabbit's central-config repo on a personal account, with the maintainer's go-ahead. Create a minimal ptr727/coderabbit with reviews.auto_review.enabled: false, and install CodeRabbit on it. Then open a test PR on a public repo above CodeRabbit's 10-star gate that carries no .coderabbit.yaml.
    5. PR D: generate ptr727/coderabbit and ptr727/pr-agent-settings from the hub, in the build_dist.py pattern, with an audit check. It extends Generate the Qodo Settings Repository From the Hub So Fleet Rules Are Authored Once and Auditable #1464, and the maintainer chose hub-generated settings repos. Qodo's key is [config] disable_auto_feedback = true. Creating either repo is an outward write needing the go-ahead at the time.
    6. Promote and release Read a Per-Repository Review Policy Before Asking Any Bot for a Review #3881, the metrics PR, and PR B together. The merge and the release each need the maintainer's go-ahead per merge-and-release.
    7. Pilot on ProjectTemplate.
      • Every repo staying automatic first gets its per-repo override applied.
      • Then ProjectTemplate's .github/pr-review.json moves to explicit, through a PR followed by configure.sh apply --checkout.
      • Then the maintainer turns off Copilot's personal "Automatic Copilot code review" toggle, and the settings repos are created.
      • Verify with a fresh PR timeline. Run a week, compare against the baseline, then move other repos by their policy file. Rollback is reverting that file.
    8. Close Stop pr_review.py Requesting Copilot on Feature Pull Requests into Develop #3758 with the pilot's verification. Comment on Decide: How the Coordinated Cutover to Explicit Bot-Review Requests Is Triggered, Scheduled, and Gated #3826 that this per-repo design superseded its single window. Close this track with close.

    External blockers

    • Settings changes need the maintainer's go-ahead at the time. That covers each settings change, the two settings repos, the Copilot personal toggle, and the CodeRabbit UI fallback. Steps 1 to 3 are safe meanwhile, since the hub default policy reproduces today's behavior.
    • Copilot's personal auto-review toggle is UI-only and still on. Every PR in the hub gets an automatic Copilot request at open until step 7.

    Internal dependencies

    State

    Parked decision queue

    No open decision issues at the time of writing (count: 0).

    What the last round did

    What not to repeat

    • Do not add logic to scripts/pr_review.py or scripts/pr_review_rounds.py. They sit at 2199 of 2200 and 696 of 700 lines. New logic goes into scripts/pr_review_side.py, scripts/pr_review_policy.py, or scripts/pr_review_payload.py, never into a raised ceiling.
    • Do not read the side reviewers' checks once in place of polling them. Last round's fix kept the after-checks ask from blocking the Copilot poll. It also made the ask miss every head whose CI outlasts Copilot's round, which is Ask an After-Checks Side Reviewer Once CI Settles, and Document the Exit 44 Holds #3885 item 1.
    • Do not restore a line with a sed pattern after a negative-control revert. A repeated clause rewrote two sites. Revert by line number, or by git diff against the commit.
    • Do not make the merge wait on CodeRabbit or Qodo, or auto-fill their missed starts. Neither leaves a reliable record of which push it reviewed, and the maintainer chose ask-once, never-gate.
    • Do not add a base filter to the Copilot auto-request. held already excludes uncovered bases.
    • Do not patch gql out in a test without attaching the default policy. GqlCase.answer attaches it, and a fake gql must also accept the policy keyword.

    New learnings

    • A negative control that passes on revert means the test never reaches the fixed path. Two of last round's tests needed a body-less read and a real Copilot history before the revert failed them. Recorded in the hub-wide verification lessons memory.
  11. ptr727
    changed the title [-]Session Handoff [review-cutover]: #3881 Merged, Metrics PR and #3885 Next[/-] [+]Session Handoff [review-cutover]: #3886 Metrics Merged With Baseline Captured, #3885 and #3887 Next[/+] on Nov 5, 2026
  12. ptr727
    added a commit that references this issue on Nov 5, 2026

    Ask an After-Checks Side Reviewer Once CI Settles, and Compare Ask Ti…

    4cc7d92

  13. ptr727 commented on Nov 5, 2026

    @ptr727
    ptr727
    on Nov 5, 2026
    OwnerAuthor

    Round 5, closed 2026-11-05: Session Handoff [review-cutover]: #3886 Metrics Merged With Baseline Captured, #3885 and #3887 Next

    This track carries the move of the PR bots, Copilot, CodeRabbit and Qodo, from automatic to policy-driven review, to cut reviewer cost per useful finding. The maintainer revised the plan on 2026-11-04:

    • Account level: automatic review is off for all three bots.
    • Per repository: a policy file .github/pr-review.json says which bots review, on which bases, when, and how often. scripts/pr_review.py obeys it, and repo-config/configure.sh applies its bot-side settings.
    • Skills and rule text: they stay agnostic to whether reviews are automatic.
    • Measurement: the effect is measured in scripts/backlog_metrics.py.

    Next steps

    1. Fix Ask an After-Checks Side Reviewer Once CI Settles, and Document the Exit 44 Holds #3885 and Compare Side-Reviewer Ask Times With the Head's Commit Time in UTC #3887 together in one PR into develop. Both touch scripts/pr_review_side.py. Done when merged into develop.
    2. PR B: repo-config/configure.sh applies and checks the policy.
      • Copilot's copilot_code_review rule in the develop and main rulesets comes from the policy rather than from main.json. This needs a jq edit in both apply_ruleset and check_ruleset.
      • apply --checkout <path> writes only the auto-trigger keys of .coderabbit.yaml and .pr_agent.toml into a local checkout, the form the maintainer chose. The caller commits through the ordinary PR flow.
      • check reads both files through the contents API and reports drift.
      • Update repo-config/README.md, which still describes today's fixed rule.
    3. Probe CodeRabbit's central-config repo on a personal account, with the maintainer's go-ahead. Create a minimal ptr727/coderabbit with reviews.auto_review.enabled: false, and install CodeRabbit on it. Then open a test PR on a public repo above CodeRabbit's 10-star gate that carries no .coderabbit.yaml.
    4. PR D: generate ptr727/coderabbit and ptr727/pr-agent-settings from the hub, in the build_dist.py pattern, with an audit check. It extends Generate the Qodo Settings Repository From the Hub So Fleet Rules Are Authored Once and Auditable #1464, and the maintainer chose hub-generated settings repos. Qodo's key is [config] disable_auto_feedback = true. Creating either repo is an outward write needing the go-ahead at the time.
    5. Promote and release Read a Per-Repository Review Policy Before Asking Any Bot for a Review #3881, Measure PR Review Return on Cost in the Backlog Report #3886, and PR B together. The merge and the release each need the maintainer's go-ahead per merge-and-release.
    6. Pilot on ProjectTemplate.
      • Every repo staying automatic first gets its per-repo override applied.
      • Then ProjectTemplate's .github/pr-review.json moves to explicit, through a PR followed by configure.sh apply --checkout.
      • Then the maintainer turns off Copilot's personal "Automatic Copilot code review" toggle, and the settings repos are created.
      • Verify with a fresh PR timeline. Run a week, compare against the 28-day baseline, then move other repos by their policy file. Rollback is reverting that file.
    7. Close Stop pr_review.py Requesting Copilot on Feature Pull Requests into Develop #3758 with the pilot's verification. Comment on Decide: How the Coordinated Cutover to Explicit Bot-Review Requests Is Triggered, Scheduled, and Gated #3826 that this per-repo design superseded its single window. Close this track with close.

    External blockers

    • The metrics server timer must change its command before its next weekly run from a checkout carrying Measure PR Review Return on Cost in the Backlog Report #3886. The report is now a uv script: uv run --no-project --script scripts/backlog_metrics.py, with uv on that host. Plain python3 now fails at import on markdown-it-py. Only the maintainer can change the timer.
    • Settings changes need the maintainer's go-ahead at the time. That covers each settings change, the two settings repos, the Copilot personal toggle, and the CodeRabbit UI fallback. Steps 1 and 2 are safe meanwhile, since the hub default policy reproduces today's behavior.
    • Copilot's personal auto-review toggle is UI-only and still on. Every PR in the hub gets an automatic Copilot request at open until step 6.

    Internal dependencies

    State

    Parked decision queue

    No open decision issues at the time of writing (count: 0).

    What the last round did

    What not to repeat

    • Do not add logic to scripts/pr_review.py, scripts/pr_review_rounds.py, or scripts/pr_review_checks.py. They sit at about 2193 of 2200, 696 of 700, and 449 of 450 lines. A one-line docstring growth in pr_review_checks.py failed test_pr_review_size this round.
    • Do not copy a gate reader into a plain-python3 script. Every gate reader chains into markdown-it. The copies drifted from the gate within one round, on fences, first-wins, refusals, and ask rules. Make the script a uv script and import the reader.
    • Do not compare a committedDate with a createdAt as text. A GitTimestamp keeps the committer's offset. Parse both with utc_instant.
    • Do not count a gate-sent ask as automation. wait posts asks under the session's own account, so only a review-request event with a bot actor reads as automation.

    New learnings

    • Patching a module attribute does not reach a function's default argument bound at definition time, so bm.run_subprocess never reached gh_graphql.request. Pass the runner explicitly. Recorded in the hub-wide verification lessons memory.
  14. ptr727
    changed the title [-]Session Handoff [review-cutover]: #3886 Metrics Merged With Baseline Captured, #3885 and #3887 Next[/-] [+]Session Handoff [review-cutover]: #3888 Merged, PR B configure.sh Next[/+] on Nov 5, 2026

  15. ptr727 commented on Nov 5, 2026

    @ptr727
    ptr727
    on Nov 5, 2026
    OwnerAuthor

    A coordination note for this track's next round, from the session that worked #3200.

    PR #3889, now merged into develop, widens the per-repository review policy into one repo-local agent policy file. The maintainer decided this on 2026-11-05.

    • .github/pr-review.json became .github/agent-policy.json. The current schema moved under a review section.
    • A new guard section sits beside it. It lists the branches on which gh-write-guard lets an agent commit in place in a primary checkout, the operational repository's standard workflow. The guard reads that section from origin/main only.
    • spec/pr-review.schema.json and spec/pr-review.default.json are now spec/agent-policy.*. scripts/pr_review_policy.py reads the review section, and a file with no review section reads the hub default's.
    • No repository carried the old file, so nothing downstream moved.

    What this changes in the plan:

    • Step 2, PR B: build on develop as it stands. configure.sh should render the whole .github/agent-policy.json from the registry: workflowModel: operational fills in guard.primaryCheckout.commitInPlace.branches: ["develop"], and an optional per-repository agentPolicy override covers the rest. The bot-side settings are then derived from the rendered review section as already planned.
    • Step 6, the pilot: it changes .github/agent-policy.json rather than .github/pr-review.json.

  16. 48 remaining items

  17. ptr727 commented on Nov 7, 2026

    @ptr727
    ptr727
    on Nov 7, 2026
    OwnerAuthor

    Test reports for next step 1, read from the repositories' own PRs at the round 18 start, 2026-11-07. The cases are ptr727/Financial-Modeling#1880 and ptr727/HomeAutomation-Config#1782, both into develop, and ptr727/HomeAutomation-Config#1785, a promotion to main. PlexCleaner has opened no PR since its policy change (#1218), so its report is still owed.

  18. coderabbitai commented on Nov 7, 2026

    @coderabbitai
    coderabbitaiBot
    on Nov 7, 2026 – with coderabbitai
    Analysis chain

    Script executed:

    #!/bin/bash
printf '%s\n' '=== Locate named files ==='
fd -HI -t f '^(\.coderabbit\.yaml|configure\.sh)$' .
printf '%s\n' '=== Map hub configure.sh ==='
ast-grep outline repo-config/configure.sh --items all || true
printf '%s\n' '=== Read named hub files ==='
for file in .coderabbit.yaml repo-config/configure.sh; do
  if [ -f "$file" ]; then
    printf '\n=== %s ===\n' "$file"
    cat -n "$file"
  else
    printf 'Not found: %s\n' "$file"
  fi
done

    Length of output: 68504

    @ptr727 Thanks for the test report.

    • CodeRabbit requests came after the other checks finished on all three PRs. PlexCleaner still needs a test.
    • Copilot’s three-round cap remains untested. The report does not confirm whether wait timed out on HomeAutomation-Config#1785.
    • HomeAutomation-Config’s step 4 is complete per the reported check. The hub’s configure.sh check reads policy files from the target repository’s default branch; I did not rerun the live check.
    • Financial-Modeling and PlexCleaner still lack policy files on main per the report. Before adding central configs, check central_config.py precondition for each settings repository.
    Learnings used

    Learnt from: ptr727
Repo: ptr727/ProjectTemplate PR: 0
File: :0-0
Timestamp: 2026-09-07T22:06:39.139Z
Learning: In `scripts/pr_review.py`, CodeRabbit review-state reporting must distinguish an explicit `@coderabbitai review` request that is rate-limited from a review that was never requested. The current detection recognizes only the auto-review rate-limit response.

    Learnt from: ptr727
Repo: ptr727/ProjectTemplate PR: 1422
File: .agents/skills/pr-review-conduct/SKILL.md:95-98
Timestamp: 2026-09-07T19:54:06.074Z
Learning: In `.agents/skills/pr-review-conduct/SKILL.md`, the “Which Reviewers a Repository Actually Has” section separates reviewer availability cases: explicitly trigger a manual-review skip, wait for a notice that specifies a rate-limit time, and escalate a Copilot account-quota refusal to the maintainer.

    Learnt from: ptr727
Repo: ptr727/ProjectTemplate PR: 0
File: :0-0
Timestamp: 2026-09-07T22:15:26.129Z
Learning: In `scripts/pr_review.py`, review-state reporting must distinguish an explicit `@coderabbitai review` request that receives a rate-limit response from a review that was never requested. The current detection recognizes only the auto-review rate-limit response.

    You are interacting with an AI system.

  19. ptr727 commented on Nov 8, 2026

    @ptr727
    ptr727
    on Nov 8, 2026
    OwnerAuthor

    Test report from ptr727/Financial-Modeling#1882, into develop. It is the first PR there cut after the policy merge (#1878), so its head carries .coderabbit.yaml with auto_review.enabled: false.

    • CodeRabbit still posted on open. A walkthrough summary comment landed 9 seconds after the PR opened, although the head config has auto review off. Its commit status read skipping, so it ran no review. So a branch cut after the change does stop the auto review, but it does not stop the summary comment. That part of CodeRabbit Clean Passes Leave No Review, and It Reads Config From the Head Branch #3929 item 2 does not hold for the summary.
    • wait held the CodeRabbit request until checks passed. @coderabbitai review came after 12/12 checks, and CodeRabbit answered "Review finished" 6 seconds later, with the note that it does not re-review commits it already reviewed. It was not rate limited this time, and wait ended with exit 0 after 657 s, not at its timeout.
    • Copilot reviewed on open, from the personal toggle. wait printed its note that a request was already open when it started. It was one round with no findings, on a PR with a single push, so the 3-round cap is still untested here.
    • Qodo posted nothing.

    🤖 Generated with Claude Code

  20. coderabbitai commented on Nov 8, 2026

    @coderabbitai
    coderabbitaiBot
    on Nov 8, 2026 – with coderabbitai
    Analysis chain

    Script executed:

    #!/bin/bash
set -u

printf '%s\n' '=== Financial-Modeling#1882 head ==='
head_sha=$(gh api repos/ptr727/Financial-Modeling/pulls/1882 --jq '.head.sha') || exit 1
printf 'Head: %s\n' "$head_sha"

printf '%s\n' '=== Find .coderabbit.yaml at the PR head ==='
gh api "repos/ptr727/Financial-Modeling/git/trees/$head_sha?recursive=1" \
  --jq '.tree[] | select(.path == ".coderabbit.yaml") | .path' || true

printf '%s\n' '=== Read .coderabbit.yaml at the PR head ==='
gh api "repos/ptr727/Financial-Modeling/contents/.coderabbit.yaml?ref=$head_sha" \
  --jq '.content' | base64 --decode || true
printf '\n%s\n' '=== ProjectTemplate issue #3929 ==='
gh issue view 3929 --repo ptr727/ProjectTemplate --json title,state,body \
  --jq '"Title: \(.title)\nState: \(.state)\n\(.body)"' || true

    Length of output: 1155

    Script executed:

    #!/bin/bash
set -u

printf '%s\n' '=== Public API: Financial-Modeling#1882 ==='
pr=$(curl -sS --max-time 15 -H 'Accept: application/vnd.github+json' \
  https://api.github.com/repos/ptr727/Financial-Modeling/pulls/1882) || pr=''
head_sha=$(printf '%s' "$pr" | jq -r '.head.sha // empty' 2>/dev/null)
if [ -n "$head_sha" ]; then
  printf 'Head: %s\n' "$head_sha"
  printf '%s\n' '=== PR head .coderabbit.yaml ==='
  curl -sS --max-time 15 -H 'Accept: application/vnd.github+json' \
    "https://api.github.com/repos/ptr727/Financial-Modeling/contents/.coderabbit.yaml?ref=$head_sha" |
    jq -r '.content // empty' | base64 --decode
else
  printf 'PR head unavailable from the public API.\n'
  printf '%s\n' "$pr" | jq -r '.message // empty' 2>/dev/null || true
fi

printf '\n%s\n' '=== Hub .coderabbit.yaml ==='
fd -HI -t f '^\.coderabbit\.yaml$' . --exec sh -c 'printf "\n--- %s ---\n" "$1"; cat -n "$1"' sh {} || true

printf '\n%s\n' '=== ProjectTemplate issue #3929 ==='
gh issue view 3929 --repo ptr727/ProjectTemplate --json title,state,body \
  --jq '"Title: \(.title)\nState: \(.state)\n\(.body)"' || true

    Length of output: 4825

    @ptr727 Thanks for the report.

    • For #3929, a walkthrough summary on open does not by itself show that an automatic review ran. In this test, the status was skipping.
    • wait requested CodeRabbit after all 12 checks passed and exited successfully after 657 seconds. CodeRabbit’s response noted that it does not re-review commits already reviewed.
    • Copilot’s three-round cap remains untested in this PR. Qodo posted nothing.

    I could not independently verify the PR: both GitHub API reads returned 404.

    Learnings used

    Learnt from: ptr727
Repo: ptr727/ProjectTemplate PR: 2253
File: docs/pr-reviewer-reference.md:0-0
Timestamp: 2026-10-01T16:22:46.080Z
Learning: In ptr727/ProjectTemplate, docs/pr-reviewer-reference.md distinguishes CodeRabbit OSS eligibility from automatic-review behavior. CodeRabbit skip notices on PRs #2249 and #2241 state that repositories with fewer than 10 stars do not receive automatic reviews. Explicit triggers still produce reviews below that gate.

    Learnt from: ptr727
Repo: ptr727/ProjectTemplate PR: 2254
File: docs/pr-reviewer-reference.md:17-17
Timestamp: 2026-10-01T17:26:49.393Z
Learning: In `docs/pr-reviewer-reference.md`, distinguish CodeRabbit's free-review eligibility from its automatic-review conditions. General documentation that public repositories qualify for free reviews does not disprove an automatic-review star gate reported by CodeRabbit's own repository skip notices. Evaluate such operational claims against the skip notices and trigger behavior.

    Learnt from: ptr727
Repo: ptr727/ProjectTemplate PR: 0
File: :0-0
Timestamp: 2026-09-07T22:06:39.139Z
Learning: In `scripts/pr_review.py`, CodeRabbit review-state reporting must distinguish an explicit `@coderabbitai review` request that is rate-limited from a review that was never requested. The current detection recognizes only the auto-review rate-limit response.

    Learnt from: ptr727
Repo: ptr727/ProjectTemplate PR: 0
File: :0-0
Timestamp: 2026-09-07T22:15:26.129Z
Learning: In `scripts/pr_review.py`, review-state reporting must distinguish an explicit `@coderabbitai review` request that receives a rate-limit response from a review that was never requested. The current detection recognizes only the auto-review rate-limit response.

    You are interacting with an AI system.

  21. ptr727 commented on Nov 8, 2026

    @ptr727
    ptr727
    on Nov 8, 2026
    OwnerAuthor

    More step-1 evidence, from HomeAutomation-Config's session about ptr727/HomeAutomation-Config#1786, its first develop PR opened with the policy already on its base. It was run through pr_review.py wait.

    • Copilot: the only round was the personal toggle's review on open. It arrived before checks passed, and wait flagged the automatic trigger as still on. wait requested no explicit round, as intended, since Copilot's bases is @default only. So the hold-until-checks behavior and the 3-round cap still have no test on a main-base PR with more than one push.
    • CodeRabbit: its auto review on open was skipped ("Auto reviews are disabled"), since the PR's head carried the policy change. wait posted @coderabbitai review once, after checks were green. That round came back rate-limited, and on the next head wait reported that the policy allows CodeRabbit no further round. Here the rate limit did not hold wait, which exited 0 after 719 s.
    • Docs clarity: the worker read wait's first exit 0 with 3 unresolved Copilot threads as a contract breach. It isn't one. In the usage text, "with no thread open" applies only to the held-head branch, and a review being present exits 0 so the driver can triage. The sentence is still easy to misread, and is worth rewording the next time that usage text changes.

  22. coderabbitai commented on Nov 8, 2026

    @coderabbitai
    coderabbitaiBot
    on Nov 8, 2026 – with coderabbitai
    Analysis chain

    Script executed:

    #!/bin/bash
set -u

printf '%s\n' '=== Locate pr_review.py ==='
fd -HI -t f '^pr_review\.py$' .

printf '\n%s\n' '=== Map wait-related symbols ==='
ast-grep outline scripts/pr_review.py --items all --match 'run_wait|parser|main' || true

printf '\n%s\n' '=== Usage text and thread references ==='
rg -n -C 5 'with no thread open|no thread open|def run_wait|review.*present|held.head|unresolved' scripts/pr_review.py scripts/README.md || true

printf '\n%s\n' '=== run_wait source ==='
python - <<'PY'
import ast
from pathlib import Path

path = Path("scripts/pr_review.py")
source = path.read_text(encoding="utf-8")
lines = source.splitlines()
tree = ast.parse(source)
for node in ast.walk(tree):
    if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name == "run_wait":
        for number in range(node.lineno, node.end_lineno + 1):
            print(f"{number:5} {lines[number - 1]}")
PY

    Length of output: 56526

    Script executed:

    #!/bin/bash
set -eu

printf '%s\n' '=== wait usage text ==='
sed -n '130,180p' scripts/pr_review.py | nl -ba -v130

printf '\n%s\n' '=== run_wait implementation ==='
sed -n '1557,1835p' scripts/pr_review.py | nl -ba -v1557

printf '\n%s\n' '=== documented exit codes ==='
sed -n '203,217p' scripts/README.md | nl -ba -v203

    Length of output: 26904

    @ptr727 Thanks for the report.

    • On HomeAutomation-Config#1786, wait requested CodeRabbit after checks passed and exited 0 despite the rate limit. On the next head, it reported that policy allowed no further CodeRabbit round.
    • Copilot’s after-checks behavior and three-round cap still need a test on a main-base PR with multiple pushes. That test should distinguish wait requests from the personal toggle’s requests.
    • The wait implementation checks for open threads on the held-head local-pass path. A present Copilot review can return 0 with threads still open for triage.

    For the next usage-text edit, I suggest:

    Exit 0 can mean a Copilot review is present, even when threads remain open. On an open held head, exit 0 requires an attested local pass, no open threads, and checks that no longer hold the poll.

    Learnings added 

    Learnt from: ptr727
URL: https://github.com/ptr727/ProjectTemplate/issues/3877

Timestamp: 2026-11-08T00:57:48.386Z
Learning: Within repository "ptr727/ProjectTemplate", only for files matching one of ["scripts/pr_review.py"]: When reviewing changes to the `wait` exit-code usage in `scripts/pr_review.py`, ensure the exit-0 wording distinguishes a present Copilot review, which can return 0 with open threads for the driver to triage, from an open held head covered by an attested local pass, which requires no open threads for exit 0.

    Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

    Learnings used

    Learnt from: ptr727
Repo: ptr727/ProjectTemplate PR: 0
File: :0-0
Timestamp: 2026-09-07T22:06:39.139Z
Learning: In `scripts/pr_review.py`, CodeRabbit review-state reporting must distinguish an explicit `@coderabbitai review` request that is rate-limited from a review that was never requested. The current detection recognizes only the auto-review rate-limit response.

    Learnt from: ptr727
Repo: ptr727/ProjectTemplate PR: 0
File: :0-0
Timestamp: 2026-09-07T22:15:26.129Z
Learning: In `scripts/pr_review.py`, review-state reporting must distinguish an explicit `@coderabbitai review` request that receives a rate-limit response from a review that was never requested. The current detection recognizes only the auto-review rate-limit response.

    You are interacting with an AI system.

  23. ptr727 commented on Nov 8, 2026

    @ptr727
    ptr727
    on Nov 8, 2026
    OwnerAuthor

    Round 17, closed 2026-11-07: Session Handoff [review-cutover]: Three Repositories on Explicit Review, Hub 2.0.1365 Released, Test Reports Next

    This track moves the PR bots (Copilot, CodeRabbit and Qodo) from automatic to policy-driven review, to cut reviewer cost per useful finding. The maintainer revised the plan on 2026-11-04:

    • Account level: automatic review is off for all three bots.
    • Per repository: .github/agent-policy.json says which bots review, on which bases, when, and how often. scripts/pr_review.py obeys it, and repo-config/configure.sh applies the bot-side settings.
    • Skills and rule text: they stay agnostic to whether reviews are automatic.
    • Measurement: scripts/backlog_metrics.py measures the effect.

    Each repository's policy file is rendered from registry/repos.json. scripts/agent_policy.py write writes the files into a checkout. configure.sh apply writes the live settings and rulesets. check diffs against the render, reading the files from the default branch. The central settings repositories are written by scripts/central_config.py.

    Four repositories now review by explicit request with the same values (#3924, released in 2.0.1365): the hub, PlexCleaner, Financial-Modeling and HomeAutomation-Config. The hub's override names Qodo too, the other three leave Qodo on the fleet default:

    Bot Trigger Bases When Rounds
    Copilot explicit, gates default branch after-checks per push, capped at 3
    CodeRabbit explicit main, develop after-checks once
    Qodo auto on the three, explicit on the hub main, develop on-open per push

    Next steps

    1. Collect the test reports from the three repository sessions on their next develop PRs, and record them on this issue. Each session was asked to report to this track's session: whether wait holds the Copilot and CodeRabbit requests until checks pass, the 3-round cap, CodeRabbit's answer to an explicit request, and Copilot reviewing on open. A fresh session reads the reports from the repositories' own PRs instead.
    2. Values check-in 1, on or after 2026-11-13. Review each bot's values against the weekly scripts/backlog_metrics.py output and the 28-day baseline, which is on the maintainer's machine, so ask for it. Fold in step 1's reports and the test findings under "What the last round did". Put each change to the maintainer as a question with a recommendation. A change is a registry override PR, then each repository writes its files, then configure.sh apply with the maintainer's go-ahead.
    3. Fix A Rate-Limited CodeRabbit Holds pr_review.py wait Until Its Timeout #3928 (a rate-limited CodeRabbit holds wait until its timeout) before more repositories move. It turns every rate-limited explicit round into a 45-minute wait.
    4. Each test repository's own promotion to main, on the maintainer's word only. Until then configure.sh check reports its policy files missing or stale on main, which is expected. HomeAutomation-Config's session opened its promotion on 2026-11-07. Done for each when configure.sh check exits 0.
    5. Remaining step-4 repositories still off main: Financial-Modeling and PlexCleaner are the two central_config.py precondition names. EspDinIoT is held on develop as a work in progress. Before any promotion, list its unverified commits, and rewrite an unsigned one rather than bypassing.
    6. Values check-in 2, on or after 2026-11-20, or one week after step 8 turns the toggle off, whichever is later.
    7. Add the central config files with central_config.py write in a clone of each settings repository once central_config.py precondition exits 0, then a PR there, each on the maintainer's go-ahead.
    8. Turn the account toggles off and verify (maintainer, after step 7). Copilot's personal "Automatic Copilot code review" toggle is still on, and wait notes it on every PR. Then move other repositories by their registry entry, each with its two check-ins.
    9. Close Stop pr_review.py Requesting Copilot on Feature Pull Requests into Develop #3758 with the pilot's verification. Comment on Decide: How the Coordinated Cutover to Explicit Bot-Review Requests Is Triggered, Scheduled, and Gated #3826 that the per-repo design superseded its single window. Close this track with close.

    External blockers

    • The maintainer's word for each repository promotion, and for each settings-repository write.
    • Copilot's personal auto-review toggle is UI-only and still on.
    • The metrics baseline is on the maintainer's machine.
    • CodeRabbit's plan rate limit binds explicit rounds on every repository, private ones included.

    Internal dependencies

    • Step 4 for Financial-Modeling and PlexCleaner before step 7. central_config.py withholds the config file until every live registry repository carries its own file on its default branch.
    • Check-in 1 before other repositories move to explicit. Its answers decide which values they inherit.
    • Step 3 before other repositories move, for the reason it states.

    State

    Parked decision queue

    0 open decision issues (count: 0).

    What the last round did

    What not to repeat

    • Don't set CodeRabbit off for a private repository. It reviews them. The maintainer wants it kept, explicit and once, because its findings are good and its quota is scarce. Test turning a bot off in isolation, never bundled into another change.
    • Don't trust docs/pr-reviewer-reference.md on which bots review private repositories until Reviewer Reference Says CodeRabbit Skips Private Repositories, but It Reviews Them #3925 lands.
    • Don't run central_config.py or agent_policy.py with bare python3. Use uv run --no-project --script, which installs their inline dependencies.
    • Don't run uv run without --no-project in a hub worktree. It writes an untracked uv.lock and .venv, which move the local review digest.
    • Don't pass the primary checkout as --checkout to merge --cleanup for a worktree made from another clone. The worktree stays registered in its own base clone. Pass that clone instead.
    • Don't rewrite a resolved registry driftNote as history. AUDIT.md "Reconcile driftNotes" deletes it, and a note restating a met baseline check goes too.

    New learnings

  24. ptr727
    changed the title [-]Session Handoff [review-cutover]: Three Repositories on Explicit Review, Hub 2.0.1365 Released, Test Reports Next[/-] [+]Session Handoff [review-cutover]: #3928 Fixed and Released as 2.0.1366, Check-In 1 Next[/+] on Nov 8, 2026

  25. ptr727 commented on Nov 8, 2026

    @ptr727
    ptr727
    on Nov 8, 2026
    OwnerAuthor

    Correction to round 18's State: this session's hub clone, the detached ProjectTemplate-hub-handoff-20261107 under the fleet worktree directory, was not removed. A safety check blocked removing that directory. It is clean, sits on main at 24535332, and has no worktrees registered from it, so the maintainer can delete it safely. A resume should not use it as its hub checkout and should clone fresh instead.

    Parked decision queue at close: 0 open decision issues.

  26. coderabbitai commented on Nov 8, 2026

    @coderabbitai
    coderabbitaiBot
    on Nov 8, 2026 – with coderabbitai
    Analysis chain

    Script executed:

    #!/bin/bash
set -u

printf '%s\n' '=== Locate named source and instruction files ==='
fd -HI -t f '^(AGENTS\.md|pr_review\.py|pr_review_side\.py|pr_review_policy\.py|central_config\.py|agent_policy\.py|backlog_metrics\.py|configure\.sh|repos\.json|agent-policy\.json|pr-reviewer-reference\.md)$' . || true

printf '\n%s\n' '=== Repository instructions ==='
if [ -f AGENTS.md ]; then cat -n AGENTS.md; fi
for dir in scripts tests repo-config; do
  if [ -d "$dir" ]; then
    fd -HI -t f '^AGENTS\.md$' "$dir" --exec sh -c 'printf "\n--- %s ---\n" "$1"; cat -n "$1"' sh {} || true
  fi
done

printf '\n%s\n' '=== Review implementation outlines ==='
for file in scripts/pr_review.py scripts/pr_review_side.py scripts/pr_review_policy.py; do
  if [ -f "$file" ]; then
    printf '\n--- %s ---\n' "$file"
    ast-grep outline "$file" --items all --match 'wait|ask|limit|review|check|policy' || true
  fi
done

printf '\n%s\n' '=== Rate-limit and side-review implementation references ==='
rg -n -C 4 -i 'rate.?limit|await_green|side\.ask|side_asks|side_asked_on_head|never_waited_on' scripts/pr_review.py scripts/pr_review_side.py scripts/pr_review_policy.py scripts/pr_review_checks.py scripts/README.md || true

printf '\n%s\n' '=== Relevant tests ==='
rg -n -C 3 -i 'rate.?limit|after.checks|side_asked_on_head' tests -g '*pr_review*' || true

printf '\n%s\n' '=== Other named files: relevant sections ==='
for file in scripts/central_config.py scripts/agent_policy.py scripts/backlog_metrics.py repo-config/configure.sh docs/pr-reviewer-reference.md .github/agent-policy.json; do
  if [ -f "$file" ]; then
    printf '\n--- %s ---\n' "$file"
    rg -n -C 2 -i 'precondition|auto.review|disable_auto_feedback|policy|rate.?limit|--days' "$file" | head -n 100 || true
  fi
done

printf '\n%s\n' '=== Registry: named repository entries ==='
python - <<'PY'
import json
from pathlib import Path

path = Path("registry/repos.json")
if path.exists():
    data = json.loads(path.read_text(encoding="utf-8"))
    names = {"ProjectTemplate", "PlexCleaner", "Financial-Modeling", "HomeAutomation-Config"}
    def walk(obj):
        if isinstance(obj, dict):
            if any(isinstance(v, str) and (v in names or v in {f"ptr727/{n}" for n in names}) for v in obj.values()):
                print(json.dumps(obj, indent=2))
            for v in obj.values():
                walk(v)
        elif isinstance(obj, list):
            for v in obj:
                walk(v)
    walk(data)
PY

    Length of output: 163747

    Script executed:

    #!/bin/bash
set -u

printf '%s\n' '=== Issue #3928 ==='
gh issue view 3928 --repo ptr727/ProjectTemplate --json number,title,state,body,url,updatedAt || true

printf '\n%s\n' '=== Issue #3929 ==='
gh issue view 3929 --repo ptr727/ProjectTemplate --json number,title,state,body,url,updatedAt || true

printf '\n%s\n' '=== Recent PlexCleaner develop pull requests ==='
gh api 'repos/ptr727/PlexCleaner/pulls?state=all&base=develop&per_page=30&sort=created&direction=desc' \
  --jq '.[] | {number,title,state,created_at,updated_at,html_url,head: .head.ref}' || true

printf '\n%s\n' '=== Current UTC time ==='
date -u '+%Y-%m-%dT%H:%M:%SZ' || true

    Length of output: 12289

    Script executed:

    #!/bin/bash
set -u

printf '%s\n' '=== Issue #3928 timeline ==='
gh api 'repos/ptr727/ProjectTemplate/issues/3928/timeline?per_page=100' \
  --jq '.[] | {event,created_at,commit_id,actor:(.actor.login // null),source:(if .source.issue then {number:.source.issue.number,title:.source.issue.title,state:.source.issue.state,html_url:.source.issue.html_url} else null end),body:(if .body then .body[0:3000] else null end)}' || true

printf '\n%s\n' '=== Issue #3928 comments ==='
gh api 'repos/ptr727/ProjectTemplate/issues/3928/comments?per_page=100' \
  --jq '.[] | {created_at,author:.user.login,body:.body}' || true

printf '\n%s\n' '=== PlexCleaner#1220 ==='
gh api 'repos/ptr727/PlexCleaner/pulls/1220' \
  --jq '{number,state,created_at,updated_at,head:.head.sha,base:.base.ref,requested_reviewers:[.requested_reviewers[].login],html_url}' || true

printf '\n%s\n' '=== PlexCleaner#1220 timeline ==='
gh api 'repos/ptr727/PlexCleaner/issues/1220/timeline?per_page=100' \
  --jq '.[] | select(.event == "review_requested" or .event == "reviewed" or .event == "committed" or .event == "commented") | {event,created_at,submitted_at,actor:(.actor.login // .user.login // null),requested_reviewer:(.requested_reviewer.login // null),body:(if .body then .body[0:1800] else null end),commit_id:(.commit_id // .sha // null)}' || true

printf '\n%s\n' '=== PlexCleaner#1220 current checks and statuses ==='
head_sha=$(gh api 'repos/ptr727/PlexCleaner/pulls/1220' --jq '.head.sha') || head_sha=''
if [ -n "$head_sha" ]; then
  gh api "repos/ptr727/PlexCleaner/commits/$head_sha/status" \
    --jq '{state,statuses:[.statuses[] | {context,state,created_at,updated_at,description}]}' || true
  gh api "repos/ptr727/PlexCleaner/commits/$head_sha/check-runs?per_page=100" \
    --jq '{total_count,check_runs:[.check_runs[] | {name,status,conclusion,started_at,completed_at}]}' || true
fi

    Length of output: 33732

    Script executed:

    #!/bin/bash
set -u

printf '%s\n' '=== Checkout revision ==='
git log -1 --format='%H %s' || true

printf '\n%s\n' '=== Rate-limit helper outlines ==='
ast-grep outline scripts/pr_review_checks.py --items all --match 'limited_since|hosted_reviews_running' || true
ast-grep outline scripts/pr_review_rounds.py --items all --match 'rate_limited_by|reviewer_nodes' || true

printf '\n%s\n' '=== Rate-limit check implementation ==='
sed -n '250,310p' scripts/pr_review_checks.py | nl -ba -v250
rg -n -C 8 'def rate_limited_by\(' scripts/pr_review_rounds.py || true

printf '\n%s\n' '=== Existing wait regression ==='
sed -n '4725,4770p' tests/test_pr_review.py | nl -ba -v4725

printf '\n%s\n' '=== Recent merged fix references ==='
git log -8 --oneline || true

    Length of output: 8884

    Script executed:

    #!/bin/bash
set -u

printf '%s\n' '=== Current handoff issue ==='
gh issue view 3877 --repo ptr727/ProjectTemplate --json title,state,body,updatedAt || true

printf '\n%s\n' '=== Fix and promotion state ==='
for number in 3930 3931; do
  gh pr view "$number" --repo ptr727/ProjectTemplate \
    --json number,title,state,mergedAt,mergeCommit,url || true
done

printf '\n%s\n' '=== PlexCleaner#1220 current head and comments ==='
head_sha=$(gh api 'repos/ptr727/PlexCleaner/pulls/1220' --jq '.head.sha') || head_sha=''
printf 'Head: %s\n' "$head_sha"
gh api 'repos/ptr727/PlexCleaner/issues/1220/comments?per_page=100' \
  --jq '.[] | {created_at,updated_at,author:.user.login,body:.body[0:1000]}' || true

if [ -n "$head_sha" ]; then
  printf '\n%s\n' '=== PlexCleaner#1220 head CodeRabbit config ==='
  gh api "repos/ptr727/PlexCleaner/contents/.coderabbit.yaml?ref=$head_sha" --jq '.content' |
    base64 --decode || true

  printf '\n%s\n' '=== PlexCleaner#1220 check runs ==='
  gh api "repos/ptr727/PlexCleaner/commits/$head_sha/check-runs?per_page=100" \
    --jq '{total_count,check_runs:[.check_runs[] | {name,status,conclusion,started_at,completed_at}]}' || true
fi

    Length of output: 18790

    @ptr727 The current handoff is round 18. Issue #3928 is closed: #3930 fixed the rate-limit reader, and #3931 promoted it. The handoff reports release 2.0.1366. I did not run the tests.

    PlexCleaner test report, partial: PlexCleaner#1220 opened after the policy change.

    • Its head has CodeRabbit automatic review disabled. An explicit request received “Review triggered.” The review was still in progress at the last read.
    • The request appeared at 01:30:06 UTC. codecov/project started at 01:30:11, so the timeline does not establish whether wait saw all checks before requesting.
    • Copilot was requested two seconds after the PR opened. The timeline alone does not establish the request’s source.
    • Qodo posted a review with two findings. Their triage outcome is not established by this read.

    The PlexCleaner session’s wait result is still needed. Copilot’s three-round cap and the live rate-limit fix also remain untested.

    For check-in 1 on or after November 13, please share the 28-day baseline and the weekly metrics report.

    Learnings added 

    Learnt from: ptr727
URL: https://github.com/ptr727/ProjectTemplate/issues/3877

Timestamp: 2026-11-08T01:32:23.780Z
Learning: Within repository "ptr727/ProjectTemplate", only for files matching one of ["registry/repos.json"]: When reviewing a change to `registry/repos.json` that sets CodeRabbit's trigger to `off` for a private repository, flag the change and recommend an explicit, once-only CodeRabbit review instead. The maintainer wants to retain CodeRabbit for private repositories, which can receive its reviews.

    Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

    Learnings used

    Learnt from: ptr727
Repo: ptr727/ProjectTemplate PR: 0
File: :0-0
Timestamp: 2026-09-07T22:06:39.139Z
Learning: In `scripts/pr_review.py`, CodeRabbit review-state reporting must distinguish an explicit `@coderabbitai review` request that is rate-limited from a review that was never requested. The current detection recognizes only the auto-review rate-limit response.

    Learnt from: ptr727
Repo: ptr727/ProjectTemplate PR: 0
File: :0-0
Timestamp: 2026-09-07T22:15:26.129Z
Learning: In `scripts/pr_review.py`, review-state reporting must distinguish an explicit `@coderabbitai review` request that receives a rate-limit response from a review that was never requested. The current detection recognizes only the auto-review rate-limit response.

    You are interacting with an AI system.

  27. ptr727 commented on Nov 8, 2026

    @ptr727
    ptr727
    on Nov 8, 2026
    OwnerAuthor

    Step-2 evidence read at the round 19 start, 2026-11-08, from the repositories' own PRs.

    PlexCleaner's report is no longer owed. ptr727/PlexCleaner#1220, into develop, is the first PlexCleaner PR since its policy change (#1218). It is still open, so this covers what has posted so far.

    • CodeRabbit: a summary comment landed on open, matching ptr727/Financial-Modeling#1882. wait posted @coderabbitai review after the required aggregator check passed. CodeRabbit answered "Review triggered", with no rate limit. One non-gating check, codecov/project, finished 6 s after the request. So wait waits for the gate rather than for every check, which is as intended.
    • Copilot: one round on open from the personal toggle. develop is not in its bases, so no explicit round followed.
    • Qodo posted on its own on open: a PR summary, and then a code review reporting 2 bugs. Qodo had posted nothing on Financial-Modeling or HomeAutomation-Config PRs before this. PlexCleaner is the first repository on the auto, on-open trigger where Qodo actually answers. This bears on check-in 1's Qodo values.

    A rate-limited CodeRabbit released the merge live after #3930. ptr727/HomeAutomation-Config#1787 is a promotion to main with a single push. CodeRabbit posted "Review limit reached" on open. wait requested once after checks passed, and CodeRabbit answered "Review rate limited". The PR merged 8 minutes after that reply. #1785, before #3930, took about an hour in the same state. This fits the "hold neither" fix. The wait exit code was not read, so this is the merge timing only.

    Still untested: the Copilot 3-round cap. #1787 had one push, like #1785. It needs a main-base PR with more than one push.

    Step 3 readiness, read from GitHub: neither repository has .github/agent-policy.json on main yet. compare main...develop reports 0 unverified commits for both: Financial-Modeling is 9 ahead and PlexCleaner 31 ahead. Financial-Modeling's only open main PR is a Dependabot bump (#1816), which is not a promotion.

  28. coderabbitai commented on Nov 8, 2026

    @coderabbitai
    coderabbitaiBot
    on Nov 8, 2026 – with coderabbitai
    Analysis chain

    Script executed:

    #!/bin/bash
set -u
printf '%s\n' '=== Checkout revision ==='
git log -1 --format='%H %s'
printf '\n%s\n' '=== Named policy files ==='
fd -HI -t f '^agent-policy\.json$' . --exec sh -c 'printf "\n--- %s ---\n" "$1"; cat -n "$1"' sh {}
printf '\n%s\n' '=== Review check outlines ==='
for f in scripts/pr_review.py scripts/pr_review_side.py scripts/pr_review_checks.py; do
  if [ -f "$f" ]; then
    printf '\n--- %s ---\n' "$f"
    ast-grep outline "$f" --items all --match 'run_wait|checks_hold|await_green|ask|held_checks_open|runs_open' || true
  fi
done
printf '\n%s\n' '=== After-checks and gating context ==='
rg -n -C 4 'after-checks|checks_hold|held_checks_open|runs_open|await_green|codecov' scripts/pr_review.py scripts/pr_review_side.py scripts/pr_review_checks.py scripts/README.md || true

    Length of output: 34882

    @ptr727 Thanks for the report. The PlexCleaner report is no longer outstanding.

    • On PlexCleaner#1220, wait requested CodeRabbit after the required aggregator passed. The hub’s check logic also considers open workflow runs and settling checks, so this example does not establish that all non-gating checks are ignored.
    • HomeAutomation-Config#1787 provides merge-timing evidence consistent with #3930. It does not establish the wait exit code.
    • The Copilot three-round cap still needs a main-base test with enough eligible heads to reach the cap and test a later head. The test should distinguish personal-toggle requests from wait requests.
    • Financial-Modeling and PlexCleaner remain on hold for promotion. Before adding central configs, check central_config.py precondition for each settings repository.
    Learnings used

    Learnt from: ptr727
Repo: ptr727/ProjectTemplate PR: 2253
File: docs/pr-reviewer-reference.md:0-0
Timestamp: 2026-10-01T16:22:46.080Z
Learning: In ptr727/ProjectTemplate, docs/pr-reviewer-reference.md distinguishes CodeRabbit OSS eligibility from automatic-review behavior. CodeRabbit skip notices on PRs #2249 and #2241 state that repositories with fewer than 10 stars do not receive automatic reviews. Explicit triggers still produce reviews below that gate.

    Learnt from: ptr727
Repo: ptr727/ProjectTemplate PR: 2254
File: docs/pr-reviewer-reference.md:17-17
Timestamp: 2026-10-01T17:26:49.393Z
Learning: In `docs/pr-reviewer-reference.md`, distinguish CodeRabbit's free-review eligibility from its automatic-review conditions. General documentation that public repositories qualify for free reviews does not disprove an automatic-review star gate reported by CodeRabbit's own repository skip notices. Evaluate such operational claims against the skip notices and trigger behavior.

    Learnt from: ptr727
Repo: ptr727/ProjectTemplate PR: 0
File: :0-0
Timestamp: 2026-09-07T22:06:39.139Z
Learning: In `scripts/pr_review.py`, CodeRabbit review-state reporting must distinguish an explicit `@coderabbitai review` request that is rate-limited from a review that was never requested. The current detection recognizes only the auto-review rate-limit response.

    Learnt from: ptr727
Repo: ptr727/ProjectTemplate PR: 0
File: :0-0
Timestamp: 2026-09-07T22:15:26.129Z
Learning: In `scripts/pr_review.py`, review-state reporting must distinguish an explicit `@coderabbitai review` request that receives a rate-limit response from a review that was never requested. The current detection recognizes only the auto-review rate-limit response.

    Learnt from: ptr727
Repo: ptr727/ProjectTemplate PR: 1686
File: :0-0
Timestamp: 2026-09-18T21:54:24.923Z
Learning: For GitHub pull request #1686 in ptr727/ProjectTemplate, a passing CodeRabbit check does not mean CodeRabbit reviewed the pull request. The explicit `@coderabbitai review` request was rate limited and produced no CodeRabbit review comments. The maintainer documented separate coverage from GitHub Copilot, local adversarial reviews, local verification gates, and CI.

    You are interacting with an AI system.

  29. ptr727 commented on Nov 8, 2026

    @ptr727
    ptr727
    on Nov 8, 2026
    OwnerAuthor

    Round 18, closed 2026-11-08: Session Handoff [review-cutover]: #3928 Fixed and Released as 2.0.1366, Check-In 1 Next

    This track moves the PR bots (Copilot, CodeRabbit and Qodo) from automatic to policy-driven review, to cut reviewer cost per useful finding. The maintainer revised the plan on 2026-11-04:

    • Account level: automatic review is off for all three bots.
    • Per repository: .github/agent-policy.json says which bots review, on which bases, when, and how often. scripts/pr_review.py obeys it, and repo-config/configure.sh applies the bot-side settings.
    • Skills and rule text: they stay agnostic to whether reviews are automatic.
    • Measurement: scripts/backlog_metrics.py measures the effect.

    Each repository's policy file is rendered from registry/repos.json. scripts/agent_policy.py write writes the files into a checkout. configure.sh apply writes the live settings and rulesets. check diffs against the render, reading the files from the default branch. The central settings repositories are written by scripts/central_config.py.

    Four repositories review by explicit request with the same values (#3924): the hub, PlexCleaner, Financial-Modeling and HomeAutomation-Config. The hub's override names Qodo too, the other three leave Qodo on the fleet default:

    Bot Trigger Bases When Rounds
    Copilot explicit, gates default branch after-checks per push, capped at 3
    CodeRabbit explicit main, develop after-checks once
    Qodo auto on the three, explicit on the hub main, develop on-open per push

    Next steps

    1. Values check-in 1, on or after 2026-11-13. Review each bot's values against the weekly scripts/backlog_metrics.py output and the 28-day baseline, which is on the maintainer's machine, so ask for it. Fold in the test reports on this issue (the round-18 comments) and the findings under "What the last round did". Put each change to the maintainer as a question with a recommendation. A change is a registry override PR, then each repository writes its files, then configure.sh apply with the maintainer's go-ahead.
    2. Collect the reports still owed. PlexCleaner has opened no PR since its policy change (D7.3 followed literally for smoke inverts D1.3's smoke-never-publishes guard #1218), so its report is still owed. The Copilot 3-round cap is still untested, since it needs a main-base PR with more than one push. A rate-limited CodeRabbit releasing wait and merge live, after Read CodeRabbit's Rate-Limit Reply With Markup Around Its Sentence #3930, is untested too. Read each from the repositories' own PRs.
    3. Promote Financial-Modeling and PlexCleaner to main, on the maintainer's word only. Until then configure.sh check reports their policy files missing on main, which is expected. Done for each when configure.sh check exits 0. EspDinIoT is held on develop as a work in progress. Before any promotion, list its unverified commits, and rewrite an unsigned one rather than bypassing.
    4. Values check-in 2, on or after 2026-11-20, or one week after step 6 turns the toggle off, whichever is later.
    5. Add the central config files with central_config.py write in a clone of each settings repository once central_config.py precondition exits 0, then a PR there, each on the maintainer's go-ahead.
    6. Turn the account toggles off and verify (maintainer, after step 5). Copilot's personal "Automatic Copilot code review" toggle is still on, and wait notes it on every PR. Then move other repositories by their registry entry, each with its two check-ins.
    7. Close Stop pr_review.py Requesting Copilot on Feature Pull Requests into Develop #3758 with the pilot's verification. Comment on Decide: How the Coordinated Cutover to Explicit Bot-Review Requests Is Triggered, Scheduled, and Gated #3826 that the per-repo design superseded its single window. Close this track with close.

    External blockers

    • The maintainer's word for each repository promotion, and for each settings-repository write.
    • Copilot's personal auto-review toggle is UI-only and still on.
    • The metrics baseline is on the maintainer's machine.
    • CodeRabbit's plan rate limit binds explicit rounds on every repository, private ones included.

    Internal dependencies

    • Step 3 before step 5. central_config.py withholds the config file until every live registry repository carries its own file on its default branch.
    • Check-in 1 before other repositories move to explicit. Its answers decide which values they inherit.

    State

    • Hub: main at 24535332 (promotion Promote Develop: Read CodeRabbit's Rate-Limit Reply With Markup Around Its Sentence #3931, carrying Read CodeRabbit's Rate-Limit Reply With Markup Around Its Sentence #3930, merged with the head pinned at 0ed3fe89). Release run 42433991462 succeeded from that tip and published 2.0.1366. No hub PR is open from this track, and develop holds nothing main lacks from it.
    • HomeAutomation-Config reached main (its Point the HomeAutomation-Config Audit's Escalation at Its Issue #1785), and configure.sh check exits 0. Financial-Modeling and PlexCleaner still have no policy files on main.
    • Settings repositories: both config files are still withheld, since the step-3 precondition is unmet.
    • This machine: the skills snapshot is current at 24535332, and the host-safety kit reads CURRENT. Claude Code loads the primary checkout, still on develop at c2373754, and cleanup --sweep reports it would fast-forward. That is the maintainer's to update. cleanup --sweep reports problems=0.
    • This session's hub clone, a detached checkout named ProjectTemplate-hub-handoff-20261107 under the fleet worktree directory, is removed at close.
    • Peer sessions on this host are working PlexCleaner, Financial-Modeling and HomeAutomation-Config. Leave their repositories to them.

    Parked decision queue

    0 open decision issues (count: 0).

    What the last round did

    What not to repeat

    • Don't read a wait exit 0 with threads open as a contract breach. In the usage text, "with no thread open" applies only to the attested held-head branch. A review being present exits 0 so the driver can triage. Merge Gate item 3 is merge's and the driver's. Worth rewording the next time that usage text changes.
    • Don't expect an explicit Copilot round on a develop PR. Copilot's bases is @default, and a develop head is covered by the attested local pass. The only Copilot round there is the personal toggle's on-open one.
    • Don't add a docstring line to scripts/pr_review.py casually. It sits at the 2200-line size ceiling, and Read CodeRabbit's Rate-Limit Reply With Markup Around Its Sentence #3930 had to revert one.
    • Don't set CodeRabbit off for a private repository. It reviews them. The maintainer wants it kept, explicit and once. Test turning a bot off in isolation, never bundled into another change.
    • Don't trust docs/pr-reviewer-reference.md on which bots review private repositories until Reviewer Reference Says CodeRabbit Skips Private Repositories, but It Reviews Them #3925 lands.
    • Don't run central_config.py or agent_policy.py with bare python3, or uv run without --no-project in a hub worktree. Use uv run --no-project --script.

    New learnings

  30. ptr727
    changed the title [-]Session Handoff [review-cutover]: #3928 Fixed and Released as 2.0.1366, Check-In 1 Next[/-] [+]Session Handoff [review-cutover]: Step-2 Reports In, Step 3 Left to Peer Sessions, Check-In 1 Next[/+] on Nov 8, 2026

  31. ptr727 commented on Nov 8, 2026

    @ptr727
    ptr727
    on Nov 8, 2026
    OwnerAuthor

    Answers given at the round 19 close, 2026-11-08:

    • Step 3, promoting Financial-Modeling and PlexCleaner: wait for the peer sessions. Each repository's own session promotes after its open PR lands. This track checks the result rather than driving it.
    • The round-18 hub clone: delete it. I checked it was clean and deleted it.

    Parked decision queue at close: 0 open decision issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    handoffA link in the session handoff chain, one open issue per track

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions