Skip to content

Request Bot PR Reviews Judiciously, and Stop on a Quota or Rate Limit #2261

Closed
#2284
Closed
Request Bot PR Reviews Judiciously, and Stop on a Quota or Rate Limit#2261
#2284
Labels
enhancementNew feature or request

Description

@ptr727
ptr727
opened on Oct 1, 2026

The Question

Copilot code review now reaches a weekly rate limit under the fleet's normal review traffic. How should the fleet spend its Copilot reviews?

What Is Known

  • The limit is weekly and separate from the monthly budget. A fresh month does not reset it. Once it is reached, every review request fails until the stated reset.
  • Reaching it is new. The limit is not. The default review level moved from lite to medium, and each review now consumes more.
  • The model cannot be changed in settings, only the level. The failure message suggests switching to an auto model, and no setting found so far offers that.
  • The failure is invisible to the fleet's tooling. The posted review body reads only "Copilot encountered an error and was unable to review this pull request". The rate-limit text appears only in the Copilot task log. scripts/pr_review.py therefore reports a refusal it cannot tell apart from a structural one, and a drive following the re-request rule keeps asking.
  • Current consumption is high by construction. pr-review-conduct re-requests Copilot after every push, so each fix push costs one full review. Parallel unattended lanes multiply that.

Options

These are not exclusive.

  1. Request Copilot at fewer moments. For example, on a pull request's first push and on the promotion pull request, with the recorded local-strict-review pass covering intermediate fix pushes. This edits pr-review-conduct, drive-pr, and the carried runbook.
  2. Lower the review level back to lite where the setting allows it, trading depth per review for reviews per week.
  3. Cap Copilot use by unattended runs, for example one lane at a time, or no Copilot request from unattended-handoff workers.
  4. Make the failure visible. pr_review.py would say a generic error refusal may be a quota, and the drive rule would stop re-requesting after one such refusal until the reset.
  5. Lean on the other reviewers while Copilot is out. CodeRabbit reviews this public low-star repo on request. This is today's fallback rather than a plan.

Waiting On

The maintainer's own research into the limit and any configuration that lifts it.

Activity

  1. ptr727
    added
    decisionA question waiting on the maintainer, alone or beside whatever else the issue carries
    on Oct 1, 2026

  2. ptr727 commented on Oct 1, 2026

    @ptr727
    ptr727
    on Oct 1, 2026
    OwnerAuthor

    Maintainer's answer, 2026-10-01: the default review level is back to lite until the real cost is known (it is not disclosed beyond a stated 1 to 5 ratio between lite and medium). Build a mechanism that is judicious about when a bot review is requested on a pull request. Fewer requests cut cost, and they also cut review time, especially for CodeRabbit, which caps the review count per pull request. Treat it as a defensive feature covering every requested bot reviewer, not only Copilot. Options 1 and 4 above are the starting shape: request at defined moments rather than after every push, with the recorded local pass covering the pushes in between, and make a quota or rate-limit refusal visible so a drive stops re-requesting.

  3. ptr727
    changed the title [-]Decide How the Fleet Spends Copilot Reviews Under a Weekly Rate Limit[/-] [+]Request Bot PR Reviews Judiciously, and Stop on a Quota or Rate Limit[/+] on Oct 1, 2026
  4. ptr727
    added
    enhancementNew feature or request
    and removed
    decisionA question waiting on the maintainer, alone or beside whatever else the issue carries
    on Oct 1, 2026

  5. ptr727 commented on Oct 1, 2026

    @ptr727
    ptr727
    on Oct 1, 2026
    OwnerAuthor

    Adding a scope item: Copilot's automatic-review triggers, which need to change together with wait's auto-request for any saving to happen.

    Two independent trigger configurations. Per GitHub's code review docs ("Automatic pull request reviews"), Copilot checks a user's own automatic-review setting and any applicable repository ruleset separately: "These configurations are not hierarchical; neither inherits from or overrides the other. A pull request is reviewed automatically when at least one configuration is enabled." Narrowing the maintainer's account setting to review-on-open only therefore changes nothing while the rulesets still trigger on push and on drafts.

    The ruleset half is fleet config. repo-config/develop.json and repo-config/main.json both carry the copilot_code_review rule with review_on_push: true and review_draft_pull_requests: true, and the live develop and main rulesets here match. Editing one repository's ruleset in the UI is drift that the next config run reverts, so the change is a hub edit to those two payloads, applied fleet-wide.

    The tooling half. scripts/pr_review.py wait requests a review whenever none covers the head and none is pending. With review_on_push off, an agent-driven loop would still get one review per pushed head, triggered by wait instead of by GitHub. The saving needs wait (and the drive skills that call it) to decide when a head deserves a review rather than requesting one on every head.

    Proposed scope for this issue:

    1. Set review_on_push: false and review_draft_pull_requests: false in both repo-config payloads, keeping review on open.
    2. Change wait's auto-request so a review is requested deliberately rather than per head, for example on the first head, on the head the drive considers final, and on promotion PRs.
    3. The maintainer narrows the account-level setting to match. That is a UI step outside the repository.

    Shipping 1 without 2 saves nothing, and shipping 2 without 1 leaves GitHub reviewing every push anyway.

    Related, under test: per the docs, Copilot chooses the review level from the level picked at request time, then "the level previously used on this pull request", then the requester's default, then the repository and owner defaults. If "previously used" means the most recent level, one manual Balanced pick on a promotion PR carries to every later round the tooling requests. Being tested now, and the result will be posted here.

  6. ptr727 commented on Oct 1, 2026

    @ptr727
    ptr727
    on Oct 1, 2026
    OwnerAuthor

    Follow-up on the review-level test mentioned above: dropped, not run.

    The level cannot be set from code. Per the live GraphQL schema, requestReviews takes only reviewer ids and union, and the ruleset's Copilot code review parameters carry only reviewOnPush and reviewDraftPullRequests. A per-PR level is a manual pick in the web UI's Reviewers section and nothing else, so every review the tooling requests runs at the effective default: the repository's setting where it overrides, the account default where it inherits. A manual Balanced pick stays an occasional exception for tricky code, so whether it carries over to later rounds on the same PR does not change this issue's design.

  7. ptr727
    added a commit that references this issue on Oct 2, 2026

    Promote develop to main: Fork Handoff Chains, Qodo Login, JSON Report (…

    7e4eb6d

  8. ptr727 commented on Oct 2, 2026

    @ptr727
    ptr727
    on Oct 2, 2026
    OwnerAuthor

    Design answers from the maintainer, 2026-10-01, each the recommended option:

    1. Merge Gate on a feature -> develop pull request. Copilot reviews a feature pull request once, on open. Each fix push after that is covered by a recorded local-strict-review pass on that head, and the gate accepts a Copilot round on an earlier head plus a recorded local pass on the current head. A promotion pull request still needs a Copilot round on its head, since that round reads the whole develop diff.
    2. Ruleset. Set review_on_push: false and review_draft_pull_requests: false in repo-config/develop.json and repo-config/main.json, keeping review on open. This is a hub edit in the pull request. Applying the change to live repositories is a separate config run, and it needs its own go-ahead. The maintainer narrows the account-level setting in the UI.
    3. Generic error round. A round reading only "Copilot encountered an error and was unable to review" is treated as a possible quota hit after one occurrence. wait stops auto-requesting on that pull request, and status reports it as a possible quota refusal for the maintainer. Neither re-requests into it.
    4. CodeRabbit. A drive prompts it at most once per pull request, on the head the drive judges final, and never again after a rate-limit marker. It stays advisory.

    A defect found while reading: wait sends its auto-request before it consults the repository quota signal, so an exhausted account still spends a request. The fix lands with this issue.

  9. ptr727
    added 2 commits that reference this issue on Oct 2, 2026

    Stop Requesting Copilot Into a Reached Rate Limit (#2277)

    5a814c1

    Request Copilot at Defined Moments and Cover Fix Pushes With an Attes…

    644b297
  10. ptr727
    closed this as completedin #2284on Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions