Skip to content

Enforce a per-run cost ceiling as a runaway guard - #61

Merged
polewskm merged 10 commits into
mainfrom
issue-50
Oct 31, 2026
Merged

polewskm merged 10 commits into
mainfrom
issue-50

Conversation

@polewskm

Copy link
Copy Markdown
Contributor

Replaces the unused budgets.run with a per-review cost ceiling, runCeiling(parts) = 8 + 4 × parts ($12 for one part, $32 for six), overridable with CODE_REVIEW_MAX_RUN_USD. review() tracks the cost of every call so far and skips each reviewer or rule-check call queued after the total reaches the ceiling. Those reviewers show skipped: run budget in the check run and skipped in the summary comment. Synthesis still runs on what was found, and the summary warns, also as a ::warning:: in the log. Unit tests stub the agent's costs. The README documents the variable.

Decisions:

  • CODE_REVIEW_MAX_RUN_USD must be a plain positive decimal (12, 2.5). Any other value, including 0, a negative number or 1e3, warns and keeps the default.
  • The ceiling is checked when each call starts. Calls already running finish, so the total can go over the ceiling by up to the per-call caps of the calls in flight.
  • A review the ceiling cut short never approves: an approve verdict becomes comment.
  • A reviewer with some calls run and some skipped stays ran, with N of M calls skipped: run budget.
  • review() and selectContext() take an optional agent function, which defaults to runAgent, so tests can stub the calls.

Closes #50

Replace the unused budgets.run with runCeiling(parts) = 8 + 4 x parts, overridable with CODE_REVIEW_MAX_RUN_USD. review() tracks the cost of its calls and skips reviewer and rule-check calls queued after the ceiling is reached, marking them skipped: run budget, while synthesis still runs on what was found and the summary warns.
@polewskm
polewskm deployed to code-review October 31, 2026 04:12 — with GitHub Actions Active

@ncode-review ncode-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 NCode Review at c6424e0: 7 new findings on the changed lines.

Comment thread test/run-ceiling.test.ts Resolved
Comment thread test/run-ceiling.test.ts Resolved
Comment thread test/run-ceiling.test.ts Resolved
Comment thread test/run-ceiling.test.ts Resolved
Comment thread CHANGELOG.md Resolved
Comment thread src/review.ts Resolved
Comment thread src/review.ts Resolved
@ncode-review

ncode-review Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

🤖 Review complete

Verdict: 💬 Comment — This PR replaces the unused budgets.run with a per-run cost ceiling that scales with diff parts and can be overridden with CODE_REVIEW_MAX_RUN_USD. Once spend reaches the ceiling, reviewer and rule-check calls that haven't started are skipped. Synthesis still runs, the summary gets a warning, and the verdict is downgraded from approve. The design holds together: it checks spend before each queued call and adds an injectable agent seam for tests, and the tests cover the main paths well. The remaining findings are small hygiene and style points.

8 reviewers run — 4 findings (run details)

Reviewer Model Findings Top
synthesis opus-5-5 4 🟡
architecture sonnet-5-5 0 🟢
completeness sonnet-5-5 0 🟢
conventions sonnet-5-5 0 🟢
correctness sonnet-5-5 0 🟢
hygiene haiku-5-5 1 🟡
intent haiku-5-5 0 🟢
rule-check sonnet-5-5 2 🟡
tests haiku-5-5 1 🟡

Skipped: api-compat, data-model, infrastructure, performance, security

Findings
  • 🟡 F1 test/run-ceiling.test.ts:19: Tests don't pin the decimal edge cases of the override regex
  • 🟡 F2 src/review.ts:285: review() gains a fifth positional parameter
  • 🟡 F3 src/config.ts:63: Unnamed literal 40 when truncating the echoed override
  • 🟡 F4 test/run-ceiling.test.ts:230: Test file has no trailing newline

Trial: findings are posted as review threads and in the check run. $0.52 on Amazon Bedrock, 60s, at 773338f. (how to interact)

@polewskm

Copy link
Copy Markdown
Contributor Author

/review

@ncode-review ncode-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 NCode Review at 686620e: 4 new findings on the changed lines.
1 more was already posted on this pull request.

Comment thread src/review.ts Resolved
Comment thread src/review.ts Resolved
Comment thread src/results.ts Resolved
Comment thread src/config.ts Resolved
@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: answering #61 (comment)

@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: Round 2 answered. No blocker or should-fix is left open; the remaining threads are nits, answered above. Recommend merging once you've resolved them.

@polewskm

Copy link
Copy Markdown
Contributor Author

/review

@ncode-review ncode-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 NCode Review at 773338f: 2 new findings on the changed lines.
2 more were already posted on this pull request.

Comment thread test/run-ceiling.test.ts Resolved
Comment thread test/run-ceiling.test.ts Resolved
@polewskm
polewskm marked this pull request as ready for review October 31, 2026 06:33
@polewskm
polewskm deployed to code-review October 31, 2026 06:33 — with GitHub Actions Active
@polewskm
polewskm enabled auto-merge October 31, 2026 06:33
@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: merged origin/main (79c2447), which brought the opt-in verification stage from #72 (#41).

  • src/config.ts: main still had budgets.run: 4; this PR's removal of it stands, next to main's new budgets.verification and maxTurns.verification.
  • src/review.ts: the ceiling gates only reviewer and rule-check calls, as before. Verification, when CODE_REVIEW_VERIFICATION=on, runs after synthesis even when the ceiling was reached. It gets the same injectable agent as the other stages. The ceiling's warning and its approve-to-comment rule apply after verification. The README's ceiling paragraph now says verification still runs.
  • test/run-ceiling.test.ts: the test charters gained main's new verificationPrompt (f8877ba).

If verification should also stop at the ceiling, that's a follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enforce a per-run cost ceiling as a runaway guard

1 participant