Repository navigation
Dispatchable evals, synthesis knobs and the decisions rule - #140
Merged
Merged
Conversation
…CODE_REVIEW_SYNTHESIS_EFFORT set synthesis
There was a problem hiding this comment.
🤖 NCode Review at af9d1c5: 3 new findings on the changed lines.
|
Superseded by the review at 43eef8c. |
…levels in one list, write issue references as issue #N
Contributor
Author
|
/review |
🤖 Review completeVerdict: ✅ Approve — Re-review of what changed since the last review at af9d1c5: 8 hunks in 5 files and changes taken back in 4 files, by correctness only. Earlier findings: F1 fixed, F2 fixed, F3 fixed. No further findings above nits. 1 reviewer run — 0 findings (run details)
Trial: findings are posted as review threads and in the check run. $0.05 on Amazon Bedrock, 11s, at 43eef8c. (how to interact) 📋 OutcomesMerged at 43eef8c: 2 fixed, 1 dismissed, 0 acknowledged, 0 ignored.
|
polewskm
marked this pull request as ready for review
November 1, 2026 05:46
polewskm
enabled auto-merge
November 1, 2026 05:46
This was referenced Nov 1, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds the means to run the remaining evals without a local Bedrock key and fixes the smoke eval's must-not failure on
main:eval.yml):workflow_dispatchonly, in thecode-reviewenvironment. Choice inputs set the subset (smokeorre-review), verification, up to 2 runs, and a cap of $5, $10 or $25 per run. Inputs reach the shell only throughenv, and outputs upload only after the same secret scan the smoke eval uses. This unblocks Run the re-review evals and record what they show #123, Measure synthesis at a lower effort or on Sonnet #135 and Add a verification stage that proves or refutes each blocker and should-fix #41's on/off comparison.CODE_REVIEW_TIERS=synthesis=<tier>andCODE_REVIEW_SYNTHESIS_EFFORT. The production defaults don't change: synthesis stays on Opus with no effort sent.accepted-no-backfillin smoke run 41244686633 (Run a cost-capped smoke eval when a prompt, charter or model changes #52's gate).#ifbranches), so the scanner stays. The measurement found one real bug, a verbatim string starting with""read as a raw string, which this PR fixes.Closes #122
Refs #52
Refs #123
Refs #135
Refs #41
Refs #57