[polewskm]

Stacked on #73; GitHub retargets this to main when #73 merges.

Adds .github/workflows/smoke-eval.yml: on a push to main that changes prompts/**, reviewers/**, src/config.ts or guidance/**, and by workflow_dispatch, it runs 4 eval cases in the code-review environment (never on pull_request; permissions: {} at the top, contents: read + issues: read for the job; actions pinned by SHA). node evals/smoke.ts check compares the run with evals/baseline-smoke.json using #73's compare(), posts the table to $GITHUB_STEP_SUMMARY, and fails when caught drops by more than 2, the must-not case is flagged, or the case sets differ. node evals/smoke.ts scan evals/out runs assertNoSecrets on every file before the upload.
evals/run.ts gains --max-cost <usd> (stops after the case that crosses it, writes what ran, exits 1), --subset smoke (the named subset in evals/cases.ts) and --precision off. All of it is unit-tested hermetically in test/smoke-eval.test.ts. npm run typecheck and npm test (190 tests) pass.

Decisions:

• Smoke subset: accepted-no-backfill (the must-not case), heap-allocation-key-export (large diff: 43 files, about 2,600 changed lines, 161 KB), secret-buffer-length and test-gap-newest-summary-selection (small C# and TypeScript). I didn't pick the 2 MB #420 cases because one of them alone costs $6–7. The subset has 3 expected concerns, so "caught drops by more than 2" fails only when all 3 are lost. That is the spec's threshold (Evals: measure precision and findings per pull request, and compare two runs #73's noise limit).
• No baseline is checked in, because recording one needs a paid run. When evals/baseline-smoke.json is missing, the job passes with a ::notice and the run is still uploaded. It still fails on a flagged must-not case, since that check doesn't need a baseline. Every run writes proposed-baseline-smoke.json (the run without agent calls and raw findings) into the artifact. The README says how to adopt it.
• The workflow uses --guidance full, because guidance-holdout/ is derived, gitignored and costs a training run to build. It uses --precision off because the precision judge makes uncached Opus calls at up to $1.50 each and would exceed $8; the gate reads only caught and must-not. compare's note now says a missing precision may mean --precision off, and states the cost caveat only when reviewCostUsd is missing.
• --max-cost is checked between cases, since a half-finished review has nothing to score, so a run can overshoot by up to one case. --only now also fails on an unknown case id, so the subset can't quietly shrink.
• GH_TOKEN (the job token) lets the eval read the issues a case's PR closes. Agents still get the scrubbed environment from src/agent.ts.

Remaining (paid runs, not done):

• Push a change to prompts/synthesis.md to main and confirm the smoke eval costs under $8 and shows the comparison. If it goes over, swap a case in subsets.smoke or raise the cap.
• Adopt the first run's proposed-baseline-smoke.json as evals/baseline-smoke.json in a follow-up PR.
• Confirm a push touching only src/publish.ts doesn't start the workflow.

Refs #52

[ncode-review]

🤖 NCode Review at 7517ea0: 6 new findings on the changed lines.

Not on a changed line:

• 🟠 F3 Should fix: No test for --precision off evals/run.ts:183. When precisionOn is false, the run skips judgePosted and totalPrecision, leaves precision undefined and omits the precision line. No test exercises any of this. Fix: Move the per-case precision decision into an exported helper, or test with a stubbed realityJudge that precision is undefined and the judge is never called when precision is off.

[ncode-review]

🤖 Review complete

Verdict: 💬 Comment — This PR adds a cost-capped smoke eval workflow that runs on pushes to main that change prompts, charters, models or guidance. It also adds --subset, --precision off and --max-cost to evals/run.ts, and a gate script, evals/smoke.ts, that compares the run with a committed baseline and scans the outputs for secrets before upload. The change is well contained: argument parsing fails loudly instead of running uncapped, and the trigger and environment restrictions are clearly reasoned. The findings are about docs wording, test coverage and parameter style, not about correctness.

8 reviewers run — 8 findings (run details)

Reviewer	Model	Findings	Top
synthesis	opus-5-5	8	🟡
architecture	sonnet-5-5	1	🟡
completeness	sonnet-5-5	0	🟢
conventions	sonnet-5-5	1	🟡
correctness	sonnet-5-5	0	🟢
hygiene	haiku-5-5	2	🟡
rule-check	sonnet-5-5	3	🟡
security	sonnet-5-5	0	🟢
tests	haiku-5-5	2	🟡

Skipped: api-compat, data-model, infrastructure, intent, performance

Findings

• 🟡 F1 evals/run.ts:300: main's wiring of --precision off and the cap failure is tested only through helpers
• 🟡 F2 evals/smoke.ts:104: The precision branch of trimmed has no test
• 🟡 F3 README.md:207: The README's description of the --subset failure doesn't match the error
• 🟡 F4 evals/cases.ts:29: Subset data lives in the case-types module, apart from cases.json
• 🟡 F5 evals/run.ts:143: capOutcome takes four positional data parameters
• 🟡 F6 evals/run.ts:167: casePrecision takes a boolean whose only job is to switch behavior
• 🟡 F7 .github/workflows/smoke-eval.yml:47: Node major version is hard-coded again
• 🟡 F8 evals/smoke.ts:111: ENOENT-as-missing handling is duplicated

_{Trial: findings are posted as review threads and in the check run. $0.67 on Amazon Bedrock, 80s, at dae21e6. (how to interact)}

[polewskm]

Autopilot: answering #80 (review) and the summary at #80 (comment).

• F3 (no test for --precision off): fixed in 25d1086. The per-case decision is now the exported casePrecision(precisionOn, input), and the flag's parsing is parsePrecision. A test with a stubbed judge shows that with precision off the result is undefined and the judge is never called, and that with it on the same finding is judged. Another test covers parsePrecision's default and its refusal of anything but on or off.
• F1, F2, F4–F7 are answered in their threads: F1, F2 and F5 are fixed in 25d1086, and F4, F6 and F7 are declined with reasons.

npm run typecheck and npm test pass (197 tests). The branch is up to date with origin/issue-51.

[polewskm]

/review

[ncode-review]

🤖 NCode Review at 25d1086: 5 new findings on the changed lines.
1 more was already posted on this pull request.

[polewskm]

Autopilot: Round 2 answered. No blocker or should-fix is left open; the remaining threads are nits, answered above. Recommend merging once you've resolved them.

The should-fix (F1, the --max-cost stop in main()) is fixed in 2da07ec. F5 in the updated summary at #80 (comment) (Node version hard-coded in the workflow) repeats an earlier finding, declined at #80 (comment). npm run typecheck and npm test pass (254 tests); the branch is up to date with origin/main.

[polewskm]

/review

[ncode-review]

🤖 NCode Review at 2da07ec: 6 new findings on the changed lines.
1 more was already posted on this pull request.

[polewskm]

Autopilot: Round 3 answered. The review-round limit (3) is reached; no blocker or should-fix is left open. The should-fix (F1, an empty --only selecting zero cases) is fixed in dae21e6, and origin/main is merged in (8dd0346). The remaining threads (F2–F5, F7) are nits, declined above; worth follow-up issues: requiring the baseline once it's committed (F2), and a test of the upload step's scan guard (F4/F7). F6 in the summary at #80 (comment) repeats the Node-version nit answered in round 1. Recommend merging once you've resolved them.

[ncode-review]

🤖 NCode Review at dae21e6: 5 new findings on the changed lines.
3 more were already posted on this pull request.

[polewskm]

Autopilot: Round 4 answered. The review-round limit (3) is reached; no blocker or should-fix is left open. The five new threads (F1–F4, F6) are nits, declined above; F6 would undo round 1's --precision off fix. F5, F7 and F8 in the summary at #80 (comment) repeat earlier threads already answered. Worth follow-up issues: a test of main()'s post-loop write order (F1) and of trimmed keeping precision (F2). origin/main (#76, shadow scoring) is merged in (25463e2): evals/run.ts keeps both --subset/--max-cost/--precision off and --shadow with its shadow scoring. Recommend merging once you've resolved them.