Evals: measure precision and comments per pull request, and compare two setups #51

Background. Each upgrade (verification, Opus on sensitive paths, re-review models, and so on) is to be compared with the baseline on recall, precision, comments per PR and cost per run, and kept only where it catches more real bugs for its cost. Phase 2's exit criterion is eval precision and recall at or above the baseline.

Results so far:

• 18 cases (16 from Identity plus 2 OpenTofu), 55 expected findings on the Identity cases, 1 must-not case.
• Held-out guidance: 10 and 9 of 55 caught, at $31 and $30.
• Rule checks: 12 of 55, at $52.
• Variance between runs is about ±2 findings, so at least 2 runs per setup.

Current state. evals/run.ts reports caught/expected, caught before synthesis, must-not violations, total findings and cost. It has no precision measure (findings that match nothing expected aren't judged) and no tool to compare two runs.

What to do.

1. Judge every unmatched posted blocker and should-fix with an Opus 5.5 call that reads the checkout. Ask whether it is a real defect a maintainer would act on: real | not-real | unsure. Report precision as (matched + real) ÷ (all posted blockers and should-fixes), with unsure shown separately. Cache verdicts by case and finding fingerprint so reruns don't re-judge.
2. Report findings per case (mean, max) and cost per case.
3. Add npm run eval:compare -- <a.json> <b.json> that prints recall, precision, findings per PR, cost and the per-case differences, and flags differences inside ±2 as noise.
4. Let the maintainer confirm a sample of real and not-real verdicts. Confirmed ones go into cases.json as expected or must-not findings, growing the set.

Acceptance. evals/out/<label>.json contains precision and findings-per-PR fields, and eval:compare on two existing outputs prints a table.

Filed from the 2026-10-31 audit of the NCodeGroup review-bot and project-template design against what was built. This issue stands alone: everything needed to do it is above.