Skip to content

Test eval review cost and explain's answer - #115

Merged
polewskm merged 7 commits into
mainfrom
issue-107
Oct 31, 2026
Merged

polewskm merged 7 commits into
mainfrom
issue-107

Conversation

@polewskm

@polewskm polewskm commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Pins two behaviors that only main() held, so tests now reach them, with no behavior or log text changed:

  • Test that judge spend stays out of the eval review cost #107: the evals' per-case tally moves into tallyCase in evals/run.ts, with tests that review cost leaves out the concern and precision judges' calls while the case cost counts them.
  • Test /review explain's answer() directly #108: writeForPublish moves verbatim to src/outputs.ts, and the explain answer's writing moves into writeAnswer in src/explain.ts, with tests that an unknown finding writes only answer.json without a model call, trace.json is written when the call ran, answered is set, and results.json is never written.
  • The branch merges main in to resolve a CHANGELOG.md conflict, so merge this with a merge commit, not by rebasing.

Closes #107
Closes #108

A first review round tested that writeAnswer refuses a secret in the answer or trace; main is merged in after #104.

@polewskm
polewskm deployed to code-review October 31, 2026 21:46 — with GitHub Actions Active

@ncode-review ncode-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 NCode Review at d52b787: 2 new findings on the changed lines.

Comment thread test/explain.test.ts Resolved
Comment thread CHANGELOG.md Resolved
@ncode-review

ncode-review Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Superseded by the review at 5d49e7e.

@polewskm

Copy link
Copy Markdown
Contributor Author

/review

@ncode-review

ncode-review Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

🤖 Review complete

Verdict: ✅ Approve — Re-review of what changed since the last review at d52b787: 1 hunk in 1 file, by correctness only. Earlier findings: F1 fixed. No further findings above nits.

1 reviewer run — 0 findings (run details)

Reviewer Model Findings Top
correctness sonnet-5-5 0 🟢

Trial: findings are posted as review threads and in the check run. $0.03 on Amazon Bedrock, 8s, at 5d49e7e. (how to interact)

📋 Outcomes

Merged at 5d49e7e: 1 fixed, 1 dismissed, 0 acknowledged, 0 ignored.

Finding Reviewer Severity Outcome
F1 tests 🟡 Nit 👎 dismissed
F2 hygiene 🟡 Nit ✅ fixed

This branch was successfully deployed

1 active deployment
code-review — 5d49e7e7 Deployed Oct 31, 2026 by polewskm via review #337
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test /review explain's answer() directly Test that judge spend stays out of the eval review cost

1 participant