Skip to content

Rate test gaps over tested wrappers and covered paths as nits - #65

Merged
polewskm merged 6 commits into
mainfrom
issue-37
Oct 31, 2026
Merged

polewskm merged 6 commits into
mainfrom
issue-37

Conversation

@polewskm

Copy link
Copy Markdown
Contributor

Calibrates test-gap severity in the specialist prompt, the tests charter and synthesis: a missing test is should-fix only when the untested path is new behavior with a plausible user-visible failure and no test below it covers it; a wrapper over tested pieces, a test that only pins current behavior, or a second test for a covered path is a nit.
Eval concerns can now name a minSeverity (judged only against findings at that severity or above), and five cases from #30 cover both sides: two expected should-fixes (the inline newest-summary choice at 1c8c1bd, the unasserted POST body at 9f052ed) and the three threads cited in the issue as must-not-be-should-fix.
The paid eval (npm run eval) was not run; npm run typecheck and npm test pass.

Decisions:

  • Must-not concerns use minSeverity: should-fix, so the bot may still raise these gaps as nits; the eval would otherwise forbid them outright.
  • The two should-fix examples are Post a new summary comment for each review #30 test findings the maintainer fixed where the logic had no test below it, as the counterpart to the three inflated ones.
  • Each case reviews the PR base against the head the original finding was raised on, with the PR body as it read at that time.

Closes #37

A missing test is should-fix only when the untested path is new behavior with a plausible user-visible failure and nothing below it covers it. Eval concerns can name a minSeverity, and five cases from #30 check both sides of the line.
@polewskm
polewskm deployed to code-review October 31, 2026 04:13 — with GitHub Actions Active

@ncode-review ncode-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 NCode Review at 1840ef6: 5 new findings on the changed lines.

Comment thread evals/run.ts Resolved
Comment thread prompts/synthesis.md Resolved
Comment thread README.md Resolved
Comment thread CHANGELOG.md Resolved
Comment thread test/engine.test.ts Resolved
@ncode-review

ncode-review Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

🤖 Review complete

Verdict: ✅ Approve — No findings.

5 reviewers run — 0 findings (run details)

Reviewer Model Findings Top
conventions sonnet-5-5 0 🟢
correctness sonnet-5-5 0 🟢
hygiene haiku-5-5 0 🟢
intent haiku-5-5 0 🟢
rule-check sonnet-5-5 0 🟢

Skipped: api-compat, architecture, completeness, data-model, infrastructure, performance, security, tests

Trial: findings are posted as review threads and in the check run. $0.31 on Amazon Bedrock, 30s, at 502f569. (how to interact)

@polewskm

Copy link
Copy Markdown
Contributor Author

/review

@ncode-review ncode-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 NCode Review at 7aa264e: 1 new finding on the changed lines.
2 more were already posted on this pull request.

Comment thread reviewers/tests.md Resolved
@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: answering the review summary at #65 (comment)

@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: Round 2 answered. No blocker or should-fix is left open; the remaining threads are nits, answered above. Recommend merging once you've resolved them.

@polewskm
polewskm marked this pull request as ready for review October 31, 2026 06:36
@polewskm
polewskm deployed to code-review October 31, 2026 06:36 — with GitHub Actions Active
@polewskm
polewskm enabled auto-merge October 31, 2026 06:36
@polewskm
polewskm merged commit ad05fd4 into main Oct 31, 2026
5 checks passed
@polewskm
polewskm deleted the issue-37 branch October 31, 2026 07:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test-gap findings are rated should-fix for wrappers and paths already covered

1 participant