Skip to content

Smoke eval baseline gate and writer checks - #121

Merged
polewskm merged 12 commits into
mainfrom
issue-93
Oct 31, 2026
Merged

polewskm merged 12 commits into
mainfrom
issue-93

Conversation

@polewskm

@polewskm polewskm commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

The smoke eval now fails when evals/baseline-smoke.json is missing after it was committed, naming the deleting commit (it still passes with a notice before the first baseline; a shallow checkout fails closed, so the workflow fetches full history). New tests pin the smoke eval workflow's order and that evals/run.ts writes its output before failing a run over --max-cost, via an exported writeRunOutput.
The loop limit, open-issue synthesis and Autopilot: replies now use the engine's writerCheck (association, then a cached permission lookup that fails closed), so a private organization member counts; open-issue author lookups are capped at 20. The /review gate stays association-only, and the README tells private members how to use it.
Two merge steps bring in origin/main, so this pull request must merge with a merge commit, not by rebasing.

Closes #93
Closes #89

A first review round moved the shared writer-check test stub into one module.

@polewskm
polewskm deployed to code-review October 31, 2026 23:11 — with GitHub Actions Active

@ncode-review ncode-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 NCode Review at d88b4de: 2 new findings on the changed lines.

Not on a changed line:

  • 🟡 F2 Nit: No test covers when main() reads the git history evals/smoke.ts:165. main() looks up deletedIn only when the baseline path is the committed default and the file is missing, and no test covers that condition. Its parts (smokeCheck, checkFiles with deletedIn, lastBaselineCommit) are each tested. Fix: Optional: move the condition into a small exported function and test it with the default path, a --baseline path and a file that exists.

Comment thread test/smoke-eval.test.ts Resolved
Comment thread test/issues.test.ts Resolved
@ncode-review

ncode-review Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Superseded by the review at fd9d544.

@polewskm

Copy link
Copy Markdown
Contributor Author

/review

@ncode-review

ncode-review Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

🤖 Review complete

Verdict: ✅ Approve — Re-review of what changed since the last review at d88b4de: 6 hunks in 4 files and changes taken back in 3 files, by correctness only. Earlier findings: F1 still open, F3 fixed. No further findings above nits.

1 reviewer run — 0 findings (run details)

Reviewer Model Findings Top
correctness sonnet-5-5 0 🟢

Trial: findings are posted as review threads and in the check run. $0.03 on Amazon Bedrock, 9s, at fd9d544. (how to interact)

📋 Outcomes

Merged at fd9d544: 1 fixed, 1 dismissed, 0 acknowledged, 0 ignored.

Finding Reviewer Severity Outcome
F1 architecture, tests 🟡 Nit 👎 dismissed
F3 hygiene 🟡 Nit ✅ fixed

This branch was successfully deployed

1 active deployment
code-review — fd9d5440 Deployed Oct 31, 2026 by polewskm via review #374
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant