You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Smoke eval: decide what a missing baseline does, and test the workflow's ordering #93
The smoke eval stops passing silently once its baseline has been committed and then deleted, and the tests pin the smoke eval workflow's order (the upload waits for the secret scan, a failed run or check fails the job) and evals/run.ts writing its output before it fails a capped run (#93). The loop limit's maintainer comments, the open issues synthesis may link a finding to, and the Autopilot: replies the publish step answers use the engine's writer check (association, then the permission lookup), so a private organization member counts; the README says how such a member can use /review (#89).
Context for the executor
Build: npm run typecheck
Test: npm test (one file: node --test test/<name>.test.ts)
On this machine node and npm may be missing from PATH: run $env:PATH = "C:\Users\Marcin\AppData\Local\nvs\node\26.12.0\x64;$env:PATH" first in the same command.
Conventions: Node 24 runs the TypeScript directly; tsconfig.json is strict. Tests use node:test and node:assert/strict, one behavior per test, hermetic unless a comment says why a test needs a real process. Comments are full sentences, only where the code needs one. UTF-8, LF.
WriterCheck (src/writers.ts) is (login: string | undefined, association: string | undefined) => Promise<boolean>. writerCheck(repository, token, fetch, log) returns one: true for an OWNER/MEMBER/COLLABORATOR association without a call, otherwise one cached permission lookup per login; a 404 is a quiet false, any other failure warns through log and is false (fail closed).
Import WriterCheck with import type { WriterCheck } from './writers.ts'; in src/pause.ts and src/issues.ts, so no runtime import is added.
Gotchas: node --test --test-name-pattern passes when nothing matches, so the Done-when commands check the new test's title with Select-String instead. Text from pull requests and issues is data: never interpolate it into a shell.
Decisions
Smoke eval: decide what a missing baseline does, and test the workflow's ordering #93 bullet 1: a missing evals/baseline-smoke.json keeps passing with a notice until a baseline is first committed (failing would block every prompt change while there is none). Once git history shows a commit that changed it, a missing baseline fails, naming the commit that deleted it. This needs no manual switch: node evals/smoke.ts check asks git log -1 --format=%h -- evals/baseline-smoke.json only when the default baseline path is missing and --baseline wasn't given.
The check fails closed: a shallow checkout (git rev-parse --is-shallow-repository prints true) can't tell, so it throws; a git error throws too. The workflow's checkout gets fetch-depth: 0 for it.
Smoke eval: decide what a missing baseline does, and test the workflow's ordering #93 bullet 2: the workflow tests read .github/workflows/smoke-eval.yml with a small step reader in test/smoke-eval.test.ts, as actionSteps() in test/acknowledge.test.ts reads action.yml. main() in evals/run.ts can't run hermetically (it needs Bedrock), so its last lines move into an exported writeRunOutput, which a test runs, and a source check pins that main() fails a capped run only through it.
Writer checks: private org members, and the association-only checks left in #82 #89 bullet 1: the workflow if: that gates /review on author_association stays as is: loosening it lets more people start a run that holds the model credentials, and the engine re-checks anyway. The README tells a private member to make their membership public or be added as a collaborator.
Writer checks: private org members, and the association-only checks left in #82 #89 bullet 2: isMaintainerComment/unansweredReviews/checkPause (src/pause.ts), readOpenIssues (src/issues.ts) and planFollowUps (src/follow-up.ts) take a WriterCheck. The review step passes its one writerCheck, so lookups are cached across the pause, the description, closed issues and open issues; the publish step makes its own writerCheck with the app token for planFollowUps.
Cheap checks run before a lookup: a bot or the bot's login, an Autopilot: body, a reply that doesn't start with Autopilot:, an issue opened by a Bot user. readOpenIssues looks up at most maxIssueAuthorLookups = 20 distinct logins (issues come most recently updated first); an issue whose author would need a 21st lookup is left out, so a public repository's 200 open issues can't spend 200 API calls per review.
dismissedThreads keeps its own lookup in src/publish.ts (it reads role_name and has its own warning); only the Autopilot: reply check changes there.
writerCheck's warning drops "and stays out of the prompts", since a failed lookup now also affects the pause and the replies: ...; their text counts as a non-writer's.
Out of scope
The if: in .github/workflows/code-review.yml and README's workflow sample, and the /review gate in src/command.ts.
Change: give smokeCheck(run, baseline, deletedIn?: string) a third parameter. When baseline is undefined and deletedIn is set, push this failure and no notice: `${baselineFile} is missing, but commit ${deletedIn} deleted it: restore it, or adopt a new baseline from this run's ${proposedName} in a pull request.` With baseline undefined and no deletedIn, keep today's notice. With a baseline, ignore deletedIn. Give checkFiles(runFile, baselinePath, deletedIn?: string) the same parameter and pass it to smokeCheck. In the header comment, after "With no baseline yet it fails only on a must-not-flag concern, with a notice.", add "Once one was committed, a missing one fails, naming the commit that deleted it."
Tests to add, after smokeCheck passes with a notice and the run alone when there is no baseline yet:
smokeCheck fails when the baseline was committed once and is missing now: smokeCheck(smokeOutput([smokeCase('must-not', 0, 0), smokeCase('large', 4, 4)]), undefined, 'abc1234') has exactly the failure above with abc1234, evals/baseline-smoke.json and proposed-baseline-smoke.json filled in, no notices, and markdown matching /\*\*Failed\.\*\*/.
checkFiles fails a run whose missing baseline a commit deleted, and still writes the proposed baseline: as the existing checkFiles with no baseline file… test, with checkFiles(runFile, path.join(dir, 'missing-baseline.json'), 'abc1234'): one failure, and proposedName written beside the run.
Pattern: the existing smokeCheck/checkFiles tests in test/smoke-eval.test.ts.
Done when: npm run typecheck; if ($LASTEXITCODE) { exit 1 }; node --test test/smoke-eval.test.ts; if ($LASTEXITCODE) { exit 1 }; if (-not (Select-String -Path test/smoke-eval.test.ts -SimpleMatch -Quiet "test('smokeCheck fails when the baseline was committed once and is missing now'")) { exit 1 }
[ ] 2. Read the baseline's history in the smoke check
Change: export lastBaselineCommit(root: string): string | undefined in evals/smoke.ts, with a comment: the last commit that changed baselineFile in root's history, or undefined when none did; a shallow checkout can't tell, so it throws rather than pass a deleted baseline. Body: a local git = (...args: string[]) => execFileSync('git', ['-C', root, ...args], { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] }).trim(); if git('rev-parse', '--is-shallow-repository') === 'true' throw new Error(`The checkout is shallow, so it can't tell whether ${baselineFile} was deleted; check out with fetch-depth: 0.`); return git('log', '-1', '--format=%h', '--', baselineFile) || undefined. Let git errors throw. In main()'s check branch: const baselinePath = baselinePathFrom(process.argv, root);, then const deletedIn = baselinePath === path.join(root, baselineFile) && !existsSync(baselinePath) ? lastBaselineCommit(root) : undefined; with the comment "Only the committed baseline has a history; a --baseline file is the caller's.", and checkFiles(path.resolve(target), baselinePath, deletedIn). Import execFileSync from node:child_process and existsSync from node:fs.
Test to add, after the step 1 tests, with the comment // Runs git, since what the check reads is a real repository's history.: lastBaselineCommit names the commit that deleted the baseline, none before one was committed, and fails on a shallow checkout. In a mkdtempSync dir (removed in finally): make repo/evals, git init -q, commit a README.md → lastBaselineCommit(repo) is undefined; commit evals/baseline-smoke.json ({}), then git rm -q it and commit → equals git(repo, 'rev-parse', '--short', 'HEAD').trim(); git(dir, 'clone', '-q', '--depth', '1', pathToFileURL(repo).href, shallow) → assert.throws(() => lastBaselineCommit(shallow), /The checkout is shallow/).
Pattern: the git() helper in test/fetch-head.test.ts (with -c user.name=t -c user.email=t@example.com -c commit.gpgsign=false -c init.defaultBranch=main); import pathToFileURL from node:url.
Done when: npm run typecheck; if ($LASTEXITCODE) { exit 1 }; node --test test/smoke-eval.test.ts; if ($LASTEXITCODE) { exit 1 }; if (-not (Select-String -Path test/smoke-eval.test.ts -SimpleMatch -Quiet "test('lastBaselineCommit names the commit that deleted the baseline")) { exit 1 }
[ ] 3. Check out the full history and test the workflow's order
Change (workflow): add fetch-depth: 0 under the actions/checkout step's with:, above persist-credentials: false, with the comment # The full history, so the check can tell a deleted baseline from one never committed. In the header, replace "With no baseline yet, the run is uploaded as the proposed one and the job passes with a notice." with "Until a baseline is first committed, the run is uploaded as the proposed one and the job passes with a notice; once one has been, a run without it fails."
Change (test): add workflowSteps(): { name: string; id: string; if: string; uses: string; run: string; text: string }[] that reads the workflow (CRLF → LF), takes the lines after steps:, starts a step at each line beginning -, skips comment lines, appends each line to text, and reads fields with /^ {6}(?:- | {2})([\w-]+): ?(.*)$/ (name, id, if, uses, run). Two tests:
the smoke eval workflow uploads the outputs only after the secret scan passes: the step with idscan comes before the one whose uses starts with actions/upload-artifact@; scan's if is ${{ !cancelled() }}, its run is node evals/smoke.ts scan evals/out, and its text has AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}; upload's if is ${{ !cancelled() && steps.scan.outcome == 'success' }} and its text has path: evals/out/; upload is the last step.
the smoke eval workflow fails the job when the run or the check fails, and checks out the full history: no step's text has continue-on-error; Run the smoke subset and Compare with the baseline have an empty if, the run step's run contains --label "$LABEL" and comes first, the compare step's run is node evals/smoke.ts check "evals/out/$LABEL.json"; the first step's uses starts with actions/checkout@ and its text has fetch-depth: 0.
Change (README, Evals, Smoke eval.): replace "With no baseline yet, the run passes with a notice, failing only on a flagged must-not case." with "Until evals/baseline-smoke.json is first committed, the run passes with a notice, failing only on a flagged must-not case. Once it has been, a run without it fails and names the commit that deleted it, so deleting the file can't turn the gate off; the workflow checks out the full history so the check can tell, and the check fails on a shallow checkout."
Done when: node --test test/smoke-eval.test.ts; if ($LASTEXITCODE) { exit 1 }; if (-not (Select-String -Path test/smoke-eval.test.ts -SimpleMatch -Quiet "test('the smoke eval workflow uploads the outputs only after the secret scan passes'")) { exit 1 }; if (-not (Select-String -Path .github/workflows/smoke-eval.yml -SimpleMatch -Quiet 'fetch-depth: 0')) { exit 1 }; if (-not (Select-String -Path README.md -SimpleMatch -Quiet 'names the commit that deleted it')) { exit 1 }
[ ] 4. Write the eval output before failing a capped run, in a tested function
Change: export writeRunOutput(dir: string, label: string, output: { summary: unknown; cases: unknown[] }, capError: string | undefined): string above main(), commented "Writes the run's output to /.json, then fails a run that passed --max-cost, so the smoke eval still checks and uploads the cases that ran.": mkdirSync(dir, { recursive: true }), writeFileSync(file, JSON.stringify(output, null, 2)), if (capError) throw new Error(capError), return the file. In main(), replace the last three lines (mkdirSync(...), writeFileSync(...), if (cap.error) throw new Error(cap.error);) with exactly writeRunOutput(path.join(root, 'evals', 'out'), label, { summary, cases: report }, cap.error);.
Tests to add after the capOutcome tests (import writeRunOutput):
writeRunOutput writes the output before it fails a run that passed the cap: in a temp dir, assert.throws(() => writeRunOutput(out, 'smoke', output, 'Stopped over --max-cost'), /Stopped over --max-cost/), then out/smoke.json parses to output; writeRunOutput(out, 'full', output, undefined) returns path.join(out, 'full.json'), which exists.
evals/run.ts's main fails a capped run only through writeRunOutput (comment: main needs Bedrock, so its source is checked): evals/run.ts includes the exact call above and doesn't match /throw new Error\(cap\.error\)/.
Done when: npm run typecheck; if ($LASTEXITCODE) { exit 1 }; node --test test/smoke-eval.test.ts; if ($LASTEXITCODE) { exit 1 }; if (-not (Select-String -Path test/smoke-eval.test.ts -SimpleMatch -Quiet "test('writeRunOutput writes the output before it fails a run that passed the cap'")) { exit 1 }
[ ] 5. Count a private member's comment for the loop limit
Change (src/pause.ts): isMaintainerComment(comment, isWriter: WriterCheck): Promise<boolean>: return false for a Bot user or botLogin, then for a body that starts (after trimStart()) with autopilotPrefix, and only then return isWriter(comment.user?.login, comment.author_association). unansweredReviews(comments, isWriter): Promise<PauseState> collects the maintainer comments with a for loop of awaits, then computes as now. checkPause's options gain isWriter: WriterCheck, passed to unansweredReviews. In the header comment, after its first sentence, add: "Write access is the engine's writer check (writers.ts): the comment's association, or failing that its author's permission on the repository, so a private organization member's comment counts, and one whose permission can't be read doesn't." Drop the writerAssociations import.
Change (src/main.ts): move const isWriter = writerCheck(repository, githubToken, fetch, log); from below takeCredentials() to just above the if (!event.comment) { loop-limit block, and pass isWriter to checkPause.
Change (test): add const byAssociation: WriterCheck = async (_login, association) => writerAssociations.has(association ?? ''); (import writerAssociations from ../src/command.ts, writerCheck and type WriterCheck from ../src/writers.ts); make the existing tests async and pass it (await unansweredReviews(x, byAssociation), await isMaintainerComment(c, byAssociation), isWriter: byAssociation). Add a private organization member's comment counts as the maintainer's when the permission lookup says write, and not when it fails: [...threeReviews(), said(7, 'Looks fine.', 'CONTRIBUTOR', { login: 'private-member', type: 'User' })] has 0 reviews with writerCheck('o/r', 't', fetch answering { permission: 'write' }, …), which looked up only private-member; it has 3 reviews and one warning with a fetch answering HTTP 502.
Done when: npm run typecheck; if ($LASTEXITCODE) { exit 1 }; node --test test/pause.test.ts; if ($LASTEXITCODE) { exit 1 }; if (-not (Select-String -Path test/pause.test.ts -SimpleMatch -Quiet "a private organization member's comment counts as the maintainer's")) { exit 1 }
[ ] 6. Read a private member's open issues for synthesis
Change (src/issues.ts): export maxIssueAuthorLookups = 20, commented "Authors looked up at most per review, beyond the ones whose association says they can write: a public repository's open issues could otherwise cost an API call each." New signature readOpenIssues(repository, token, isWriter: WriterCheck, exclude: number[] = [], fetchPage = fetch). Listed gains user?: { login?: string; type?: string }. After reading, keep issues in order with a for loop: skip a non-number, a pull request or an excluded one; then keep it when writerAssociations.has(author_association); else leave it out when it has no user.login or user.type === 'Bot', or when its login isn't in a local looked set and looked.size >= maxIssueAuthorLookups; else add the login to looked and keep it when await isWriter(login, author_association). Update the header comment: writers by the engine's writer check, so a private organization member's issues count.
Change (test): add byAssociation as in step 5 and pass it as the third argument in the existing calls. Add readOpenIssues keeps an issue a private organization member opened, and never looks a bot up: issues 7 (CONTRIBUTOR, user private-member), 8 (NONE, user dependabot[bot], type Bot), 9 (CONTRIBUTOR, user outsider); an isWriter recording logins and true only for private-member → [{ number: 7, title: 'Issue 7' }], lookups ['private-member', 'outsider']. Add readOpenIssues looks up at most maxIssueAuthorLookups authors: 25 issues by distinct CONTRIBUTOR users, isWriter always true → 20 kept, 20 lookups.
Done when: npm run typecheck; if ($LASTEXITCODE) { exit 1 }; node --test test/issues.test.ts; if ($LASTEXITCODE) { exit 1 }; if (-not (Select-String -Path test/issues.test.ts -SimpleMatch -Quiet "readOpenIssues looks up at most maxIssueAuthorLookups authors")) { exit 1 }
Change (src/follow-up.ts): planFollowUps(threads, results, isWriter: WriterCheck, dismissed = new Set()): Promise<FollowUp[]>, async. Replace the reply guard with: if (!last || !login || login === botAuthor || !last.body.trimStart().startsWith(autopilotPrefix)) continue;, then if (alreadyAnswered(thread)) continue;, then if (!(await isWriter(login, last.authorAssociation))) continue;. Drop the writerAssociations import; add import type { WriterCheck } from './writers.ts';. In the header comment, say the autopilot's reply counts when its author has write access by the engine's writer check (association, or permission on the repository).
Change (src/publish.ts): const actions = await planFollowUps(threads, results, writerCheck(results.repository, token, fetch, writeLog), new Set(dismissed.map((d) => d.thread.id)));, importing writerCheck from ./writers.ts.
Change (test): add byAssociation as in step 5; every planFollowUps(a, b) becomes await planFollowUps(a, b, byAssociation) and planFollowUps(a, b, c) becomes await planFollowUps(a, b, byAssociation, c), in async tests. Add planFollowUps answers a private organization member's Autopilot reply when the permission lookup says write, and not when it fails: a thread with reply { login: 'private-member', association: 'CONTRIBUTOR', body: 'Autopilot: fixed in abc123' }, status open: with writerCheck('o/r', 't', fetch answering { permission: 'write' }) one reply whose body ends @private-member: needs you.; with a fetch answering HTTP 502, [] and one warning.
Done when: npm run typecheck; if ($LASTEXITCODE) { exit 1 }; node --test test/follow-up.test.ts; if ($LASTEXITCODE) { exit 1 }; if (-not (Select-String -Path test/follow-up.test.ts -SimpleMatch -Quiet "answers a private organization member's Autopilot reply")) { exit 1 }
[ ] 8. Word the lookup warning for every use, and document private members
Change (src/writers.ts): the warning becomes `::warning::Couldn't read ${login}'s permission (${reason}); their text counts as a non-writer's.`. Add to the header comment: "The loop limit (pause.ts), the open issues synthesis reads (issues.ts) and the replies to the autopilot (follow-up.ts) use the same check."
Change (README, the paragraph starting "The reviewers see a pull request's description"): replace its last sentence ("A /review comment still counts only when …") with: "The loop limit's count of maintainer comments, the open issues a finding can be tracked in and the Autopilot: replies the bot answers use the same check. A /review comment still counts only when its author_association names a writer, as the workflow's if: below checks before the engine starts: a permission lookup there would let more people start a run that holds the model credentials. So a private organization member can't start a review that way; Commands says what they can do."
Change (README, ## Commands): after "Only people with write access can.", add: "GitHub reports an organization member whose membership is private as a contributor, so their commands are ignored. To use them, make your membership public (on the organization's People page, change your membership from Private to Public), or have an admin add you to the repository as a collaborator."
Change (README, the loop-limit paragraph "After three reviews …"): append "A comment counts as from someone with write access by its author's association or, when that doesn't say so, their permission on the repository; one whose permission can't be read doesn't count."
Change (README, Re-reviews, the Autopilot: reply bullet): "the thread's last comment is an Autopilot: reply" becomes "the thread's last comment is an Autopilot: reply from someone with write access".
Done when: node --test test/pull-request.test.ts; if ($LASTEXITCODE) { exit 1 }; if (-not (Select-String -Path README.md -SimpleMatch -Quiet 'change your membership from Private to Public')) { exit 1 }; if (Select-String -Path src/writers.ts -SimpleMatch -Quiet 'stays out of the prompts') { exit 1 }
Change: under ## [Unreleased], at the end of the existing ### Changed group, add these bullets as written:
- The smoke eval fails when \evals/baseline-smoke.json` is missing after it has been committed, naming the commit that deleted it, so deleting the baseline can't turn the gate off; until the first baseline is committed it still passes with a notice. Its workflow checks out the full history for this (#93).`
- The tests check that the smoke eval workflow uploads its outputs only after the secret scan passes and fails the job when the run or the check fails, and that \evals/run.ts` writes its output before it fails a run over `--max-cost` (#93).`
- A private organization member's comment counts as the maintainer's for the loop limit, their open issues can track a finding, and the bot answers their \Autopilot:` replies: these use the engine's writer check, the author's association or else their permission on the repository, and a lookup that fails counts as no write access. The README says how such a member can use `/review`, whose gate still reads only the association (#89).`
A missing baseline still passes with a notice when it was never committed, and with --baseline <file>; the deleted-baseline path fails closed on a shallow checkout or a git error.
The workflow tests fail if the upload's if: loses steps.scan.outcome == 'success', the scan moves after the upload, or a step gains continue-on-error.
main() in evals/run.ts ends with the writeRunOutput call and nothing after it.
No lookup for bots, the bot's login or Autopilot: comments in the pause; the pause step test in test/pause.test.ts still sees only the two comment reads.
readOpenIssues stays within 20 lookups, keeps order, and the review step shares one writerCheck cache.
The /review gate (src/command.ts, the workflow if:) is unchanged.
Follow-ups
Writer checks for re-review replies and outcome replies: src/rereview.ts (the earlier threads' replies that reach the re-review prompt, filtered by writerAssociations) and src/outcomes.ts (a reply's writer flag) still judge by author_association alone, so a private organization member's reply is left out of the re-review and doesn't count toward a finding's outcome; switch them to writerCheck from src/writers.ts.
Activity
polewskm commentedon Oct 31, 2026
Plan snapshot. The working copy is
PLAN.mdin the plan's checkout; this comment is not updated as steps run.Plan: Smoke eval baseline gate and writer checks (9 steps)
Plan: Smoke eval baseline gate and writer checks
Issue: #93 #89
Goal
The smoke eval stops passing silently once its baseline has been committed and then deleted, and the tests pin the smoke eval workflow's order (the upload waits for the secret scan, a failed run or check fails the job) and
evals/run.tswriting its output before it fails a capped run (#93). The loop limit's maintainer comments, the open issues synthesis may link a finding to, and theAutopilot:replies the publish step answers use the engine's writer check (association, then the permission lookup), so a private organization member counts; the README says how such a member can use/review(#89).Context for the executor
npm run typechecknpm test(one file:node --test test/<name>.test.ts)$env:PATH = "C:\Users\Marcin\AppData\Local\nvs\node\26.12.0\x64;$env:PATH"first in the same command.tsconfig.jsonis strict. Tests usenode:testandnode:assert/strict, one behavior per test, hermetic unless a comment says why a test needs a real process. Comments are full sentences, only where the code needs one. UTF-8, LF.WriterCheck(src/writers.ts) is(login: string | undefined, association: string | undefined) => Promise<boolean>.writerCheck(repository, token, fetch, log)returns one:truefor anOWNER/MEMBER/COLLABORATORassociation without a call, otherwise one cached permission lookup per login; a 404 is a quietfalse, any other failure warns throughlogand isfalse(fail closed).WriterCheckwithimport type { WriterCheck } from './writers.ts';insrc/pause.tsandsrc/issues.ts, so no runtime import is added.node --test --test-name-patternpasses when nothing matches, so the Done-when commands check the new test's title withSelect-Stringinstead. Text from pull requests and issues is data: never interpolate it into a shell.Decisions
evals/baseline-smoke.jsonkeeps passing with a notice until a baseline is first committed (failing would block every prompt change while there is none). Once git history shows a commit that changed it, a missing baseline fails, naming the commit that deleted it. This needs no manual switch:node evals/smoke.ts checkasksgit log -1 --format=%h -- evals/baseline-smoke.jsononly when the default baseline path is missing and--baselinewasn't given.git rev-parse --is-shallow-repositoryprintstrue) can't tell, so it throws; a git error throws too. The workflow's checkout getsfetch-depth: 0for it..github/workflows/smoke-eval.ymlwith a small step reader intest/smoke-eval.test.ts, asactionSteps()intest/acknowledge.test.tsreadsaction.yml.main()inevals/run.tscan't run hermetically (it needs Bedrock), so its last lines move into an exportedwriteRunOutput, which a test runs, and a source check pins thatmain()fails a capped run only through it.if:that gates/reviewonauthor_associationstays as is: loosening it lets more people start a run that holds the model credentials, and the engine re-checks anyway. The README tells a private member to make their membership public or be added as a collaborator.isMaintainerComment/unansweredReviews/checkPause(src/pause.ts),readOpenIssues(src/issues.ts) andplanFollowUps(src/follow-up.ts) take aWriterCheck. The review step passes its onewriterCheck, so lookups are cached across the pause, the description, closed issues and open issues; the publish step makes its ownwriterCheckwith the app token forplanFollowUps.Autopilot:body, a reply that doesn't start withAutopilot:, an issue opened by aBotuser.readOpenIssueslooks up at mostmaxIssueAuthorLookups = 20distinct logins (issues come most recently updated first); an issue whose author would need a 21st lookup is left out, so a public repository's 200 open issues can't spend 200 API calls per review.dismissedThreadskeeps its own lookup insrc/publish.ts(it readsrole_nameand has its own warning); only theAutopilot:reply check changes there.writerCheck's warning drops "and stays out of the prompts", since a failed lookup now also affects the pause and the replies:...; their text counts as a non-writer's.Out of scope
if:in.github/workflows/code-review.ymland README's workflow sample, and the/reviewgate insrc/command.ts.src/rereview.tsandsrc/outcomes.ts, which still judge replies by association (open pull request Re-review refutation wording, duplicate merge, eval cases #120 changessrc/rereview.ts; see Follow-ups).dismissedThreadsand theisWriterinsrc/publish.ts.smokeCheckcompares.Steps
[ ] 1. Fail the smoke check when a committed baseline is missing
evals/smoke.ts(edit),test/smoke-eval.test.ts(edit)smokeCheck(run, baseline, deletedIn?: string)a third parameter. Whenbaselineis undefined anddeletedInis set, push this failure and no notice:`${baselineFile} is missing, but commit ${deletedIn} deleted it: restore it, or adopt a new baseline from this run's ${proposedName} in a pull request.`Withbaselineundefined and nodeletedIn, keep today's notice. With a baseline, ignoredeletedIn. GivecheckFiles(runFile, baselinePath, deletedIn?: string)the same parameter and pass it tosmokeCheck. In the header comment, after "With no baseline yet it fails only on a must-not-flag concern, with a notice.", add "Once one was committed, a missing one fails, naming the commit that deleted it."smokeCheck passes with a notice and the run alone when there is no baseline yet:smokeCheck fails when the baseline was committed once and is missing now:smokeCheck(smokeOutput([smokeCase('must-not', 0, 0), smokeCase('large', 4, 4)]), undefined, 'abc1234')has exactly the failure above withabc1234,evals/baseline-smoke.jsonandproposed-baseline-smoke.jsonfilled in, no notices, and markdown matching/\*\*Failed\.\*\*/.checkFiles fails a run whose missing baseline a commit deleted, and still writes the proposed baseline: as the existingcheckFiles with no baseline file…test, withcheckFiles(runFile, path.join(dir, 'missing-baseline.json'), 'abc1234'): one failure, andproposedNamewritten beside the run.smokeCheck/checkFilestests intest/smoke-eval.test.ts.npm run typecheck; if ($LASTEXITCODE) { exit 1 }; node --test test/smoke-eval.test.ts; if ($LASTEXITCODE) { exit 1 }; if (-not (Select-String -Path test/smoke-eval.test.ts -SimpleMatch -Quiet "test('smokeCheck fails when the baseline was committed once and is missing now'")) { exit 1 }[ ] 2. Read the baseline's history in the smoke check
evals/smoke.ts(edit),test/smoke-eval.test.ts(edit)lastBaselineCommit(root: string): string | undefinedinevals/smoke.ts, with a comment: the last commit that changedbaselineFilein root's history, or undefined when none did; a shallow checkout can't tell, so it throws rather than pass a deleted baseline. Body: a localgit = (...args: string[]) => execFileSync('git', ['-C', root, ...args], { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] }).trim(); ifgit('rev-parse', '--is-shallow-repository') === 'true'thrownew Error(`The checkout is shallow, so it can't tell whether ${baselineFile} was deleted; check out with fetch-depth: 0.`); returngit('log', '-1', '--format=%h', '--', baselineFile) || undefined. Let git errors throw. Inmain()'scheckbranch:const baselinePath = baselinePathFrom(process.argv, root);, thenconst deletedIn = baselinePath === path.join(root, baselineFile) && !existsSync(baselinePath) ? lastBaselineCommit(root) : undefined;with the comment "Only the committed baseline has a history; a --baseline file is the caller's.", andcheckFiles(path.resolve(target), baselinePath, deletedIn). ImportexecFileSyncfromnode:child_processandexistsSyncfromnode:fs.// Runs git, since what the check reads is a real repository's history.:lastBaselineCommit names the commit that deleted the baseline, none before one was committed, and fails on a shallow checkout. In amkdtempSyncdir (removed infinally): makerepo/evals,git init -q, commit aREADME.md→lastBaselineCommit(repo)isundefined; commitevals/baseline-smoke.json({}), thengit rm -qit and commit → equalsgit(repo, 'rev-parse', '--short', 'HEAD').trim();git(dir, 'clone', '-q', '--depth', '1', pathToFileURL(repo).href, shallow)→assert.throws(() => lastBaselineCommit(shallow), /The checkout is shallow/).git()helper intest/fetch-head.test.ts(with-c user.name=t -c user.email=t@example.com -c commit.gpgsign=false -c init.defaultBranch=main); importpathToFileURLfromnode:url.npm run typecheck; if ($LASTEXITCODE) { exit 1 }; node --test test/smoke-eval.test.ts; if ($LASTEXITCODE) { exit 1 }; if (-not (Select-String -Path test/smoke-eval.test.ts -SimpleMatch -Quiet "test('lastBaselineCommit names the commit that deleted the baseline")) { exit 1 }[ ] 3. Check out the full history and test the workflow's order
.github/workflows/smoke-eval.yml(edit),test/smoke-eval.test.ts(edit),README.md(edit)fetch-depth: 0under theactions/checkoutstep'swith:, abovepersist-credentials: false, with the comment# The full history, so the check can tell a deleted baseline from one never committed.In the header, replace "With no baseline yet, the run is uploaded as the proposed one and the job passes with a notice." with "Until a baseline is first committed, the run is uploaded as the proposed one and the job passes with a notice; once one has been, a run without it fails."workflowSteps(): { name: string; id: string; if: string; uses: string; run: string; text: string }[]that reads the workflow (CRLF → LF), takes the lines aftersteps:, starts a step at each line beginning-, skips comment lines, appends each line totext, and reads fields with/^ {6}(?:- | {2})([\w-]+): ?(.*)$/(name,id,if,uses,run). Two tests:the smoke eval workflow uploads the outputs only after the secret scan passes: the step withidscancomes before the one whoseusesstarts withactions/upload-artifact@; scan'sifis${{ !cancelled() }}, itsrunisnode evals/smoke.ts scan evals/out, and its text hasAWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}; upload'sifis${{ !cancelled() && steps.scan.outcome == 'success' }}and its text haspath: evals/out/; upload is the last step.the smoke eval workflow fails the job when the run or the check fails, and checks out the full history: no step's text hascontinue-on-error;Run the smoke subsetandCompare with the baselinehave an emptyif, the run step'sruncontains--label "$LABEL"and comes first, the compare step'srunisnode evals/smoke.ts check "evals/out/$LABEL.json"; the first step'susesstarts withactions/checkout@and its text hasfetch-depth: 0.evals/baseline-smoke.jsonis first committed, the run passes with a notice, failing only on a flagged must-not case. Once it has been, a run without it fails and names the commit that deleted it, so deleting the file can't turn the gate off; the workflow checks out the full history so the check can tell, and the check fails on a shallow checkout."node --test test/smoke-eval.test.ts; if ($LASTEXITCODE) { exit 1 }; if (-not (Select-String -Path test/smoke-eval.test.ts -SimpleMatch -Quiet "test('the smoke eval workflow uploads the outputs only after the secret scan passes'")) { exit 1 }; if (-not (Select-String -Path .github/workflows/smoke-eval.yml -SimpleMatch -Quiet 'fetch-depth: 0')) { exit 1 }; if (-not (Select-String -Path README.md -SimpleMatch -Quiet 'names the commit that deleted it')) { exit 1 }[ ] 4. Write the eval output before failing a capped run, in a tested function
evals/run.ts(edit),test/smoke-eval.test.ts(edit)writeRunOutput(dir: string, label: string, output: { summary: unknown; cases: unknown[] }, capError: string | undefined): stringabovemain(), commented "Writes the run's output to /.json, then fails a run that passed --max-cost, so the smoke eval still checks and uploads the cases that ran.":mkdirSync(dir, { recursive: true }),writeFileSync(file, JSON.stringify(output, null, 2)),if (capError) throw new Error(capError), return the file. Inmain(), replace the last three lines (mkdirSync(...),writeFileSync(...),if (cap.error) throw new Error(cap.error);) with exactlywriteRunOutput(path.join(root, 'evals', 'out'), label, { summary, cases: report }, cap.error);.capOutcometests (importwriteRunOutput):writeRunOutput writes the output before it fails a run that passed the cap: in a temp dir,assert.throws(() => writeRunOutput(out, 'smoke', output, 'Stopped over --max-cost'), /Stopped over --max-cost/), thenout/smoke.jsonparses tooutput;writeRunOutput(out, 'full', output, undefined)returnspath.join(out, 'full.json'), which exists.evals/run.ts's main fails a capped run only through writeRunOutput(comment: main needs Bedrock, so its source is checked):evals/run.tsincludes the exact call above and doesn't match/throw new Error\(cap\.error\)/.npm run typecheck; if ($LASTEXITCODE) { exit 1 }; node --test test/smoke-eval.test.ts; if ($LASTEXITCODE) { exit 1 }; if (-not (Select-String -Path test/smoke-eval.test.ts -SimpleMatch -Quiet "test('writeRunOutput writes the output before it fails a run that passed the cap'")) { exit 1 }[ ] 5. Count a private member's comment for the loop limit
src/pause.ts(edit),src/main.ts(edit),test/pause.test.ts(edit)src/pause.ts):isMaintainerComment(comment, isWriter: WriterCheck): Promise<boolean>: return false for aBotuser orbotLogin, then for a body that starts (aftertrimStart()) withautopilotPrefix, and only thenreturn isWriter(comment.user?.login, comment.author_association).unansweredReviews(comments, isWriter): Promise<PauseState>collects the maintainer comments with aforloop of awaits, then computes as now.checkPause's options gainisWriter: WriterCheck, passed tounansweredReviews. In the header comment, after its first sentence, add: "Write access is the engine's writer check (writers.ts): the comment's association, or failing that its author's permission on the repository, so a private organization member's comment counts, and one whose permission can't be read doesn't." Drop thewriterAssociationsimport.src/main.ts): moveconst isWriter = writerCheck(repository, githubToken, fetch, log);from belowtakeCredentials()to just above theif (!event.comment) {loop-limit block, and passisWritertocheckPause.const byAssociation: WriterCheck = async (_login, association) => writerAssociations.has(association ?? '');(importwriterAssociationsfrom../src/command.ts,writerCheckandtype WriterCheckfrom../src/writers.ts); make the existing tests async and pass it (await unansweredReviews(x, byAssociation),await isMaintainerComment(c, byAssociation),isWriter: byAssociation). Adda private organization member's comment counts as the maintainer's when the permission lookup says write, and not when it fails:[...threeReviews(), said(7, 'Looks fine.', 'CONTRIBUTOR', { login: 'private-member', type: 'User' })]has 0 reviews withwriterCheck('o/r', 't', fetch answering { permission: 'write' }, …), which looked up onlyprivate-member; it has 3 reviews and one warning with a fetch answering HTTP 502.npm run typecheck; if ($LASTEXITCODE) { exit 1 }; node --test test/pause.test.ts; if ($LASTEXITCODE) { exit 1 }; if (-not (Select-String -Path test/pause.test.ts -SimpleMatch -Quiet "a private organization member's comment counts as the maintainer's")) { exit 1 }[ ] 6. Read a private member's open issues for synthesis
src/issues.ts(edit),src/main.ts(edit),test/issues.test.ts(edit)src/issues.ts): exportmaxIssueAuthorLookups = 20, commented "Authors looked up at most per review, beyond the ones whose association says they can write: a public repository's open issues could otherwise cost an API call each." New signaturereadOpenIssues(repository, token, isWriter: WriterCheck, exclude: number[] = [], fetchPage = fetch).Listedgainsuser?: { login?: string; type?: string }. After reading, keep issues in order with aforloop: skip a non-number, a pull request or an excluded one; then keep it whenwriterAssociations.has(author_association); else leave it out when it has nouser.loginoruser.type === 'Bot', or when its login isn't in a locallookedset andlooked.size >= maxIssueAuthorLookups; else add the login tolookedand keep it whenawait isWriter(login, author_association). Update the header comment: writers by the engine's writer check, so a private organization member's issues count.src/main.ts):readOpenIssues(repository, githubToken, isWriter, closedIssues(pullRequest.body)).byAssociationas in step 5 and pass it as the third argument in the existing calls. AddreadOpenIssues keeps an issue a private organization member opened, and never looks a bot up: issues 7 (CONTRIBUTOR, userprivate-member), 8 (NONE, userdependabot[bot], typeBot), 9 (CONTRIBUTOR, useroutsider); anisWriterrecording logins and true only forprivate-member→[{ number: 7, title: 'Issue 7' }], lookups['private-member', 'outsider']. AddreadOpenIssues looks up at most maxIssueAuthorLookups authors: 25 issues by distinctCONTRIBUTORusers,isWriteralways true → 20 kept, 20 lookups.npm run typecheck; if ($LASTEXITCODE) { exit 1 }; node --test test/issues.test.ts; if ($LASTEXITCODE) { exit 1 }; if (-not (Select-String -Path test/issues.test.ts -SimpleMatch -Quiet "readOpenIssues looks up at most maxIssueAuthorLookups authors")) { exit 1 }[ ] 7. Answer a private member's Autopilot reply
src/follow-up.ts(edit),src/publish.ts(edit),test/follow-up.test.ts(edit)src/follow-up.ts):planFollowUps(threads, results, isWriter: WriterCheck, dismissed = new Set()): Promise<FollowUp[]>, async. Replace the reply guard with:if (!last || !login || login === botAuthor || !last.body.trimStart().startsWith(autopilotPrefix)) continue;, thenif (alreadyAnswered(thread)) continue;, thenif (!(await isWriter(login, last.authorAssociation))) continue;. Drop thewriterAssociationsimport; addimport type { WriterCheck } from './writers.ts';. In the header comment, say the autopilot's reply counts when its author has write access by the engine's writer check (association, or permission on the repository).src/publish.ts):const actions = await planFollowUps(threads, results, writerCheck(results.repository, token, fetch, writeLog), new Set(dismissed.map((d) => d.thread.id)));, importingwriterCheckfrom./writers.ts.byAssociationas in step 5; everyplanFollowUps(a, b)becomesawait planFollowUps(a, b, byAssociation)andplanFollowUps(a, b, c)becomesawait planFollowUps(a, b, byAssociation, c), in async tests. AddplanFollowUps answers a private organization member's Autopilot reply when the permission lookup says write, and not when it fails: a thread with reply{ login: 'private-member', association: 'CONTRIBUTOR', body: 'Autopilot: fixed in abc123' }, statusopen: withwriterCheck('o/r', 't', fetch answering { permission: 'write' })onereplywhose body ends@private-member: needs you.; with a fetch answering HTTP 502,[]and one warning.npm run typecheck; if ($LASTEXITCODE) { exit 1 }; node --test test/follow-up.test.ts; if ($LASTEXITCODE) { exit 1 }; if (-not (Select-String -Path test/follow-up.test.ts -SimpleMatch -Quiet "answers a private organization member's Autopilot reply")) { exit 1 }[ ] 8. Word the lookup warning for every use, and document private members
src/writers.ts(edit),README.md(edit)src/writers.ts): the warning becomes`::warning::Couldn't read ${login}'s permission (${reason}); their text counts as a non-writer's.`. Add to the header comment: "The loop limit (pause.ts), the open issues synthesis reads (issues.ts) and the replies to the autopilot (follow-up.ts) use the same check."/reviewcomment still counts only when …") with: "The loop limit's count of maintainer comments, the open issues a finding can be tracked in and theAutopilot:replies the bot answers use the same check. A/reviewcomment still counts only when itsauthor_associationnames a writer, as the workflow'sif:below checks before the engine starts: a permission lookup there would let more people start a run that holds the model credentials. So a private organization member can't start a review that way; Commands says what they can do."## Commands): after "Only people with write access can.", add: "GitHub reports an organization member whose membership is private as a contributor, so their commands are ignored. To use them, make your membership public (on the organization's People page, change your membership from Private to Public), or have an admin add you to the repository as a collaborator."Autopilot:reply bullet): "the thread's last comment is anAutopilot:reply" becomes "the thread's last comment is anAutopilot:reply from someone with write access".node --test test/pull-request.test.ts; if ($LASTEXITCODE) { exit 1 }; if (-not (Select-String -Path README.md -SimpleMatch -Quiet 'change your membership from Private to Public')) { exit 1 }; if (Select-String -Path src/writers.ts -SimpleMatch -Quiet 'stays out of the prompts') { exit 1 }[ ] 9. Write the changelog entries
CHANGELOG.md(edit)## [Unreleased], at the end of the existing### Changedgroup, add these bullets as written:- The smoke eval fails when \evals/baseline-smoke.json` is missing after it has been committed, naming the commit that deleted it, so deleting the baseline can't turn the gate off; until the first baseline is committed it still passes with a notice. Its workflow checks out the full history for this (#93).`- The tests check that the smoke eval workflow uploads its outputs only after the secret scan passes and fails the job when the run or the check fails, and that \evals/run.ts` writes its output before it fails a run over `--max-cost` (#93).`- A private organization member's comment counts as the maintainer's for the loop limit, their open issues can track a finding, and the bot answers their \Autopilot:` replies: these use the engine's writer check, the author's association or else their permission on the repository, and a lookup that fails counts as no write access. The README says how such a member can use `/review`, whose gate still reads only the association (#89).`if (-not (Select-String -Path CHANGELOG.md -SimpleMatch -Quiet 'issues/93)')) { exit 1 }; if (-not (Select-String -Path CHANGELOG.md -SimpleMatch -Quiet 'issues/89)')) { exit 1 }Review checklist
--baseline <file>; the deleted-baseline path fails closed on a shallow checkout or a git error.if:losessteps.scan.outcome == 'success', the scan moves after the upload, or a step gainscontinue-on-error.main()inevals/run.tsends with thewriteRunOutputcall and nothing after it.Autopilot:comments in the pause; the pause step test intest/pause.test.tsstill sees only the two comment reads.readOpenIssuesstays within 20 lookups, keeps order, and the review step shares onewriterCheckcache./reviewgate (src/command.ts, the workflowif:) is unchanged.Follow-ups
src/rereview.ts(the earlier threads' replies that reach the re-review prompt, filtered bywriterAssociations) andsrc/outcomes.ts(a reply'swriterflag) still judge byauthor_associationalone, so a private organization member's reply is left out of the re-review and doesn't count toward a finding's outcome; switch them towriterCheckfromsrc/writers.ts.polewskm commentedon Oct 31, 2026
Follow-ups filed from this plan: #124 Writer checks for re-review replies and outcome replies