Follow-ups from the 2026-10-31 autopilot run #84

Follow-ups that review rounds on the 2026-10-31 autopilot run's pull requests raised and declined as out of scope (or as nits past the three-round limit). Each one links the thread where it came up. None is a known defect in what landed; they are candidates to plan. Split any of them into its own issue when you pick it up.

Behavior

• Re-reviews and security: run security on a re-review when security-sensitive files changed. A re-review of a sensitive: path still uses the standard tier, not Opus (Re-review only what changed since the last review, and say when it's done #77 F2, Read per-repository settings from .github/code-review/ on the default branch #69 merge note).
• Verification: demote a refuted security finding or blocker to a question instead of dropping it, and require a refutation to cite code outside the changed lines (Add an opt-in verification stage that proves or refutes each blocker and should-fix #72 F1). Decide whether verification should also stop at the run ceiling (Enforce a per-run cost ceiling as a runaway guard #61 merge note).
• Suggestion blocks: limit them to trusted authors, or label them as generated (Post a suggestion block for a mechanical fix of a few lines #63 F1/F2).
• /review from private org members is still blocked by the workflow if: on author_association. The engine now checks the permissions API (Show a private organization member's description and closed issues to the reviewers #67). Close the loop on the bot's own threads, and pause after three unanswered reviews #82's pause, issue linking and Autopilot reply counting still go by association alone (Close the loop on the bot's own threads, and pause after three unanswered reviews #82 merge note).
• Template adoption detection: count a PR as an adoption or update only when _commit changes, or the files match the template's tag (Leave findings on template-owned files to the template's repository #70 F1). Use one template-owned list: the light review keeps its own, separate from src/template.ts (Read per-repository settings from .github/code-review/ on the default branch #69 merge note).
• Re-review thread reader: response.json() sits outside the try/catch, so a non-JSON body or a dropped connection fails the run (Re-review only what changed since the last review, and say when it's done #77 F3).
• Acknowledgement token: revoke it in an always() step. Treat Bedrock throttling or quota errors as a refusal (Acknowledge a review within seconds and always complete its check run #74).
• Smoke eval: while no baseline is committed, the gate passes with a notice. Decide whether that should fail instead (Run a cost-capped smoke eval when a prompt, charter or model changes #80 F2).
• closed runs still get the code-review environment and id-token: write. A second job without them would be tighter (Record each finding's outcome when a pull request closes #81 F2).
• IsPackable set in Directory.Build.props/.targets isn't read for the api-compat floor (Selection floor: add api-compat for public API changes and data-model for serialized contracts #66 F2).
• Settings: a minimum tier for the security reviewer, and a warning for very broad skip: globs (Read per-repository settings from .github/code-review/ on the default branch #69 F9).
• Light review and .txt: every .txt file counts as documentation unless its name is excluded. Count .txt only in doc/, docs/ or the repository root, or keep one shared exclusion list that the README and tests read (Answer late review findings on #62, #78 and #79 #85, Answer late review findings on #62, #78 and #79 #85 (comment)).
• Validate CODE_REVIEW_SHADOW names against the reviewer list (Shadow mode for a new reviewer or prompt #76 F2).
• Log failed write-permission lookups (Show a private organization member's description and closed issues to the reviewers #67 F3).
• Strip control characters from file names in logs (Never post a finding that says a changed file isn't changed #71 F5).

Structure

• One GitHub REST client for every read, with a timeout and a configurable host. Today githubReader, tokenApi, githubApi and the outcomes helper overlap (/review explain <finding>: answer whether a finding still holds #75 F1, Acknowledge a review within seconds and always complete its check run #74 F9, Record each finding's outcome when a pull request closes #81 F11, Monthly report of precision per reviewer and repository, and cost per fixed finding #83 F6).
• One GraphQL thread reader shared by re-reviews (src/rereview.ts) and outcomes (src/outcomes.ts) (Record each finding's outcome when a pull request closes #81).
• Shared escaping of mentions and links in every bot comment (/review explain <finding>: answer whether a finding still holds #75 F3).
• Move the C# member scan out of gather.ts into its own module (Selection floor: add api-compat for public API changes and data-model for serialized contracts #66 F2/F5). Move stoppedBy out of provider.ts (Acknowledge a review within seconds and always complete its check run #74 F2). Move comment parsing into threads.ts (Re-review only what changed since the last review, and say when it's done #77 F1).
• Use a YAML library for .github/code-review/config.yml, and one shared path-item check (Read per-repository settings from .github/code-review/ on the default branch #69).
• One shared type for the eval output file, used by its writer and its reader (Evals: measure precision and findings per pull request, and compare two runs #73 F1).

Tests

• review() building each part's prompt, and project reviewers through review() (Give learned guidance rules only to reviews of the files they apply to #68, Read per-repository settings from .github/code-review/ on the default branch #69 F1).
• The Bedrock fallback on selection and context calls (Retry on Bedrock when the API credits run out in the middle of a review #64 F1/F2).
• The smoke eval: upload waits for the secret scan, workflow gating, and main() writing its output before it fails a capped run (Run a cost-capped smoke eval when a prompt, charter or model changes #80).
• Judge spend stays out of the review cost (Evals: measure precision and findings per pull request, and compare two runs #73 F2). A two-part test for the changed-file marking (Never post a finding that says a changed file isn't changed #71). A direct test of answer() (/review explain <finding>: answer whether a finding still holds #75).

Remaining from closed or partly done issues

• Release: require the changelog section for the version, and say why Dependabot didn't propose the bumps #33: test the Dependabot pre-release theory (publish a release that isn't a pre-release, or run gh release edit v0.3.1 --prerelease=false). Auto-merge for consumer bumps belongs in project-template.
• Add a verification stage that proves or refutes each blocker and should-fix #41: the verification eval (2 runs each, on and off). Turn verification on by default only if it shows a gain.
• Record each finding's outcome when a pull request closes #47: add closed to project-template's caller and the 12 consumers, with the pin bump to the first release that contains Record each finding's outcome when a pull request closes #81.
• Run a cost-capped smoke eval when a prompt, charter or model changes #52: the first paid smoke-eval run, then commit evals/baseline-smoke.json.
• /review teach: turn a correction into lasting guidance #45: give the NCode Review app Issues: read and write, and include NCodeGroup/code-review in its installation.
• Monthly report of precision per reviewer and repository, and cost per fixed finding #48: dispatch the monthly report once some PRs have closed with outcomes, and check it against a hand count.
• Dismiss the CodeQL false positives declined in their threads (src/writers.ts, src/outcomes.ts, the acknowledgement script, and /review explain <finding>: answer whether a finding still holds #75's three).