Skip to content

Acknowledge a review within seconds and always complete its check run - #74

Merged
polewskm merged 13 commits into
mainfrom
issue-20
Oct 31, 2026
Merged

polewskm merged 13 commits into
mainfrom
issue-20

Conversation

@polewskm

Copy link
Copy Markdown
Contributor

Within seconds, before setup-node and npm ci, the action now reacts 👀 to a /review comment and creates the "Code review" check run in_progress ("Review started at ; it usually takes a few minutes", linked to the run). It does this for automatic reviews too. The early steps use the review step's own decision (decideReview in src/command.ts, now shared with main.ts). The acknowledgement's app token is minted from the existing app-private-key input, used only by src/acknowledge.ts (plain node, no npm packages) and revoked before the review step starts. It is never in the review step's env, and test/acknowledge.test.ts checks the env blocks in action.yml. The publish step updates that check run (PATCH) rather than posting a second one. A final always() step completes it when nothing was published:

  • neutral "Review budget used up" with the reason when no provider is available; the job no longer fails
  • failure with the error and a link to the log
  • cancelled when the run was cancelled

No consumer workflow change is needed: no new input or permission.

Decisions

  • Plain node before setup-node. The acknowledgement runs with the runner's own node, which needs TS type stripping (Node ≥ 22.18; ubuntu-latest has 22.23). The tests pass on Node 22.23.3 here. If a runner's Node can't run it, those steps continue on error and the review runs as before.
  • Separate token, revoked early. The acknowledgement mints its own token (skip-token-revoke: true) and revokes it in a finally, so no app token is valid while the agents run. The publish and finish steps mint theirs after the review step, as before.
  • Bedrock check. To report "key refused", chooseProvider makes one 1-token Bedrock call before it relies on the key. Only HTTP 401/403 counts as refused; any other answer or a network error goes on with Bedrock as before.
  • Job result for "no provider". The job is green, with a ::warning::, and the check is neutral, so automation can carry on.
  • Failure without an acknowledgement. A failure after the run has decided which commit to review still posts a completed check, even if the acknowledgement didn't happen.
  • Placeholder summary comment skipped. I left out the optional placeholder summary comment (item 3 of the issue); the reaction and the check run cover it.
  • Pull request moved. If the pull request moves between the acknowledgement and the review, the review gets its check on the reviewed commit. The acknowledged check completes neutral, "Reviewed instead".

Manual checks that need a live run (from the issue's Acceptance)

  • /review from a writer gets 👀 and an in-progress "Code review" check within about ten seconds, in a run whose model call later fails.
  • A run that fails ends with the failure (and its reason) on the check run, not an in-progress check. Also check the "Review budget used up" neutral check when no provider is available.
  • When the review finishes, no second summary comment or second "Code review" check run appears.
  • A non-/review comment, a fork pull request, and a /review from someone without write access get nothing.
  • An automatic review (opened / ready for review) shows the in-progress check, and a ready-for-review on an already-reviewed commit shows nothing new.

Tests: npm run typecheck and npm test (111 pass). The e2e tests were not run.

Closes #20

An early step, before setup-node and npm ci, makes the review step's own decision and, for a run that will review, adds 👀 to the /review comment and creates the Code review check run in progress, with a token minted for it alone and revoked before the review step starts. The publish step completes that check run; a final step completes it with the reason when there's no review: neutral 'Review budget used up' when no provider is available, failure with the error, or cancelled.

Closes #20
@polewskm
polewskm deployed to code-review October 31, 2026 04:40 — with GitHub Actions Active
Comment thread src/acknowledge.ts Fixed
Comment thread src/github.ts Dismissed
Comment thread src/github.ts Dismissed
Comment thread src/main.ts Fixed

@ncode-review ncode-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 NCode Review at 932db29: 9 new findings on the changed lines.

Not on a changed line:

  • 🟡 F5 Nit: Step outcomes passed as a positional comma-separated string action.yml:104. CODE_REVIEW_STEP_OUTCOMES joins four outcomes with commas, and finish.ts reads them back by position. Nothing ties the order in action.yml to the destructuring in finish.ts. Fix: Pass one env var per step outcome, or a JSON object, so finish.ts reads each outcome by name.
  • 🟡 F9 Nit: GitHub request headers still built by hand outside github.ts src/main.ts:69. readClosedIssues in main.ts and lastReviewedHead in comments.ts still build the accept and authorization headers themselves, while the new code uses the shared helpers in github.ts. Fix: Send these requests through a shared helper in github.ts, or accept the duplication on purpose.

Comment thread src/main.ts Resolved
Comment thread src/finish.ts Resolved
Comment thread src/acknowledge.ts Resolved
Comment thread action.yml Resolved
Comment thread src/checks.ts Resolved
Comment thread src/checks.ts Resolved
Comment thread src/acknowledge.ts Resolved
Comment thread src/provider.ts Resolved
Comment thread src/checks.ts Resolved
@ncode-review

ncode-review Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

🤖 Review complete

Verdict: 💬 Comment — This PR adds an early acknowledgement step that reacts 👀 and opens an in-progress "Code review" check run before setup-node and npm ci, so the reviewer sees it within seconds. A finish step then always completes that check run: neutral when the budget is used up or the review is skipped, failure when it breaks, cancelled when the run is cancelled. The design holds together: shared code that needs no npm packages lives in checks.ts and github.ts, review decisions go through decideReview, and secrets are filtered out of status text. The main risks are that the acknowledgement token is not revoked if the ack script dies before it runs, and that provider.ts now depends on how check runs are worded. The rest is minor cleanup.

8 reviewers run — 8 findings (run details)

Reviewer Model Findings Top
synthesis opus-5-5 8 🟠
architecture sonnet-5-5 2 🟠
completeness sonnet-5-5 1 🟡
conventions sonnet-5-5 1 🟡
correctness sonnet-5-5 1 🟡
hygiene haiku-5-5 2 🟡
infrastructure haiku-5-5 0 🟢
rule-check sonnet-5-5 3 🟡
security sonnet-5-5 1 🟠

Skipped: api-compat, data-model, intent, performance, tests

Findings
  • 🟠 F1 action.yml:66: Acknowledgement app token is not revoked if the ack script never runs
  • 🟠 F2 src/provider.ts:3: Provider selection now depends on check-run wording
  • 🟡 F3 src/provider.ts:38: Bedrock pre-flight is a real model call, and only 401/403 count as refusals
  • 🟡 F4 src/github.ts:19: githubApi parses every non-204 success as JSON
  • 🟡 F5 src/main.ts:115: run.headSha is assigned twice
  • 🟡 F6 src/github.ts:36: fetchPullRequest rebuilds headers that githubApi and comments.ts already build
  • 🟡 F7 src/checks.ts:199: Check-run functions repeat the (api, repository) parameter pair
  • 🟡 F8 src/finish.ts:26: Calls nested inside other calls' arguments

Trial: findings are posted as review threads and in the check run. $0.77 on Amazon Bedrock, 92s, at 997bd51. (how to interact)

…mes by name

- Move post() and decide() into src/acknowledgement.ts as functions of their
  api, env and fetch, and test the token revocation, the head check and the
  step outputs.
- Move the review step's status record and the finish step's status into
  checks.ts (stoppedRecord, finishStatus) and test the secret fallbacks and
  the stopped=true output.
- Pass each step outcome to the finish step in its own variable.
- Share required() and take() through src/env.ts.
- Map no provider to the budget status in main.ts, so checks.ts no longer
  loads provider.ts and config.ts.
- Assert the exact app-token, publish and finish conditions in action.yml.
Comment thread src/main.ts Dismissed
@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: answering #74 (review) and the summary #74 (comment).

  • F5 (step outcomes as a positional string): fixed in 70ce898. The finish step gets CODE_REVIEW_SETUP_OUTCOME, CODE_REVIEW_INSTALL_OUTCOME, CODE_REVIEW_REVIEW_OUTCOME and CODE_REVIEW_PUBLISH_OUTCOME, read by name in stepOutcomesFrom, and a test asserts each variable comes from its step.
  • F9 (GitHub headers built by hand outside github.ts): declined for this PR. readClosedIssues in main.ts and lastReviewedHead in comments.ts aren't changed by it; like fetchPullRequest, they send an optional read-only token and treat a refusal as "skip" rather than throwing as githubApi does, so sharing one helper is a separate refactor. It could be a follow-up.
  • The main risk the summary names, the untested glue, is fixed in 70ce898; F1–F4, F6–F8, F10 and F11 are answered in their threads.

@polewskm

Copy link
Copy Markdown
Contributor Author

/review

@ncode-review ncode-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 NCode Review at 70ce898: 7 new findings on the changed lines.

Comment thread src/main.ts Resolved
Comment thread action.yml Resolved
Comment thread src/main.ts Resolved
Comment thread src/command.ts Resolved
Comment thread src/checks.ts Resolved
Comment thread src/command.ts Resolved
Comment thread test/acknowledge.test.ts Resolved
…nt without a pull request before the fork rule

stoppedBy in provider.ts maps no provider to a used-up budget that passes the job and any other error to a failure, and main's error handler uses it. decideReview now checks for a missing pull request before the skip rules, so such an event reads as having no pull request rather than as a fork.
@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: Round 2 answered. No blocker or should-fix is left open; the remaining threads are nits, answered above. Recommend merging once you've resolved them.

@polewskm
polewskm marked this pull request as ready for review October 31, 2026 06:44
@polewskm
polewskm deployed to code-review October 31, 2026 06:44 — with GitHub Actions Active
@polewskm
polewskm enabled auto-merge October 31, 2026 06:44

@ncode-review ncode-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 NCode Review at 997bd51: 8 new findings on the changed lines.

Comment thread action.yml Resolved
Comment thread src/provider.ts Resolved
Comment thread src/provider.ts Resolved
Comment thread src/github.ts Resolved
Comment thread src/main.ts Resolved
Comment thread src/github.ts Resolved
Comment thread src/checks.ts Resolved
Comment thread src/finish.ts Resolved
@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: merged origin/main into issue-20 (d70bcc7), plus one follow-up commit (e54c366).

  • src/publish.ts: main moved postThreads into src/comments.ts with an injectable api (Share the comment pager with postThreads and harden lastReviewedHead #59). That replaces this PR's local copy, which had only switched it to api. publish.ts now imports main's postThreads and calls postThreads(api, results, [token]) with this PR's githubApi(token). The check-run acknowledgement and completion are unchanged.
  • e54c366: main's src/results.ts now value-imports severityRank from src/review.ts, which loads @anthropic-ai/claude-agent-sdk. Through checks.ts → comments.ts → results.ts, that broke the test requiring the acknowledgement and finish scripts to load no npm package, because they run before npm ci. Severity and severityRank now live in a new import-free src/severity.ts. review.ts re-exports them, so evals/run.ts and other importers are unchanged.

npm run typecheck and npm test pass: 168 of 168.

# Conflicts:
#	CHANGELOG.md
#	src/results.ts
#	src/review.ts
@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: Round 3 answered: #74 (review) and the summary #74 (comment). The review-round limit (3) is reached. No verified blocker or should-fix is left open. F1 repeats round 2's F2 and still holds: the decide step loads the same modules first, and post revokes the token in its finally. F2 is about where the code lives, not a behavior defect. The remaining threads are nits or repeats, declined above, and a few could be follow-up issues. origin/main is merged in (e09963f, a CHANGELOG conflict resolved by keeping both entries), and npm run typecheck and npm test pass (207 of 207). Recommend merging once you've resolved the threads.

@polewskm
polewskm merged commit e178a33 into main Oct 31, 2026
5 checks passed
@polewskm
polewskm deleted the issue-20 branch October 31, 2026 16:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Acknowledge a /review comment immediately, so it doesn't look like nothing is happening

2 participants