Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/code-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:
environment: code-review
timeout-minutes: 30
concurrency:
group: code-review-${{ startsWith(github.event.comment.body, '/review explain') && format('explain-{0}', github.event.comment.id) || github.event.pull_request.number || github.event.issue.number }}
group: code-review-${{ (startsWith(github.event.comment.body, '/review explain') || startsWith(github.event.comment.body, '/review teach')) && format('command-{0}', github.event.comment.id) || github.event.pull_request.number || github.event.issue.number }}
cancel-in-progress: true
permissions:
id-token: write
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),

### Added

- `/review teach <rule>`, `/review teach F3 <rule>` and `/review teach org <rule>` turn a correction into lasting guidance: with no model call, the publish step opens an issue labelled `review-guidance`, in this repository or, for `org`, in NCodeGroup/code-review, quoting the rule as data with a link to the comment and, for `F3`, the finding's file, line and text, and replies with a link to it. It creates the label when it's missing and uses a second app token that can write only issues, so a review never asks for issues write; when the app lacks that permission the reply says so. `npm run train` copies the rules of `teach org` issues closed as completed into `training/feedback.md` before it generalizes. Writers only, top-level comments only, no forks, as for `/review` ([#45](https://github.com/NCodeGroup/code-review/issues/45)).
- `/review explain F3`, or `/review explain <link to a finding comment>`, with an optional question, answers in a top-level comment whether the finding still holds: one Sonnet agent with Read, Grep and Glob, locked down like the reviewers and capped at about $0.50, reads the latest commit again. An id means the newest review's finding; an id or link that isn't one gets a comment listing the newest review's ids. Writers only, no forks, as for `/review` ([#44](https://github.com/NCodeGroup/code-review/issues/44)).
- Evals measure precision: each posted blocker and should-fix that matches no expected concern is judged `real`, `not-real` or `unsure` by Opus reading the checkout, with verdicts cached in `evals/verdicts.json`. Each output also reports findings per case (mean, max) and the review's cost per case. `npm run eval:compare` compares two runs, marking recall differences of 2 or less as noise, and `npm run eval:confirm` adds confirmed verdicts to `cases.json` as expected or must-not-flag concerns ([#51](https://github.com/NCodeGroup/code-review/issues/51)).
- A finding whose fix is a mechanical edit of up to five lines, all of them added by the pull request, carries a GitHub suggestion block on a comment that covers those lines, so it applies with one click; any other fix stays in prose. The hidden marker on each inline comment also names its reviewers, and comments posted before still count as already posted ([#54](https://github.com/NCodeGroup/code-review/issues/54)).
Expand Down
29 changes: 23 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ The agents run on the Claude Agent SDK with no settings, hooks, MCP servers or C
and Glob, and a hook that refuses any path outside the pull request's checkout after resolving symlinks. Their
environment holds no secret: model calls go through a localhost proxy that adds the credentials. The app's private key
reaches only the step that mints its token, after the agents exit, and that token can write check runs on the one
repository under review. Results and the trace are scanned for secrets before they're written.
repository under review. A `/review teach` run also mints one that can write only issues, in the one repository its
issue opens in. Results and the trace are scanned for secrets before they're written.

Besides the reviewers, **rule checks** go through the rules the context stage picked for the pull request, twelve at a time, on Sonnet: each call checks its rules one by one against every part of the diff, searching the changed files for what the rule's check names. They report as `rule-check` and run on every review, outside the reviewer roster, so selection can't skip them. In the evals they caught about one more expected finding in 55, at about 70% more cost. `CODE_REVIEW_RULE_CHECKS=off` turns them off.

Expand All @@ -33,8 +34,8 @@ on:
permissions: {}
jobs:
review:
# A pull request when it opens or is marked ready, or a top-level `/review` or `/review explain` comment on one from
# someone with write access. The engine checks the comment again and skips anything else.
# A pull request when it opens or is marked ready, or a top-level `/review`, `/review explain` or `/review teach`
# comment on one from someone with write access. The engine checks the comment again and skips anything else.
if: >-
github.event.sender.type != 'Bot' && (
(github.event_name == 'pull_request_target' &&
Expand All @@ -47,7 +48,7 @@ jobs:
environment: code-review
timeout-minutes: 30
concurrency:
group: code-review-${{ startsWith(github.event.comment.body, '/review explain') && format('explain-{0}', github.event.comment.id) || github.event.pull_request.number || github.event.issue.number }}
group: code-review-${{ (startsWith(github.event.comment.body, '/review explain') || startsWith(github.event.comment.body, '/review teach')) && format('command-{0}', github.event.comment.id) || github.event.pull_request.number || github.event.issue.number }}
cancel-in-progress: true
permissions:
id-token: write
Expand Down Expand Up @@ -123,13 +124,29 @@ Comment on a pull request, as a top-level comment, not a reply in a review threa
| `/review` | Reviews the pull request's latest commit. Pushes start no review, so use this after changes. |
| `/review explain F3 [question]` | Reads the code again and answers about the newest review's F3 in a comment: "Re F3 (link): …", ending with "Still holds because …" or "On reflection this doesn't hold: …". The question is optional and may go on over several lines. |
| `/review explain <link> [question]` | The same for the inline finding comment the link points to, from any review of this pull request. Finding ids restart at F1 in every review, so use a link for an earlier review's finding. |
| `/review teach <rule>` | Opens an issue labelled `review-guidance` in this repository with the rule, to add to its `.github/code-review/guidance.md`, and replies with a link to it. The rule may go on over several lines. |
| `/review teach F3 <rule>` | The same for a rule that comes from the newest review's F3, such as "don't flag X because Y": the issue also quotes the finding with its file, line and link. A finding comment's link works in place of the id. |
| `/review teach org <rule>` | Opens the issue in NCodeGroup/code-review instead, for a rule every NCodeGroup repository's review follows. |

`/review explain` finds an id among the bot's inline finding comments of the newest review; a finding that went in a
review's body instead has no comment to point at. An id or link it can't find gets a short comment listing the newest
review's ids, with no model call. Otherwise one Sonnet agent with Read, Grep and Glob, locked down like the reviewers,
reads the pull request's latest commit, within a budget of about $0.50. The `if:` above already lets the command through,
since it checks only that the comment starts with `/review`. The `concurrency` group above gives each explain comment a
group of its own, so asking while a review is running doesn't cancel that review, and a review doesn't cancel an answer.
since it checks only that the comment starts with `/review`. The `concurrency` group above gives each explain or teach
comment a group of its own, so either command while a review is running doesn't cancel that review, and a review
doesn't cancel an answer.

`/review teach` makes no model call and writes no code. The review step words the issue: the rule quoted as data in a
fenced block, with its mentions defused, who taught it and a link to the comment, and for `teach F3` the finding's file,
line, text and link. The publish step, after everything else, opens it with a second app token that can write only
issues, in only that repository, creating the `review-guidance` label when it's missing, and replies with a short
comment linking it. An empty or overlong rule (over 2,000 characters), or a finding id or link it can't find, gets a
reply instead. A `teach org` issue in the public NCodeGroup/code-review names neither a private repository nor its
pull request. `/run-issues` or a person turns the issue into a pull request; for a `teach org` rule, closing the issue
as completed is enough, because `npm run train` copies the rules of closed `review-guidance` issues into
`training/feedback.md`, under "Taught with /review teach", before it generalizes. Closing one as not planned drops it.
This needs the app's Issues: Read and write permission and, for `teach org`, its installation on NCodeGroup/code-review;
without them the reply says the app lacks the permission and quotes the rule to add by hand.

To skip the automatic review on a pull request, label it `skip-review` before it opens or is marked ready.

Expand Down
27 changes: 22 additions & 5 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ inputs:
required: false
default: base
github-token:
description: A read-only token (contents and pull requests), used to read the pull request a /review comment is on and the finding a /review explain comment asks about, and to fetch the pull request's head.
description: A read-only token (contents and pull requests), used to read the pull request a /review comment is on and the finding a /review explain or /review teach comment names, and to fetch the pull request's head.
required: false
default: ${{ github.token }}
app-client-id:
Expand Down Expand Up @@ -56,8 +56,8 @@ runs:

# The model credentials stay in this step's process; the agents it starts get a proxy address and a placeholder.
# The app's private key is not in this step's environment.
# For a comment that isn't a /review or /review explain command from someone with write access, it stops here and the
# later steps skip.
# For a comment that isn't a /review, /review explain or /review teach command from someone with write access, it stops
# here and the later steps skip.
- name: Review
id: review
shell: bash
Expand All @@ -77,7 +77,7 @@ runs:
# comments.
- name: Get the app's token
id: app-token
if: steps.review.outputs.reviewed == 'true' || steps.review.outputs.answered == 'true'
if: steps.review.outputs.reviewed == 'true' || steps.review.outputs.answered == 'true' || steps.review.outputs.taught == 'true'
uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0
with:
client-id: ${{ inputs.app-client-id }}
Expand All @@ -86,12 +86,29 @@ runs:
permission-checks: write
permission-pull-requests: write

# For /review teach only: a second token, able to write only issues, in the one repository the guidance issue opens
# in (this one, or NCodeGroup/code-review for /review teach org). It is separate so a review never asks for issues
# write, and it may fail: when the app lacks issues write there, or isn't installed there, the publish step replies
# that it lacks the permission instead of opening the issue.
- name: Get the app's token for the guidance issue
id: issues-token
if: steps.review.outputs.taught == 'true'
continue-on-error: true
uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0
with:
client-id: ${{ inputs.app-client-id }}
private-key: ${{ inputs.app-private-key }}
owner: ${{ steps.review.outputs.guidance-owner }}
repositories: ${{ steps.review.outputs.guidance-repository }}
permission-issues: write

- name: Publish
if: steps.review.outputs.reviewed == 'true' || steps.review.outputs.answered == 'true'
if: steps.review.outputs.reviewed == 'true' || steps.review.outputs.answered == 'true' || steps.review.outputs.taught == 'true'
shell: bash
env:
CODE_REVIEW_OUT: ${{ runner.temp }}/code-review
CODE_REVIEW_APP_TOKEN: ${{ steps.app-token.outputs.token }}
CODE_REVIEW_ISSUES_TOKEN: ${{ steps.issues-token.outputs.token }}
run: node "$GITHUB_ACTION_PATH/src/publish.ts"

- name: Upload the trace
Expand Down
34 changes: 29 additions & 5 deletions src/command.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Pull request comment commands: `/review`, and `/review explain <finding>` with an optional question. Only a top-level
// comment from someone with write access counts: a reply inside a review thread would start a workflow from the pull
// request's own code.
// Pull request comment commands: `/review`, `/review explain <finding>` with an optional question, and
// `/review teach [org | <finding>] <rule>`. Only a top-level comment from someone with write access counts: a reply
// inside a review thread is a pull_request_review_comment event, which carries no issue, so it is never a command.

// Which finding `/review explain` asks about: an id such as F3, which means the newest review's F3; a link to one of the
// bot's inline finding comments; or something that is neither, which gets the list of valid ids.
Expand All @@ -9,7 +9,14 @@
| { kind: 'url'; host: string; repository: string; number: number; commentId: number }
| { kind: 'invalid'; text: string };

export type Command = { kind: 'review' } | { kind: 'explain'; ref: FindingRef; question: string };
// `/review teach`: a rule for this repository, optionally derived from a finding, or (`org`) one for every NCodeGroup
// repository. The rule is kept whole here; the teach step refuses an empty or overlong one.
export type TeachScope = 'repository' | 'org';

export type Command =
| { kind: 'review' }
| { kind: 'explain'; ref: FindingRef; question: string }
| { kind: 'teach'; scope: TeachScope; ref?: FindingRef; rule: string };

const writers = new Set(['OWNER', 'MEMBER', 'COLLABORATOR']);

Expand All @@ -36,11 +43,27 @@
return { kind: 'url', host: url.host.toLowerCase(), repository: where[1]!, number: Number(where[2]), commentId: Number(comment[1]) };
}

// The longest rule `/review teach` accepts; a longer one gets a reply asking for a shorter one, not a cut-off rule.
export const maxRuleChars = 2_000;

// `/review teach <rest>`: `org <rule>` is an NCodeGroup-wide rule; a first word that is a finding id or a link to a
// finding comment names the finding the rule comes from; anything else is all rule.
function parseTeach(rest: string): Extract<Command, { kind: 'teach' }> {
const org = /^org(?:\s+([\s\S]*))?$/.exec(rest);
if (org) return { kind: 'teach', scope: 'org', rule: (org[1] ?? '').trim() };
const first = /^(\S+)(?:\s+([\s\S]*))?$/.exec(rest);
const ref = first ? parseFindingRef(first[1]!) : undefined;
if (ref && ref.kind !== 'invalid') return { kind: 'teach', scope: 'repository', ref, rule: (first![2] ?? '').trim() };
return { kind: 'teach', scope: 'repository', rule: rest };
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: No test for a first word that is an invalid link, which falls back to rule text F4 · tests

When the first word is an invalid finding link, such as an http URL or a URL that isn't a comment link, parseTeach keeps the whole text as the rule. No test covers this fallback.

Consequence: A change in this behavior, for example a link silently becoming part of the rule or becoming an error, would go unnoticed.

Fix: Add a parseCommand('/review teach https://example.com/x a rule') assertion that the whole text is kept as the rule.

Evidence

The parseCommand tests cover F-ids, valid links and 'organizations', but not an invalid link as the first word.

Confidence 0.30.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Autopilot: fixed in #85 (f296477): parseCommand tests now cover an https://example.com link, an http: finding link and a pull request link with no comment anchor. Each keeps the whole text as the rule.

// The comment must be exactly the command, apart from surrounding whitespace, so a sentence that merely mentions

Check notice on line 60 in src/command.ts

View check run for this annotation

NCode Review / Code review

F4 Nit: No test for a first word that is an invalid link, which falls back to rule text

When the first word is an invalid finding link, such as an http URL or a URL that isn't a comment link, `parseTeach` keeps the whole text as the rule. No test covers this fallback. Consequence: A change in this behavior, for example a link silently becoming part of the rule or becoming an error, would go unnoticed. Fix: Add a `parseCommand('/review teach https://example.com/x a rule')` assertion that the whole text is kept as the rule.
// /review doesn't start a run. `/review explain <ref>` may go on with a question, on the same line or the next ones.
export function parseCommand(body: string): Command | undefined {
const text = body.trim();
if (text === '/review') return { kind: 'review' };
const teach = /^\/review[ \t]+teach(?:\s+([\s\S]*))?$/.exec(text);
if (teach) return parseTeach((teach[1] ?? '').trim());
const explain = /^\/review[ \t]+explain(?:\s+(\S+))?(?:\s+([\s\S]*))?$/.exec(text);
if (!explain) return undefined;
const ref = explain[1] ? parseFindingRef(explain[1]) : { kind: 'invalid' as const, text: '' };
Expand All @@ -49,7 +72,8 @@

export interface CommentEvent {
issue?: { number?: number; pull_request?: unknown };
comment?: { body?: string; author_association?: string };
comment?: { body?: string; author_association?: string; html_url?: string; user?: { login?: string } };
repository?: { private?: boolean };
}

export function commandFromEvent(event: CommentEvent): { command: Command; number: number } | undefined {
Expand Down
Loading
Loading