Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
a589e4c
Read per-repository settings from .github/code-review/ on the default…
polewskm Oct 31, 2026
02aa579
Merge remote-tracking branch 'origin/main' into issue-43
polewskm Oct 31, 2026
dc5a450
Answer the review of the repository settings
polewskm Oct 31, 2026
ae0c55f
Merge remote-tracking branch 'origin/main' into issue-43
polewskm Oct 31, 2026
45c6c29
Test guidance.md's applyTo: and the config path-item checks, and docu…
polewskm Oct 31, 2026
b99ec44
Merge remote-tracking branch 'origin/main' into issue-43
polewskm Oct 31, 2026
a0d8543
Merge remote-tracking branch 'origin/main' into issue-43
polewskm Oct 31, 2026
f92dc0e
Give the settings tests' charters the verification prompt main added
polewskm Oct 31, 2026
1a656ff
Merge remote-tracking branch 'origin/main' into issue-43
polewskm Oct 31, 2026
0b94102
Deep reviews for sensitive paths and a light review for documentation…
polewskm Oct 31, 2026
29b8917
Keep scripts and git hooks out of a light review
polewskm Oct 31, 2026
0879cc8
Test a deep verification call that throws, and describe Results.depth…
polewskm Oct 31, 2026
feaefc5
Merge pull request #78 from NCodeGroup/issue-42
polewskm Oct 31, 2026
9b9d410
Merge remote-tracking branch 'origin/main' into issue-43
polewskm Oct 31, 2026
7078ba1
Adapt the deep-review and Bedrock fallback tests to each other
polewskm Oct 31, 2026
9b11132
Merge remote-tracking branch 'origin/main' into issue-43
polewskm Oct 31, 2026
f6a60a2
Merge remote-tracking branch 'origin/main' into issue-43
polewskm Oct 31, 2026
27a2ae3
Give the settings and depth tests' fixtures what main's re-review and…
polewskm Oct 31, 2026
eb39f13
Merge remote-tracking branch 'origin/main' into issue-43
polewskm Oct 31, 2026
35ac1a9
Move the review depth labels and the credits reason to their own modu…
polewskm Oct 31, 2026
beaa856
Merge remote-tracking branch 'origin/main' into issue-43
polewskm Oct 31, 2026
925328c
Merge remote-tracking branch 'origin/main' into issue-43
polewskm Oct 31, 2026
61a3147
Set authorCanWrite in the depth and settings tests' pull requests
polewskm Oct 31, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),

### Added

- Per-repository settings in `.github/code-review/`, read from the default branch's checkout only, so a pull request that edits them is reviewed with the default branch's copy: `config.yml` (`sensitive:`, `skip:`, `policies:`, `tiers:`), `guidance.md` rules for the context stage, and `reviewers/*.md` project reviewers that join the roster as `project:<name>`. Files `skip:` leaves out are listed in the check run as skipped by config, and a pull request whose every file it skips runs no model; policies reach every reviewer's conventions within a budget of their own, and a setting the engine can't use, including a `tiers:` name that matches no reviewer, is a warning in the check run, never a failed review ([#43](https://github.com/NCodeGroup/code-review/issues/43)).
- Shadow mode for a new reviewer or prompt: a charter with `shadow: true` in its frontmatter, or named in `CODE_REVIEW_SHADOW`, runs as usual, but its findings go only to the trace and a collapsed "Shadow findings (not posted)" section of the check run, never to synthesis, the threads, the summary or the verdict. A `prompts/<name>.shadow.md` runs beside the live prompt on the share of reviews `CODE_REVIEW_SHADOW_RATE` sets (0 by default), sampled by pull request and head, with both outputs in the trace. The evals score shadow reviewers apart from the live ones ([#53](https://github.com/NCodeGroup/code-review/issues/53)).
- `/review teach <rule>`, `/review teach F3 <rule>` and `/review teach org <rule>` turn a correction into lasting guidance: with no model call, the publish step opens an issue labelled `review-guidance`, in this repository or, for `org`, in NCodeGroup/code-review, quoting the rule as data with a link to the comment and, for `F3`, the finding's file, line and text, and replies with a link to it. It creates the label when it's missing and uses a second app token that can write only issues, so a review never asks for issues write; when the app lacks that permission the reply says so. `npm run train` copies the rules of `teach org` issues closed as completed into `training/feedback.md` before it generalizes. Writers only, top-level comments only, no forks, as for `/review` ([#45](https://github.com/NCodeGroup/code-review/issues/45)).
- `/review explain F3`, or `/review explain <link to a finding comment>`, with an optional question, answers in a top-level comment whether the finding still holds: one Sonnet agent with Read, Grep and Glob, locked down like the reviewers and capped at about $0.50, reads the latest commit again. An id means the newest review's finding; an id or link that isn't one gets a comment listing the newest review's ids. Writers only, no forks, as for `/review` ([#44](https://github.com/NCodeGroup/code-review/issues/44)).
Expand All @@ -17,6 +18,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
- A finding whose fix is a mechanical edit of up to five lines, all of them added by the pull request, carries a GitHub suggestion block on a comment that covers those lines, so it applies with one click; any other fix stays in prose. The hidden marker on each inline comment also names its reviewers, and comments posted before still count as already posted ([#54](https://github.com/NCodeGroup/code-review/issues/54)).
- A per-review cost ceiling guards against a runaway review: $8 plus $4 for each part of the diff, overridable with `CODE_REVIEW_MAX_RUN_USD`. Reviewer calls not started once it is reached are skipped and shown as `skipped: run budget` in the check run, synthesis still runs on what was found, and the summary warns that the review is incomplete ([#50](https://github.com/NCodeGroup/code-review/issues/50)).
- A verification stage, off by default: with `CODE_REVIEW_VERIFICATION=on`, a Sonnet call with the checkout proves or refutes each merged blocker and should-fix, a file's findings at a time at up to $0.50 a call. Confirmed findings get the code path in their evidence, refuted ones are dropped and listed with the reason under "Refuted by verification" in the check run and in the trace, and ones it can't settle become questions one severity lower. The summary names the refuted findings, and verification turns a request for changes with nothing serious left into a comment, never an approval. Nits aren't verified, and the calls show as `verification` in the cost table. It stays off until the eval shows a gain, and the eval report now counts posted blockers and should-fixes and refuted findings ([#41](https://github.com/NCodeGroup/code-review/issues/41)).
- Deep and light reviews. A pull request that changes a file `sensitive:` in `.github/code-review/config.yml` matches gets `correctness` and `security` on Opus for the parts of the diff with it, `security` added if selection didn't choose it, and verification on Opus for the findings in those files; the summary and check run say "Deep review (sensitive paths)". A pull request that changes only documentation, the changelog, or a template update's template-owned files, with no code, script or git hook among them, gets `hygiene` alone on Haiku, with no context, selection, synthesis or verification call, its findings posted as reported, and says "Light review (documentation only)". `CODE_REVIEW_TIERS` still overrides any reviewer's model, and the check run's reviewer table has a Model column that lists each model a reviewer ran on ([#42](https://github.com/NCodeGroup/code-review/issues/42)).

### Changed

Expand Down
81 changes: 79 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,14 @@ A rule in the learned guidance (`guidance/`) that came only from instruction fil

Each model call has its own cost cap, and a whole review has a **cost ceiling** that guards against a runaway review only: $8 plus $4 for each part of the diff, so $12 for most pull requests and $32 for one reviewed in six parts, far above the $0.60 to $2.10 a normal review costs and the $6 to $7 of a 2 MB pull request. Once the calls so far reach it, the reviewer and rule-check calls that haven't started are skipped, each such reviewer shows `skipped: run budget` in the check run, synthesis, and verification when it's on, still run on what was found, and the summary warns that the review is incomplete and doesn't approve. Shadow calls count toward it too, but the shadow-prompt twins start after every live reviewer call, and a skipped shadow reviewer doesn't make the review incomplete. To change it, set `CODE_REVIEW_MAX_RUN_USD` to a positive number of dollars in the calling job's `env`; any other value warns and keeps the default.

**Verification** is off by default. `CODE_REVIEW_VERIFICATION=on` adds a stage after synthesis: each blocker and should-fix it merged goes to a Sonnet call with Read, Grep and Glob in the checkout, behind the same path guard as the reviewers, with its diff hunk and its reviewers' charters, a file's findings per call at up to $0.50 each. The call proves or refutes the finding with a concrete code path. A confirmed finding stays, with the path added to its evidence; a refuted one is dropped and listed in the check run under "Refuted by verification", with the reason; one it can neither prove nor refute stays as a question, one severity lower. Nits aren't verified. The summary, which synthesis wrote before verification, gets a sentence naming the findings verification refuted and counting the ones it made questions. Verification never makes a review an approval: a request for changes with no blocker or should-fix left becomes a comment, since a verifier misled by the change's own code can refute a real finding. A [re-review](#re-reviews)'s findings go through it too. The calls show as `verification` in the check run's cost table and the trace records each one. It stays off until an eval shows it catches as much at fewer blockers and should-fixes, for what it costs ([#41](https://github.com/NCodeGroup/code-review/issues/41)).
**Verification** is off by default. `CODE_REVIEW_VERIFICATION=on` adds a stage after synthesis: each blocker and should-fix it merged goes to a Sonnet call, or an Opus call for a deep review's sensitive files, with Read, Grep and Glob in the checkout, behind the same path guard as the reviewers, with its diff hunk and its reviewers' charters, a file's findings per call at up to $0.50 each ($1 on Opus). The call proves or refutes the finding with a concrete code path. A confirmed finding stays, with the path added to its evidence; a refuted one is dropped and listed in the check run under "Refuted by verification", with the reason; one it can neither prove nor refute stays as a question, one severity lower. Nits aren't verified. The summary, which synthesis wrote before verification, gets a sentence naming the findings verification refuted and counting the ones it made questions. Verification never makes a review an approval: a request for changes with no blocker or should-fix left becomes a comment, since a verifier misled by the change's own code can refute a real finding. A [re-review](#re-reviews)'s findings go through it too. The calls show as `verification` in the check run's cost table and the trace records each one. It stays off until an eval shows it catches as much at fewer blockers and should-fixes, for what it costs ([#41](https://github.com/NCodeGroup/code-review/issues/41)).

Most pull requests get the **standard** review above. Two kinds get another ([#42](https://github.com/NCodeGroup/code-review/issues/42)), named at the start of the summary and in the check run:

- **Deep review (sensitive paths).** When a changed file that isn't generated matches a `sensitive:` glob in the repository's [settings](#repository-settings), the parts of the diff with such a file get `correctness` and `security`, added if selection didn't choose them, on Opus instead of Sonnet, and verification, when it's on, checks the findings in those files on Opus. The other parts of a large diff keep their usual models. In the evals, Opus reviewers caught four times what the all-Haiku baseline did, at about twice Sonnet's cost, so Opus is kept to the code a repository marks sensitive. The check run's Model column shows each reviewer's models, both when its parts differ. Documentation in a sensitive folder makes the review deep too, not light.
- **Light review (documentation only).** When every changed file is documentation (a `.md`, `.txt` or `.rst` file, or anything in a `doc/` or `docs/` folder, `CHANGELOG.md` among them, but not a `PublicAPI.*.txt` listing), or the pull request is a template update, changing `.template-version` or `.copier-answers.yml` and otherwise only files the project template owns or documentation, and no changed file is code, a script or a git hook (a `.ps1` or `.sh` file in `docs/` or in the template's `.template/` folder included), `hygiene` alone reviews it, on Haiku, and its findings are posted as it reported them, except one that says a changed file isn't changed, which no review posts: no context, selection, rule-check, synthesis or verification call. It costs a few cents instead of $0.60 to $2.10. A small code change still gets the standard review. Agent instructions such as `CLAUDE.md` and `.github/skills/` are Markdown, so they count as documentation; a repository that wants them reviewed closely lists them in `sensitive:`.

`CODE_REVIEW_TIERS` overrides any reviewer's model in both, as it does in a standard review: `correctness=sonnet` keeps a deep review's `correctness` on Sonnet.

### Shadow reviewers and prompts

Expand Down Expand Up @@ -255,7 +262,8 @@ of review converge:
that differ from that diff at the last reviewed head, which the summary comment records, and what the author took
out since. Merging the default branch in or rebasing onto it changes nothing by itself, and nothing from the
default branch is reviewed;
- only `correctness` reviews them, on Sonnet, with the earlier findings and the replies in their threads (from people
- only `correctness` reviews them, on Sonnet unless [`tiers:`](#repository-settings) or `CODE_REVIEW_TIERS` names
another model, never deep or light, with the earlier findings and the replies in their threads (from people
with write access). It checks each unresolved finding against the code now (fixed, still open, or answered), and it
doesn't raise again what was fixed or answered with a reason, or ask to undo what an earlier finding asked for;
- it posts a new finding only if it is a blocker or should-fix on a line changed since the last review, never a nit;
Expand All @@ -272,6 +280,75 @@ A `/review` of the commit already reviewed, or a last reviewed head that can't b
gets a full review instead. A review whose reviewers all failed doesn't count as the last review: its summary records
the head reviewed before it, so the next re-review still covers what it missed.

## Repository settings

A repository keeps its own review settings in `.github/code-review/`, which the project template never renders, so a
template update never touches it. The engine reads the folder from the `base` checkout of the default branch only,
never from the pull request, so a pull request can't soften its own review: one that edits these files is reviewed
with the copy already on the default branch, and the edit itself is reviewed like any other change.

- `config.yml`, every key optional:
- `sensitive:` path globs for a [deep review](#how-a-review-runs): a pull request that changes a file they match
gets `correctness` and `security` on Opus for the parts of the diff with it, and verification on Opus for the
findings in it. None by default.
- `skip:` path globs that are never reviewed, such as vendored code. The check run lists each changed file they
skip as "skipped by config", so nothing is dropped silently. When they skip every changed file, no model runs and
the summary says nothing was reviewed.
- `policies:` files, by their path from the repository root, that every reviewer gets as conventions, after the
conventions from `AGENTS.md`, with their path and within a budget of 40,000 characters of their own. A path that
is absolute, contains `..` or a backslash, or resolves outside the checkout through a symlink is left out.
- `tiers:` each reviewer's model, `haiku`, `sonnet` or `opus`, over the engine's defaults. A project reviewer runs
on `haiku`, like the engine's `hygiene` and `tests`, unless `tiers:` names it. A name that matches no reviewer on
the roster, engine or project, is a warning in the check run.
- `guidance.md`: the project's own review rules, in the format of `.github/instructions/*.instructions.md`. They join
the rules the context stage picks from, for every file unless an `applyTo:` frontmatter narrows them.
- `reviewers/*.md`: the project's own reviewers, in the format of [`reviewers/*.md`](reviewers), named with lowercase
letters, digits and hyphens. Each joins the roster as `project:<name>`, beside the engine's reviewers, never
replacing one; selection may add it, and a `paths:` frontmatter, a list of globs or one comma-separated string,
adds it to every pull request that changes a matching file. `shadow: true` in its frontmatter runs it in shadow, as
for the engine's charters ([Shadow reviewers and prompts](#shadow-reviewers-and-prompts)). Frontmatter with any
other key, a bad glob or a `shadow:` that isn't `true` or `false` is a warning, and the reviewer loads without its
paths. At most 10 are loaded.

`config.yml` is a small subset of YAML: top-level keys whose values are lists or, for `tiers:`, a mapping, in block or
`[a, b]` style, with comments and quoted strings. Quote a glob that starts with `*`, since YAML reads an unquoted one as
an alias. A setting the engine can't use, such as an unknown key, a missing policy or anything outside the subset,
gives a warning in the check run, and the review runs with the defaults for that file; it never fails the run. Globs
work like `applyTo:`: `**` spans folders, `*` and `?` stay within one.

For example, NCodeGroup/Identity could mark its JOSE, JWT, JWK and secrets projects sensitive, which about 1 in 6 of
its pull requests touch; adding its authentication, client and server projects would make about 1 in 3 deep, at about
$1.80 to $2.80 a review. Which set Identity uses is the maintainer's decision, and no repository is configured yet:

```yaml
# .github/code-review/config.yml
sensitive:
- "src/NCode.Jose/**"
- "src/NCode.Identity.Jwt/**"
- "src/NCode.Identity.Jwk/**"
- "src/NCode.Identity.Secrets/**"
skip:
- "src/**/Vendored/**"
policies:
- docs/api-compatibility.md
tiers:
api-compat: sonnet
project:oauth-conformance: sonnet
```

```markdown
<!-- .github/code-review/reviewers/oauth-conformance.md -->
---
paths: ["src/**/OAuth/**", "src/**/Oidc/**", "src/**/Jose/**"]
---
# oauth-conformance

Does this change still conform to the OAuth 2.0, OpenID Connect and JOSE specifications it implements?
```

Every reviewer, `api-compat` among them, then gets `docs/api-compatibility.md` in its conventions, and the trace's
`settings` lists the policies, project reviewers, sensitive globs and tiers the review used.

## Opting a repository in

The review's two keys are organization secrets, stored once and visible only to the repositories that opted in:
Expand Down
Loading
Loading