Skip to content

Give learned guidance rules only to reviews of the files they apply to - #68

Merged
polewskm merged 11 commits into
mainfrom
issue-36
Oct 31, 2026
Merged

polewskm merged 11 commits into
mainfrom
issue-36

Conversation

@polewskm

Copy link
Copy Markdown
Contributor

Each learned guidance bullet that came only from instruction files now carries their globs as a tag (*Applies to:* `**/*.cs`). The engine drops tagged bullets that match no changed file, and each part of a large diff gets only the bullets for its files, so a C# rule no longer reaches a TypeScript review. Untagged bullets still apply everywhere.
Training saves each instruction file's applyTo with its extracted rules and writes the tags through one formatter. node training/run.ts --retag rewrites the tags without calling a model, and it regenerated the committed guidance/ (87 bullets are now **/*.cs). A test fails when the committed guidance drifts from its sources.

Decisions:

  • Globs keep only their file pattern so they carry over to any repository: Identity's src/**/*.cs and tests/**/*.cs both become **/*.cs.
  • A bullet is tagged only when every source is a scoped instruction file. If any source is a decision record, a review, a doc, or an instruction file that applies to **, the bullet stays untagged and applies everywhere, as before. This keeps more rules in reviews rather than fewer.
  • I added the applyTo values to the 10 committed training/extracted/.../instructions/*.json files by hand, using Identity's current instruction frontmatter. A later extract run keeps them in sync without re-extracting.
  • I didn't run the paid training or the evals.

Closes #36

…tion-file sources apply to

Training records each instruction file's applyTo with its extracted rules and tags a guidance bullet learned only from instruction files with their globs, generalized to any repository (src/**/*.cs becomes **/*.cs). The engine drops tagged bullets that match no changed file, and each part of a large diff gets only the bullets for its files. node training/run.ts --retag rewrites the tags without a model.

Refs #36
Generated by node training/run.ts --retag: rules learned only from Identity's C# instruction files now apply to **/*.cs.

Refs #36
@polewskm
polewskm deployed to code-review October 31, 2026 04:16 — with GitHub Actions Active

@ncode-review ncode-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 NCode Review at df3325b: 7 new findings on the changed lines.

Comment thread src/review.ts Resolved
Comment thread training/run.ts Resolved
Comment thread src/gather.ts Resolved
Comment thread src/gather.ts Resolved
Comment thread src/gather.ts Resolved
Comment thread CHANGELOG.md Resolved
Comment thread test/engine.test.ts Resolved
@ncode-review

ncode-review Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

🤖 Review complete

Verdict: 💬 Comment — This PR records an instruction file's applyTo globs when rules are extracted, tags each learned guidance rule that comes only from instruction files with *Applies to:*, and filters guidance by those tags. The filtering happens per review in readGuidance and per diff part in reviewJobs. There's also a --retag path that rewrites the tags without calling a model. The design holds together: prompt assembly moves into a testable reviewJobs, and the bullet format is centralized in src/guidance.ts. The main gap is that the review() wiring now building prompts from those jobs has no test. The rest are nits.

8 reviewers run — 5 findings (run details)

Reviewer Model Findings Top
synthesis opus-5-5 5 🟠
architecture sonnet-5-5 0 🟢
completeness sonnet-5-5 0 🟢
conventions sonnet-5-5 0 🟢
correctness sonnet-5-5 1 🟡
hygiene haiku-5-5 1 🟡
intent haiku-5-5 0 🟢
rule-check sonnet-5-5 1 🟡
tests haiku-5-5 2 🟠

Skipped: api-compat, data-model, infrastructure, performance, security

Findings
  • 🟠 F1 src/review.ts:359: No test covers how review() builds each job's prompt
  • 🟡 F2 training/run.ts:279: generalGlob turns directory-wildcard globs such as **/charts/** into **, and no test pins this
  • 🟡 F3 src/review.ts:148: Rule-batch label uses the unscoped batch size
  • 🟡 F4 src/gather.ts:214: The bullet marker '- **' is repeated as a literal
  • 🟡 F5 CHANGELOG.md:16: Is Guidance learned from C# instruction files reaches reviews of other languages #36 the right issue for this changelog entry?

Trial: findings are posted as review threads and in the check run. $1.00 on Amazon Bedrock, 92s, at 2cd3885. (how to interact)

# Conflicts:
#	CHANGELOG.md
#	training/run.ts
…format in one module

review() builds its reviewer calls with reviewJobs and names its outcomes with outcomeNames, and extract() decides reuse with reuseExtraction, so tests cover them without a model: each part gets only its files' rules, an emptied rule batch doesn't run or report, and a changed or removed applyTo is saved without re-extracting. Also tests multi-glob tags, a topic with no bullets left, and retag refusing a malformed bullet.

formatBullet, parseBullet and bulletGlobs move to src/guidance.ts and share one pattern for the Applies to field, so training and the engine can't drift apart on the layout.
@polewskm

Copy link
Copy Markdown
Contributor Author

/review

@ncode-review ncode-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 NCode Review at 00418a4: 5 new findings on the changed lines.

Not on a changed line:

  • 🟡 F4 Nit: No test runs the --retag file loop training/run.ts:402. Tests cover retagGuidance, but not retag(), which loops over the folders, skips missing ones and writes files back. Fix: Optional: make retag take directories, then add a temp-folder test that asserts a stale tag is rewritten and an up-to-date file is left alone.

Comment thread src/review.ts Resolved
Comment thread src/context.ts Resolved
Comment thread src/guidance.ts Resolved
Comment thread test/engine.test.ts Resolved
Comment thread src/review.ts Resolved
@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: answering F4 (No test runs the --retag file loop) from #68 (review) and #68 (comment).

declined: nit; retag() is a thin loop over tested pieces. retagGuidance has its own tests, including the refusal paths, and the test that runs retagGuidance over every file in guidance/ fails when any file is out of date. Making retag take directories for a temp-folder test is not worth another round.

@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: Round 2 answered. No blocker or should-fix is left open; the remaining threads are nits, answered above. Recommend merging once you've resolved them.

@polewskm
polewskm marked this pull request as ready for review October 31, 2026 06:38
@polewskm
polewskm deployed to code-review October 31, 2026 06:38 — with GitHub Actions Active
@polewskm
polewskm enabled auto-merge October 31, 2026 06:38

@ncode-review ncode-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 NCode Review at 2cd3885: 3 new findings on the changed lines.
2 more were already posted on this pull request.

Comment thread src/review.ts Resolved
Comment thread training/run.ts Resolved
Comment thread src/review.ts Resolved
# Conflicts:
#	README.md
#	src/review.ts
#	test/engine.test.ts
#	training/run.ts
# Conflicts:
#	README.md
#	src/review.ts
@polewskm

Copy link
Copy Markdown
Contributor Author

Autopilot: Round 3 answered. The review-round limit (3) is reached; no blocker or should-fix showing a real defect is left open. The three new threads (a test for review()'s prompt wiring, a test pinning generalGlob on folder-only globs, and the rule-batch label) are declined above; any of them could be a follow-up issue. origin/main is merged in (b9042e2: README and src/review.ts conflicts with the run cost ceiling resolved, typecheck and 161 tests pass). Recommend merging once you've resolved the threads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Guidance learned from C# instruction files reaches reviews of other languages

1 participant