Route reviewer file assignments by applicability instead of PR intent alone #657

Activity

1. zzwong commented on Oct 8, 2026

   

   zzwong

   on Oct 8, 2026

   ContributorAuthor

   More actions

   Outcome receipt

   • Change: Default per-file applicability guidance for reviewer selection and assignments.
   • Hypothesis: Relevant runtime edits in mixed changes reach the appropriate specialist without requiring changed-line excerpts.
   • Baseline: Local selector trial, three observations per case; mixed-case target assignment 0/3, other cases 3/3.
   • Result: Guidance assigned each case target 3/3; nine subsequent full dry-run reviews detected and retained all nine targets with reviewer prompt frozen. No fresh full-review control; earlier results used another CR version and are descriptive only.
   • Status: PENDING for released ordinary-review outcome; local pre-release assignment gate passed.
   • Target assessment: Local gate met; general efficacy inconclusive.
   • Learning: Generic applicability guidance was sufficient for these familiar cases; excerpts added no observed assignment coverage. Broader mixed-case scope increased mean selector cost about 29%.
   • Rollout/rollback: Draft implementation only; revert default selection instructions to roll back.
   • Evidence: Aggregate results and validation are in the linked draft PR; raw captures remain local. No unsupported major/blocking or product-impact findings came from the tested specialist.
   • Pending owner/check: PR owner; validate three ordinary mixed-change reviews within one week after release, checking assigned relevant files, review cost, and unsupported impact claims.
2. zzwong commented on Oct 8, 2026

   

   zzwong

   on Oct 8, 2026

   ContributorAuthor

   More actions

   Outcome receipt — follow-up holdout pilot

   • Change: Candidate default selector applicability guidance in draft PR fix(pipeline): assign reviewers by per-file applicability #659.
   • Hypothesis: Useful review findings improve across consumers without unacceptable noise or cost.
   • Baseline: 18 local full dry-run reviews across six previously untuned PRs, three per PR; two supported issues with three issue-observations, no unsupported high-severity escalation, cost $0.612431.
   • Result: 18 paired candidate reviews found the same issues at the same frequency; one unsupported blocking-severity escalation, cost $0.670018 (+9.4%). Median time 84.192s to 77.121s, descriptive. Optional feedback observations four to one.
   • Status: VERIFIED for local pilot; release outcome remains PENDING.
   • Target assessment: Holdout quality gate missed; no reproducible finding gain. Prior familiar-case assignment result remains valid.
   • Learning: Assignment improvement in a known mixed-change case did not translate to a general quality win in this small sample. Model severity variance remains consequential even with correct underlying findings.
   • Rollout/rollback: PR remains draft and unmerged; no installation/configuration changed. Revert default instructions if later deployed and found harmful.
   • Evidence: Aggregate results and limits added to PR fix(pipeline): assign reviewers by per-file applicability #659. All 36 runs used fresh execution; selector context and reviewer prompt hashes matched. Raw source and captures stay local.
   • Limits: Six historical PRs, retained prior discussion, no complete defect inventory or certified clean controls, source-checked but unblinded adjudication. No universal efficacy claim.
   • Pending owner/check: PR owner; reassess adoption before marking ready or releasing, using narrower scope or additional independent evidence.
3. zzwong commented on Oct 8, 2026

   

   zzwong

   on Oct 8, 2026

   ContributorAuthor

   More actions

   Outcome receipt — Sol 6.1 medium repeat

   • Change: proposed default per-file applicability guidance in fix(pipeline): assign reviewers by per-file applicability #659.
   • Hypothesis: a reproducible supported-finding gain without excessive noise or cost.
   • Baseline: 18 old-default local dry-run reviews across six pinned historical PRs, three repetitions each, Sol 6.1 medium reviewers. One supported issue observation; median 48.154s; accepted API-equivalent cost $2.868670.
   • Result: 18 guidance reviews, zero supported issue observations; median 52.862s; accepted cost $2.981092. No reproducible gain or loss. Default inflated one supported minor concern to blocking; guidance had none. The previously observed coverage-repair concern was missed by both arms.
   • Status: VERIFIED for the local experiment; production outcome untested.
   • Target assessment: missed preregistered quality gate, as in the Luna xhigh pilot.
   • Learning: Sol medium does not establish a general selector benefit. These trials do not establish the best production reviewer model/effort.
   • Rollout/rollback: no rollout; fix(pipeline): assign reviewers by per-file applicability #659 stays draft. Recommend narrower adoption around the demonstrated assignment gap before a global-default merge.
   • Evidence: aggregate results and controls added to fix(pipeline): assign reviewers by per-file applicability #659. Raw captures, source adjudication, manifests and per-PR timings remain on the local machine.

   Attempt accounting: 36 accepted reviews from 37 attempts. One reviewer startup returned no structured output with zero reviewer tokens and was replaced with identical inputs; no semantic miss excluded. Captured cost was $5.861972 including $0.012210 for that failed attempt. Accepted-review guidance cost increased 3.9%; pooled median review time increased 9.8%. Frozen saved-rate API-equivalent estimates are not cash spend or current pricing claims.

   Limitations: small selected historical sample, retained discussions, unblinded grading, natural reviewer cohorts, no complete gold inventory. Absence of findings does not prove clean code. Reusing the same cases controls the model comparison but is not a new holdout.
4. zzwong

   added

   bugSomething isn't workingSomething isn't working

   area:pipelineEnd-to-end review pipelineEnd-to-end review pipeline

   on Oct 16, 2026