Repository navigation
Description
Description
Planning backlog under the alpha specification. The owner's October 19 merge-mission invocation authorizes scoped offline implementation and guarded merges. The retired project remains frozen; credentials, App installation and live activation retain separate authority and evidence gates.
Measure whether the specified dual-provider alpha improves actionable reviews, then dogfood it solely on this repository. Depend on eligibility, adapters/isolation, protocol and durable publication work. Preparing frozen fixtures is offline work; live model evaluation/installation needs passing gates and explicit owner authority.
Implementation prerequisites: Prove public-target subscription support and native App eligibility, Build both subscription adapters and the isolated source reader, Implement adversarial rounds and typed per-PR configuration, Build the durable alpha controller and scoped review publisher.
Why
Extra sessions can improve results without the second provider adding value. The reviewer must also handle candidate changes to its own prompts/auth/runtime without executing them against credentials.
Goal
A reproducible comparison with visible limits, followed by a small owner-only pilot using a separately promoted trusted release and independent human merge decisions.
Trigger decision — October 21, 2026
Franz chose @sfl review plus automatic PR-open/head-push review on October 21, 2026 at 01:19 EDT, via Discord relayed by the Chief. Implement both offline with fake transports and normal review/merge gates. Discovery stays polling-based; no webhook, machine user, live inference, subscription use, deployment or paid usage is authorized by this decision. Native-picker requests are no longer required. PR #222 records the failed native experiment; #221 is closed unmerged.
Acceptance Criteria
-
Freeze at least 20 held-out PR cases, including at least 10 clean controls, and label evidence before tuning. Define actionable/material defects and matching rules; preserve disagreements in ground truth. Separate tuning and evaluation cases.
-
Compare adversarial, independent-pair, Codex-only, Claude-only and single-provider self-critique with the same session budget as adversarial. Hold snapshot/context/tool capabilities fixed, disclose remaining asymmetry and pin versions. Specify run order/repeats and resource caps before live runs.
-
Report precision/recall numerators/denominators, clean-control failures, corroboration/dispute/concession outcomes, materially supported defects, p50/p95 latency and queue time, errors/quota waits and available usage. Explain missing usage and uncertainty; do not infer subscription capacity from price.
-
Assess provisional 90% actionable precision, 70% material-defect recall and 90% clean-control goals with sample-size limits. Report whether dual beats independent pair, each single provider and the equal-session control. Inconclusive/poor results stop rollout; do not silently change Franz's dual default.
-
Before alpha exists, use manual review. Later use a separately promoted pinned trusted release/policy/reader/publisher. Candidate changes to reviewer source/prompts/auth/allowlist/runtime stay inert. Human merge/checks remain independent and merges never automatically promote a build.
-
After eligibility, billing, isolation, recovery and activation gates pass, demonstrate one clean and one known-defect PR plus dispute/late finding, quota/provider failure, cancellation and moved-revision behavior. Alpha accepts only Franz and repository ID
1169772257; no broader organization rollout. Authorized automatic PR-open/head-push triggers must pass the same snapshot, generation, safety and activation gates as manual requests. -
Run a bounded two-week pilot, report quota pressure/interactive subscription impact and maintenance/recovery observations. Provisional targets are p50 under 10 minutes and p95 under 20 minutes when quota is available and the owner host is online; show all deferred cases separately. Keep App/worker stop and rollback steps verified.
-
Agents may create and revise ground-truth labels and actionability judgments from independent evidence. Every label records its agent/provider/model/session, time, rationale and evidence. Defects cite a concrete later fix/revert commit or PR, or linked bug/regression issue, plus affected candidate lines. Clean labels state the checked survival window in days and what was searched, with coverage limits. Evaluated Codex/Claude review outputs and their paraphrases are never label evidence. Preserve label revision history and recorded Franz overrides. This replaces the prior owner-only label rule under Franz's October 20, 2026 instruction at 8:10 PM EDT. The corpus-freeze/hash and credentials/activation gates remain unchanged.
-
Evaluate production modes at natural session counts, adversarial four, independent-pair two and each single provider one. Only the single-provider self-critique control matches four total sessions: one initial review and three bounded self-critiques, with a deterministic ledger preserving retractions/late findings. Use equal per-session limits and report different total costs; do not repeat shorter modes or invent aggregation rules.
Offline label policy and all 20 original labels merged in PR #216, 14 clean, 6 defect, none unlabeled. PR #217 adds a reproducible bounded 20-case corpus with the same counts, preserving the original labels and documenting its one eligibility-based substitution. Historical model exposure is unknown, so this corpus is not claimed to be model-unseen held-out evidence. Comparative runs, control-worker qualification, promotion and pilot criteria remain open.
Definition of Done
- Commit corpus manifest, frozen labels, evaluation methodology/commands and sanitized results with denominators and remaining blockers. Link actual gate evidence rather than treating model approval as validation.
- Record trusted-release promotion/rollback provenance, human-review bootstrap and owner-only pilot report in the source issue. No autonomous approvals, fixes or merges.
Validation
- Run the documented offline corpus/score harness and
go -C reviewer vet ./...plusgo -C reviewer test -race -count=1 ./...after module creation. Supply exact reproducible live-evaluation commands only after authorization, never invent successful runs. - Reproduce benchmark scoring from retained sanitized outputs and frozen labels. Audit equal-budget controls and quality/error denominators.
- Embed current-head screenshots of clean, known-defect inline review, disputes/late findings, incomplete/quota, cancelled and superseded GitHub states as uploaded attachments rendered in PR Validation; include a recording where practical or explain omission. Verify the running artifact predates each candidate PR and candidate changes cannot execute or promote it.
- Record validation output/evidence at the exact PR head SHA. For code changes, the reviewer reruns applicable commands locally; GitHub Actions CI is currently disabled.
Source continuation, October 21, 2026
The corpus remains 20 cases, 14 clean, 6 defect and 0 unlabeled, frozen SHA-256 807196990ee0adda9c8a2f87b0c9b6a5819baf886fa4196b58f6b102d7ea4910. PR #230, PR #231 and PR #232 merged control, release-decision and evaluation-runner source. Actual comparative model runs remain zero. Their 39 synthetic native phases are transport/protocol/recovery observations, not model-quality results. PR #234 merged four more controlled synthetic process-loss cases. PR #235 merged as 00d6a708 after a clean exact-head review; source-VM integration remains work. No corpus label, credential, promotion, inference or pilot gate is waived.
PR #236 merged as 0f333755 after clean exact-head review, all four threads resolved and verified delivery/cleanup. PR #237 merged as 60d64113 with the guest-role original-descriptor fix, clean exact-head review, full vet/race and six passing synthetic-client VM runs covering normal completion and cancellation. PR #238 is under review for equivalent original-instance cleanup in two inert diagnostics. Its 48 Python tests, actual escaped-child/replacement checks and KVM fixture pass; the inert host sealed-exec comparison still reports incompatibility. Production source-VM acquisition and startup integration, aggregate qualification and separate live gates remain open. Source work is not exhausted.
Activity
HemSoft commentedon Oct 19, 2026
Offline scoring and four-session self-critique controls are prepared in PR #183, currently awaiting required review. No live model benchmark or pilot result is claimed.
Two independent evidence routes were checked. First, retrieve 20 actual historical PR head/base candidates and their source links; the retained candidate manifest has zero human labels and is explicitly not frozen ground truth. Second, inspect retained corpus artifacts and run the actual offline score CLI against that unlabeled manifest: it exits 1 with no score output. The generated unit-test labels are synthetic arithmetic/protocol fixtures only. They cannot satisfy the named-human label criterion or establish review quality.
Exact remaining need: Franz must select/confirm held-out eligibility and assign evidence-backed labels for at least 20 cases, including ten clean controls and material-defect cases, then judge every real output's actionability with a rationale. Real runs additionally require passing #173/#174/#176 gates and explicit activation. Compare the natural 4/2/1/1 session arms with both separate four-session single-provider controls under the frozen common limits/order; preserve failures, missing runs, usage/turn availability and uncertainty.
The separately promoted trusted release, actual rendered GitHub cases, rollback checks and two-week owner-only pilot remain future work. Merging source will not promote a reviewer build. Human labels, provider permission, App identity and elapsed pilot observations cannot be replaced with fake results or automatic rollout decisions.
HemSoft commentedon Oct 19, 2026
PR #183 is merged as
3cc932d. The one owner-authorized retry completed with3P2/noP1 at36fdb420; all three findings are fixed in579fb35 and all17threads resolved. Vet and the full race suite passed. The merge used the invoked bounded-review rule; corrected579fb35 was tested but not freshly clean-reviewed. No GitHub CI pass is claimed. Actions remain disabled and no live reviewer was activated.Remaining acceptance: Named-human ground truth/actionability, actual supported live comparisons and two-week contained pilot.
Attempt1: Inspected all20 retained real candidates on published main; every human_label is null. No labels or successful benchmark are invented.
Attempt2: Ran the actual score CLI against that manifest on main3cc932d: exit1, missing structured field Version, zero score bytes. Synthetic CLI/protocol tests are not quality evidence.
Exact next owner/external action: A named human must freeze/label at least20 held-out cases with at least10 clean controls and material defects; pass prior live gates, authorize actual comparisons and conduct the two-week pilot.
Human judgments, external activation and elapsed pilot observations are external; the agent can execute the authorized frozen methodology afterward. This issue remains open; offline source completion is not live acceptance.
HemSoft commentedon Oct 19, 2026
PR #184 merged as
67077bd. Exact head8c3bdb3 passed vet/race and a clean connected-Codex review with no findings. Issue #175 is closed after its explicit fake-first protocol/configuration contract was fully audited. The new source broker and actual fake-worker executable path address the prior read-capture implementation gap. This does not qualify official-client confinement or activate the alpha.Remaining: Human ground-truth/actionability, actual supported comparisons and the two-week contained pilot.
Attempt1: Freshly inspected the20 candidates on published main; all human_label fields are null. Source-broker tests do not create labels or quality evidence.
Attempt2: Ran the actual score CLI on main67077bd against the candidate-only manifest: exit1, missing structured field Version, zero score bytes. No frozen human-labeled corpus/results/pilot is supplied.
Exact next owner/external action: A named human must select/freeze/label>=20 held-out cases with>=10 clean controls and material defects; pass live gates and authorize actual comparisons/pilot.
Agent can execute the frozen methodology after prerequisites. It cannot invent human judgments, account grants or elapsed pilot observations. This issue stays open.
HemSoft commentedon Oct 21, 2026
Merged PR #216: evidence-backed agent labels now replace the human-only rule, with recorded actor/model/session/time, independent later evidence, revision history and Franz overrides. Original 20 candidates: 14 clean, 6 defect, 0 unlabeled. All six defects reproduce before their later fixes and pass afterward. Clean labels state seven-day observation windows and retirement/coverage limits; evaluated review outputs were excluded. Vet/race checks and exact-head review passed; delivery and cleanup are verified.
Corpus freeze remains separate: original #156 exceeds the unchanged 128-path cap. A bounded twenty-case selection retaining #156's label outside evaluation and adding independently researched #148 is prepared for the next normal PR. Credentials, App installation, activation and live benchmark/pilot gates remain unchanged.
HemSoft commentedon Oct 21, 2026
Offline corpus follow-up merged in #217 after a clean exact-head review. Original 20 labels remain 14 clean, 6 defect, 0 unlabeled, as implemented in #216. The frozen bounded benchmark has 20 cases, 14 clean and 6 defect. Original PR156 remains labeled clean but has 335 paths above the unchanged128-path cap; PR148 was selected by merge chronology and packet eligibility before its independent clean label. Corpus hash
807196990ee0adda9c8a2f87b0c9b6a5819baf886fa4196b58f6b102d7ea4910. Full vet/race and fresh-clone offline source restoration/reproduction passed. No plan, benchmark inference, pilot or activation performed; remaining comparative/pilot acceptance stays open.HemSoft commentedon Oct 22, 2026
Offline execution progress, October 21, 2026 EDT:
7dd5d612016b9ea0bd396230f0f5fc6342943d88, all four threads resolved, then merged asbd5cd59ec873ec18c849ad96bd179d5242bef9cfwith the identical tree. Vet and the full opt-in race suite passed, main package 243.811s. Fresh exact-source qualification completed 39 native VM phases across a 16-phase six-arm run and stopped-result fault runs of 1, 9 and 13 phases. It checked request-bound recovery, charged incomplete retirement, no replay, physical stop, returned synthetic auth and runtime disposal. These were local canned TLS responses with fresh synthetic auth, with no model inference.807196990ee0adda9c8a2f87b0c9b6a5819baf886fa4196b58f6b102d7ea4910. Historical model exposure is unknown.The native execution fixture runs one synthetic case and reports the other 19 missing; it produces no quality score. All 20 real corpus packets were separately restored locally without fetching, but actual comparative model runs remain zero. Execution and recovery contract.
Actual independently assessed six-arm results, provider/account eligibility, aggregate containment/recovery, trusted release/plan approval, GitHub rendering cases and the owner-authorized two-week pilot remain open. PR #233 prepares their dated gate checklist under normal review; it grants no activation. This issue remains open.
HemSoft commentedon Oct 23, 2026
Reconciliation as of October 23, 2026 (US Eastern). Comparative model runs: zero. No quality, latency, quota or pilot result exists.
Delivered offline:
807196990ee0…4910. Historical model exposure is unknown, so it is not claimed as held-out.Today's #250 E2E uses canned reports. It shows transport and publication, not review quality.
Open, with the gate for each:
Recommendation: keep open. No comparative, pilot or rendering criterion can be met offline.
🤖 Generated with Claude Code
HemSoft commentedon Oct 28, 2026
Park comparative benchmarks and the two-week gate. Franz judges usefulness and noise on three real reviews.
The parked work stops. Revival requires the original unresolved gates, as stated in the approved plan.
HemSoft commentedon Oct 29, 2026
Closed per Franz (2026-10-29 19:34 ET, set-it-free-loop post: 'Agreed'): SFL stays inside pmd (project-manager). The standalone alpha controller, adapters and publisher are not being built. Next SFL work is project-manager #75 (review status) and #76 (changes between rounds). Reopen if that direction changes.