Skip to content

Define the dual-subscription adversarial reviewer alpha #172

Closed as not planned

Description

@HemSoft

Description

Franz requests a new owner-operated GitHub App reviewer. Both his Codex and Claude subscriptions must participate in adversarial review in v1, with per-PR controls and alpha limited to this repository. This issue is the shared implementation contract and tracking record.

Verified on October 19, 2026 as HemSoft: repository hemsoft-dev/set-it-free-loop is public, ID 1169772257; Actions is disabled, registered workflows are zero, and all 68 historical issues were closed before this backlog. The project freeze remains in force. The owner's October 19 merge-mission invocation now authorizes scoped offline implementation and guarded merges. Installing an App, creating credentials, starting inference or deploying still requires separate explicit activation authority and passing gates. Offline implementation need not wait for provider eligibility evidence.

The retired controller contains hs-buddy/App IDs. Build a separate Go module, provisionally reviewer/, and preserve retired source/history. Closed #125, #137, and #139 delivered the former native-Codex/org pilot. This is a different, dual-provider self-review alpha.

Why

Franz wants useful inline reviews through an authorized @sfl review comment and automatically when a PR opens or its head receives a push, using subscriptions he already owns. Independent model findings plus peer critique may improve reviews, but that benefit needs measurement. Subscription support remains unresolved for this exact public target. The model-free App experiment failed native request creation; Franz selected the two polling triggers instead.

Goal

An owner-only review request in this repository runs the chosen mode against an immutable snapshot and publishes an advisory COMMENT review with a concise summary, source-linked findings, disagreements, reviewed revisions and coverage limits. The requested default is adversarial, using both providers. A failed dual run never silently becomes a successful single-provider review.

Trigger decision — October 21, 2026

Franz chose @sfl review plus automatic PR-open/head-push review on October 21, 2026 at 01:19 EDT, via Discord relayed by the Chief. Implement both offline with fake transports and normal review/merge gates. Discovery stays polling-based; no webhook, machine user, live inference, subscription use, deployment or paid usage is authorized by this decision. Native-picker requests are no longer required. PR #222 records the failed native experiment; #221 is closed unmerged.

Acceptance Criteria

  • Implement both official, unmodified CLI adapters in v1. Default to adversarial; allow explicit independent-pair, codex-only, and claude-only. No API billing, provider substitution, repository visibility change or machine-user identity is an inferred fallback. The two triggers below are explicitly authorized.

  • Record provider/client/auth support for unattended, owner-only review of this public repository. OpenAI's cached-account CI guide excludes public/open-source targets. A private polling host is not a documented exception. Require applicable documentation or written provider confirmation before live use; a successful inference does not establish permission. Claude subscription authentication and credential-use rules also require an account-specific evidence record. Native Codex GitHub review is not a proven peer-critique API.

  • Implement both polling triggers: an original authorized @sfl review PR comment and automatic PR opened/head synchronize. Preserve comment author/ID/body hash/time for manual requests, and verified PR-event or explicit polling-observation provenance for automatic requests, without inventing a human requester. Qualify duplicate discovery, downtime, cancellation, new heads and re-request by a fresh comment. Record the native-picker failure as historical evidence, not an activation prerequisite.

  • Default protocol has four bounded reviewer sessions. R1 gives each provider the same initial evidence packet, immutable head/base-tip/merge-base and source capabilities, without peer output. Record actual reads. After both R1 outputs validate, R2 gives each provider both frozen reports and requires a cited disposition for every peer finding ID. Retractions remain in the ledger. Proposed tests are never reported as executed. R2 is peer critique; authors do not see the peer's R2 objection in this default protocol.

  • Use a deterministic Go ledger, no third adjudicator model. Preserve finding IDs, provenance, objections, anchors and severity differences through deduplication. corroborated requires explicit source-grounded peer confirmation and remains model-assessed. disputed preserves substantive objections, unverified includes insufficient evidence and new R2 findings, retracted records origin withdrawal, and invalid records schema/source failures. Schema/protocol failure fails the phase. Independent duplication and valid anchors alone never establish correctness.

  • Show corroborated findings inline by default; show disputes and late/unverified findings in clearly labeled summary sections. Never erase an allegation solely because its peer rejects it. One explicitly configured additional paired round may address disputes/late findings, capped at six sessions. Sessions are not individual inference calls; bound internal turns/tools/retries separately.

  • Both providers and all required phases must validate on the captured snapshot/configuration for dual_complete. Preserve valid phases for bounded same-snapshot resume. Distinguish queued, running, quota wait, authentication failure, provider unavailable, invalid output, incomplete/expired, cancelled, stale, publication unknown and completed outcomes. Alpha does not publish substantive partial findings. Non-substantive status may report INCOMPLETE and its cause. No incomplete state implies clean/pass.

  • Use typed owner commands, initially /sfl config mode=..., show, reset, and /sfl cancel. Controller defaults plus sticky per-PR records select pinned allowlisted models and budgets. Only Franz's verified immutable GitHub ID can configure or issue @sfl review. Automatic runs are authorized by the enrolled repository policy and attributed to the PR event, not a forged owner comment. Configuration alone never starts review. Ignore candidate-head files/body and model output as control inputs.

  • Record command/event ID, immutable author ID, body hash and effective config/version/hash at admission, with exact models, clients, auth modes, budget and revisions. Reject edited-before-admission commands, unknown keys, raw flags/endpoints/paths/credentials and unlisted models. Later command edits/deletion cannot rewrite accepted state. Config changes affect the next request with an acknowledgment; current/resumable jobs retain their captured config. Explicit cancel advances generation; replacement work needs a fresh authorized comment or a new head event after cancellation. A head push creates fresh automatic work, including after completion; duplicate observations of the same request never rerun it. Base-only movement invalidates unpublished work and does not itself authorize a new review.

  • Use a private polling service and durable SQLite state, one active job, sequential provider sessions. Enroll only repository ID 1169772257 and its exact installation, with App selected-repository scope. Reject other IDs before provider work or writes. Persist manual-comment and automatic-event/observation identities, cancellation/replacement generations, job phases, lease and outbox. Reconcile paginated history after downtime; when requester attribution is uncertain, do not start inference.

  • Separate controller, confined source reader, provider workers and publisher. Models receive no GitHub token; publisher receives no provider credentials. Each pinned official client has separate protected auth and disposable execution state. Suppress ambient hooks/settings/plugins/MCP and candidate instructions. The sole source tool reads immutable job objects through a separate identity with no provider credentials. No arbitrary shell/filesystem/network access, host home, SSH agent, Docker socket, LAN/private-network or other job state. Enforce restrictions outside CLI permission prompts and prove them using hostile fixtures before credential-bearing probes.

  • Check subscription auth before every phase; reject API-key mode and cross-provider credential inheritance. Serialize and persist Codex refresh state outside disposable jobs, without sharing Franz's interactive auth file. Prove paid overflow disabled or an enforceable block. Apply finite per-phase/job/day limits for sessions, duration, turns, tool calls, bytes, memory, retries and retention. Quota waits have expiry/reset policy. Do not upload raw transcripts or credential-bearing artifacts.

  • Recheck authorization, request generation, head and base-tip immediately before and after publication. Either revision moving invalidates unpublished work even if merge base is unchanged. Never mark request B fulfilled by A's review. Requests are tracked durably by their comment/event identity; COMMENT publication never clears or fulfills a later request. Preserve uncertainty and pause for reconciliation when dispatch or request provenance is ambiguous. Never dispatch A after cancellation or replacement is known to the publisher. Failure to establish an acceptable recovery path blocks live publication. Persist publication intent and reconcile ambiguous POSTs by author/snapshot/request marker before retrying. GitHub's review API has no atomic compare-and-publish across these states; every initial review names its snapshot, and supersession is recorded locally and annotated remotely when access permits.

  • Bootstrap with manual reviews and independent human merges/checks. Later dogfood uses a separately promoted pinned trusted release, policy, reader and publisher. Candidate code, including reviewer prompts/runtime/auth/allowlist changes, remains inert input. Merging never automatically promotes the reviewer. No organization expansion, autonomous fixes/merges or executable candidate tests in alpha. Automatic PR-open/head-push review is explicitly authorized for offline implementation and retains all live-activation gates.

  • Evaluate at least 20 held-out cases, including 10 clean controls, with frozen labels and defect evidence. Compare adversarial, independent-pair, both single providers, and equal-session-budget single-provider self-critique. Report precision/recall denominators, clean errors, concessions/disputes, usage, latency and queue time. Provisional goals are 90% actionable precision, 70% material-defect recall and 90% clean-control success; report small-sample uncertainty. Poor dual results stop rollout, not silently change the owner's default.

  • A configuration acknowledgment identifies its durable version. Requests created after that acknowledgment use that version; earlier requests retain the preceding version regardless of discovery order. Ambiguous ordering defers admission. Recovery reconstructs the same mapping from persisted records.

  • independent-pair runs two independent R1 sessions without cross-examination. Each single-provider mode runs one R1 session. Completion records the selected mode and its required phases; selecting these modes is not incomplete execution. Without peer examination, findings remain unverified in the summary. Four/six-session rules apply to adversarial mode.

  • R3 receives both frozen R2 reports and finding lineage. It permits responses to objections and peer examination of late findings. A late finding may become model-assessed corroborated only after explicit source-grounded peer confirmation in R3. Preserve prior objections/dispositions; unresolved objections remain disputed. New R3 allegations stay unverified, with no further round. Selecting R3 requires both additional phases for completion.

  • Record omitted paths, missing history, truncation, binaries, submodules and reader-limit exhaustion. Unreadable or omitted material never counts as clean. Preserve source-valid findings that cannot anchor in the diff as summary findings.

  • Render model text as untrusted content: neutralize mentions, prohibit remote images/raw HTML and allow only explicitly approved links. Generate source links from trusted snapshot/path data.

  • If a pinned client cannot remove a forbidden capability, record that provider's live use as blocked. Do not relax the boundary or declare v1 complete. A different isolation contract, such as a model-accessible shell confined to a snapshot guest, requires an explicit owner decision.

  • Select the optional paired extra round in trusted per-PR configuration before request admission. Alpha does not offer extension after completion; changing the round setting requires a fresh authorized review request. Both selected extra phases must validate for completion.

  • Agents may create and revise ground-truth labels and actionability judgments from independent evidence. Every label records its agent/provider/model/session, time, rationale and evidence. Defects cite a concrete later fix/revert commit or PR, or linked bug/regression issue, plus affected candidate lines. Clean labels state the checked survival window in days and what was searched, with coverage limits. Evaluated Codex/Claude review outputs and their paraphrases are never label evidence. Preserve label revision history and recorded Franz overrides. This replaces the prior owner-only label rule under Franz's October 20, 2026 instruction at 8:10 PM EDT. The corpus-freeze/hash and credentials/activation gates remain unchanged.

  • Ignore non-owner commands without replying or making GitHub writes. Rate-limit owner acknowledgments and record the head repository ID in every job.

  • Persist configuration acknowledgment intent/version/unique remote marker before dispatch. An uncertain acknowledgment write defers dependent admission until paginated remote reconciliation recovers its identity/time. Never blindly duplicate an acknowledgment; test a crash after acceptance but before local response persistence.

  • Evaluate production modes at natural session counts, adversarial four, independent-pair two and each single provider one. Only the single-provider self-critique control matches four total sessions: one initial review and three bounded self-critiques, with a deterministic ledger preserving retractions/late findings. Use equal per-session limits and report different total costs; do not repeat shorter modes or invent aggregation rules.

  • Polling is reconciliation, not webhook delivery: pin PR identity/head repository/head/base and retain stable observation generations. Coalesce obsolete heads discovered after downtime; never invent unavailable push actors, timestamps or event IDs. Repeated identical snapshots do not rerun. A visible A→B→A transition is a new generation; document the inherent polling gap for unseen intermediate pushes. Fresh manual comments remain a durable explicit re-request. Preserve owner-only cancellation and config-ack ordering, including uncertain acknowledgments and equal timestamps.

  • When trigger implementation is complete, publish an exact checklist of remaining live-activation blockers: provider public-target/account permission, subscription/auth and paid-overflow evidence, completed containment/recovery/control/benchmark qualification, pinned trusted-release promotion, required GitHub rendering proof and explicit live owner authorization. No trigger test or merge grants activation.

Definition of Done

  • Publish and cross-link five bounded implementation issues for eligibility, adapters/source isolation, protocol/configuration, controller/publication, and evaluation/dogfood.
  • Record Astra and Opus review dispositions and any remaining objections. Both reviewed two rounds and approved backlog publication after bounded amendments. Incorporated changes cover public-target gates, request races, config ordering, mode/R3 completion, coverage/rendering, freeze authority, client capability failure, partial-output prohibition and the original human evaluation-label rule, superseded by the owner instruction above. Their approval concerns the plan, not demonstrated support or security.
  • Keep pass/fail/blocked gate evidence, activation authority and implementation progress visible here. Issue creation changes no runtime state.
  • Close this tracking issue only when all implementation issues are closed as completed, or Franz explicitly records a closure decision identifying remaining blocked gates.

Implementation order: eligibility evidence may proceed in parallel with authorized offline work; adapters/source reader → protocol/configuration → controller/publisher → evaluation/dogfood. Live work always waits for its applicable gates. Implementation issues:

Validation

  • Review every implementation issue against this contract and its dependency order. Neither provider can be deferred out of v1.
  • Reproduce repository identity/freeze/Actions checks as HemSoft; verify no credentials or automation were restored during planning.
  • Require focused offline tests and documented Go gates in each changed module. The new reviewer/ module exists; go -C reviewer vet ./... and go -C reviewer test -race -count=1 ./... have passed at recorded exact heads. The October 21 source reconciliation below identifies the current evidence and its limits.
  • Require current-head screenshots of implemented GitHub review states as uploaded attachments rendered in implementation PR Validation sections, plus a recording where practical. No current alpha UI exists; do not start frozen runtime to obtain planning screenshots.

Merged activation checklist, October 21, 2026

PR #233 merged the dated six-gate checklist as ab260ad2d6883ddc4a22446a58ebfb93161296c6, after clean required review of e4c4a82a3b2178a4e9346c2955d0cdcbd3f00c38 and five addressed threads. It separates implemented mechanisms from unproved physical and live evidence and prepares the selected-trigger provider questions without sending them.

The six live gate groups remain open: applicable provider permission; account/auth/paid-overflow evidence; complete aggregate containment/recovery; actual comparative evaluation plus independently qualified trusted release/plan and pilot; final selected App/trigger/rendering qualification; separate explicit owner activation. PR #234 merged as 917d854 after a clean exact-head review. Its four controlled synthetic supervisor/driver-loss cases prove original-unit stop/recovery at those points, not arbitrary host crashes or aggregate qualification. The original host noswap/sealed-exec failures remain failures. PR #235 merged as 00d6a708 after a clean exact-head review and all 15 threads resolved. It delivers an inert measured VM prerequisite diagnostic and a design for a distinct VM source adapter; it grants no activation. This issue remains open.

Protocol source reconciliation, October 21, 2026

Three checked protocol criteria concern implemented offline behavior. Config mode/round selection and the engine preserve natural two/one-session modes and require both preselected extra phases. Engine tests exercise selected-mode completion, unverified findings without peer examination, four/six-session execution and same-captured-job resume. Configuration tests verify typed next-request changes, captured-config immutability and refusal of unlisted or excessive rounds.

Rendering source and its hostile-content tests verify neutralized mentions, escaped model HTML/Markdown, removed external model links and trusted snapshot-generated source links. This proves offline renderer behavior, not actual GitHub presentation.

These seven protocol source/test blobs are unchanged by merged PR #234. Its final reviewed head df3ec524348b8d18caaf2a0c59c7287e16c25106 passed vet and the full race suite, 195.255s wall, before guarded merge as 917d854. PR #232 also retains reviewed-and-merged exact-head validation. These are offline source checks and grant no activation authority.

The broader provider/account, aggregate physical containment/recovery, actual comparative benchmark/pilot, final App/trigger/rendering, trusted promotion and owner-activation requirements remain unchecked and open. This reconciliation does not close this tracking issue or any implementation issue.

PR #236 merged as 0f333755 after clean exact-head review, all four threads resolved and verified delivery/cleanup. PR #237 merged as 60d64113 with the guest-role original-descriptor fix, clean exact-head review, full vet/race and six passing synthetic-client VM runs covering normal completion and cancellation. PR #238 merged as 39bc435 after a clean exact-head review, zero threads and verified delivery/cleanup. It binds two inert diagnostics to their captured original unit. Its 48 Python tests, actual escaped-child/replacement checks and KVM fixture pass; the inert host sealed-exec comparison still reports incompatibility. Production source-VM acquisition and startup integration, aggregate qualification and separate live gates remain open. Source work is not exhausted.

Activity

  1. HemSoft commented on Oct 19, 2026

    @HemSoft
    CollaboratorAuthor

    Merge mission closeout: research/plan documents (#178), eligibility evidence (#179), offline adapters/source reader (#180) and bounded protocol/configuration (#181) merged. #182 was superseded by the complete controller/evaluation source in #183; that tested PR is still open because its required current-head connected-Codex review failed with an unknown provider error.

    The two review recovery routes were the original controller PR and the combined current-head implementation against surviving main. Every known finding is fixed, but a failed review is not clean evidence. The last reviewed head included P1, fixed in 36fdb42; the bounded-review shortcut therefore cannot clear the missing fresh review.

    The alpha tracker stays open. #173 requires applicable public-target support, account/overflow evidence and explicit App identity/activation authority. #174 requires qualified official workers/source isolation. #175 still needs the reviewed controller integration merged. #176 requires the real authorized App/transport experiment and rendered recovery proof. #177 requires named-human labels, actual live comparisons and the two-week pilot. Offline tests, real no-auth probes, primary-source research and authenticated installation inspection were exercised; they do not substitute for these external facts.

    All source is preserved, main is clean/synchronized, Actions remains disabled with zero workflows, and the single active PR branch/worktree is retained. No App, worker credentials, paid API fallback or live activation was created. This is an incomplete product, not a successful live alpha.

  2. HemSoft commented on Oct 19, 2026

    @HemSoft
    CollaboratorAuthor

    PR #183 is merged as 3cc932d. The one owner-authorized retry completed with3P2/noP1 at36fdb420; all three findings are fixed in579fb35 and all17threads resolved. Vet and the full race suite passed. The merge used the invoked bounded-review rule; corrected579fb35 was tested but not freshly clean-reviewed. No GitHub CI pass is claimed. Actions remain disabled and no live reviewer was activated.

    Remaining acceptance: Unfinished children #173-#177, rather than an unmerged source PR.

    Attempt1: Inspected all five merged implementation/research PRs and the integrated main tree.

    Attempt2: Independently revisited authenticated repository/App inventory and the actual corpus/score gate; no live eligibility or human labels are supplied.

    Exact next owner/external action: Pass the child owner/provider/qualification gates, or explicitly dispose of the alpha.

    The agent can execute later authorized integration and evaluation; owner/provider/human facts cannot be fabricated. This issue remains open; offline source completion is not live acceptance.

  3. HemSoft commented on Oct 19, 2026

    @HemSoft
    CollaboratorAuthor

    PR #184 merged as 67077bd. Exact head8c3bdb3 passed vet/race and a clean connected-Codex review with no findings. Issue #175 is closed after its explicit fake-first protocol/configuration contract was fully audited. The new source broker and actual fake-worker executable path address the prior read-capture implementation gap. This does not qualify official-client confinement or activate the alpha.

    Remaining: Completion depends on unresolved173/174/176/177. The175 offline protocol component is closed.

    Attempt1: Audited merged source and the12-criterion fake-first175 contract, then closed175 with evidence from181/183/184.

    Attempt2: Independently revisited support/account/App state and human corpus/scorer state; none supplies live acceptance.

    Exact next owner/external action: Pass remaining child gates or explicitly dispose of the alpha.

    The agent completed available offline protocol work; owner/provider/human facts remain external. This issue stays open.

  4. HemSoft commented on Oct 23, 2026

    @HemSoft
    CollaboratorAuthor

    Tracking update, October 23, 2026 (US Eastern). The merge mission merged #243–#254 on October 22 and 23, using fakes only. No App installation, credential, inference, paid usage or activation.

    Children:

    • #173 eligibility: blocked. Two self-serve attempts on October 22 found the unattended public-target case excluded (OpenAI) or not covered (Anthropic). Sending the prepared provider questions needs Franz's authorization.
    • #174 adapters and isolation: the offline work listed for it is merged. That covers hostile fixtures (#248), exhaustion (#249), aggregate E2E (#250) and operator recovery (#251). Its acceptance items AC4 and AC7 are still not met.
    • #175 protocol: closed.
    • #176 controller and publisher: offline mechanisms are delivered. Selected-App, rendering and activation qualification is gated.
    • #177 evaluation and pilot: zero comparative model runs.

    Criterion met today: "If a pinned client cannot remove a forbidden capability, record that provider's live use as blocked." #248 did this for Codex. Hostile /v1/models metadata made the pinned client advertise apply_patch and MCP resource tools, and our response policy refused their use (HOSTILE-FIXTURES.md). This box can be checked.

    Neither provider can be deferred out of v1, so v1 cannot be complete while Codex is blocked.

    Remaining gates, by owner:

    1. Provider permission: authorize the prepared questions, or record a decision. (Franz)
    2. Account entitlement, auth and refresh, and disabled paid overflow, per account. (Franz)
    3. Codex code mode: whether model-written JavaScript in the confined code host is acceptable. Also choose how to handle the metadata boundary: rely on the response policy, build a host request policy, or wait for a client fix. (Franz)
    4. Selected-App trigger, permission and rendering qualification. (Needs App and credential authority.)
    5. Trusted release promotion, then the comparative evaluation and the two-week pilot. (Franz, after gates 1–4.)
    6. Explicit live activation. (Franz)

    Open offline follow-ups:

    Recommendation: keep open. Under the Definition of Done, it closes only when every child is closed as completed, or when Franz records a closure decision that names the blocked gates.

    🤖 Generated with Claude Code

  5. HemSoft commented on Oct 28, 2026

    @HemSoft
    CollaboratorAuthor

    Closing as not planned, superseded by the approved native-review plan. The dual alpha was not completed. Record of Franz’s approval: he said "yes" on Discord at 00:40 EDT on October 28, 2026.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions