Repository navigation
Recover a stopped provider attempt through an explicit operator action - #251
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
c79df87 to
f0a4bc2
Compare
5ee0566 to
34c9d96
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 34c9d961fc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
34c9d96 to
d92ee65
Compare
|
@codex review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
6be4696 to
08fd121
Compare
Refs #174 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #174 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #174 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #174 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #174 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #174 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A resumed job's retry can repeat the session hash and stopped provider evidence, so an earlier attempt's complete-looking launch under the same provider root passed recovery and released the current lease in the no-auth-journal case. DurableReviewer now passes the attempt ID to the native reviewer, the version 2 stopped audit receipt records it, and recovery and the candidate scan accept only a receipt bound to the outstanding attempt. Receipts without the ID fail closed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A recovery retried after Complete advanced a never-exported lease to committing or committed reported returned auth, and the durable recovery receipt kept that false credential history. Settlement now splits the commit stages by transfer stage: an empty transfer is unexported and a returned transfer is returned, each refused if the stopped receipt contradicts it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A cancellation or replacement can fence a job while its stopped attempt still holds the lease. Recovery reported and recorded auth_failed or provider_unavailable although the release transaction then restored cancelled or stale. The store release now returns the job's final state after the PR fence, and both the result and the recovery receipt use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
close-provider-recover no longer compiled after the rebase onto the tightened harness, and it had no declared end state. It now expects one source unit and four native units from PR 2's review, both auth stores usable at the end, and two owner journal records: PR 1's retained attempt and PR 2's, each with one staged export. A sample of the unit observer and the launch directories after recovery proves that startup and recovery started no unit. The verdict unit test covers the attached recovery end state and the ways it can drift. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebased harness still accepted only version 1 stopped audit receipts, but this branch writes version 2 with the durable attempt ID, so every real launch would have counted as unaudited. The projection now requires version 2 and a well-formed attempt ID, and the session verdict binds each launch to its job: a completed launch must name the attempt the job recorded for that artifact's session and repeat its provider evidence, and an interrupted launch must name a recorded attempt of its own session that no artifact used. close-provider, close-provider-reopen and close-provider-recover also check that the interrupted launch names the leased attempt, and recovery judges PR 2's launches together with PR 1's recovered job. The pre-rebase recovery runs are marked as history: their receipts are version 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the rebase onto the tightened qualification harness, an attached root no longer treats its native launches as pre-existing by default, so close-provider-recover would have counted the interrupted launch as its own. It now proves the retained launch with e2eRetainedLaunches before excluding it, as close-provider-reopen does. That rule now also binds the version 2 receipt: it must name the leased attempt's durable ID and repeat the attempt's recorded stopped evidence, so a version 1 receipt, which names no attempt, is refused. The reopen case's separate attempt check is folded into it. close-provider-recover also checks exact requests per PR: PR 1 retains only its admitted opened request when attached, after recovery (which releases the job but not its request) and at the end; PR 2 ends with only the owner comment request, admitted as its reviewed job's request; no other PR has one. The final projection is recorded as Requests. Store close and evidence replacement already run through the shared paths. The qualification document describes both rules for the recovery case, notes that the e144bc5 measurement predates recovery and version 2 receipts, and adds the pending measurement section its history links to. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
close-provider-recover now takes its retained job through e2eAttachedJob, as close-provider-reopen does, so a root retained by another revision's close-provider is refused before recovery; the job must also still have an outstanding attempt. Its request checks use the shared helpers: PR 1 is judged with e2eOpenedRequestVerdicts against the leased job when attached and after recovery, and the end state with the new e2eRecoveredRequestVerdicts, whose job request IDs go through e2eRequestOf so a missing job or a zero request ID matches nothing. The after-recovery and final projections are recorded as Requests through checkRequests. TestE2ERetainedLaunches projects the retained launch under the release policy and session limits from an interrupted receipt that passes production validation, and refuses one that fails it even when the job records the same evidence. TestE2ERequestVerdicts covers the recovered helper with missing jobs and zero request IDs. The qualification document applies the attached-release rule to recovery, states the source export binding for its two records, and keeps the unmeasured head section, noting the e144bc5 run predates operator recovery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the restack onto #250's round 7 harness, close-provider-recover still called the removed assertReviewed and assertNativeSessions helpers and the two-argument e2eAttachedJob, so the package did not build. It now judges its end state with the completed-case rules, with PR 1 recovered instead of reviewed: - Exactly one job per PR, no PR 1 intent, one PR 2 intent and no other write, at attach, after recovery and at the end. - PR 1's job is the retained job itself, auth_failed without lease or outstanding attempt, charged for its one session with no artifact or source access, recording only the leased attempt with the interrupted launch's evidence and one recovery receipt bound to that attempt, round, launch and evidence digest with the auth retired. No PR 1 POST or review. - PR 2 passes e2eReviewedVerdicts: exact R1/R2 rounds, backend checks, intent binding and the remote review matched through E2ERemoteReview. - Two source records, PR 1's retained one and PR 2's, each bound to its own job's snapshot and the current release and runtime hashes. - The interrupted launch is still present with its receipt unchanged and names PR 1's recorded attempt; PR 2's four launches name their artifacts' attempts; nothing else launched. - PR 2's acquisition is the only upstream exchange (checkUploads). The attached job goes through e2eAttachedJob with PR 1's fake upstream snapshot. e2eDurableCountVerdicts states exact per-PR counts and shares its counting with e2eDurableSetVerdicts; e2eReviewFixture parametrizes the completed fixture by PR for e2eRecoveredFixture and TestE2ERecoveredVerdicts, which names the verdict each edit must fail. The final-verdict recovery fixture's native units carry the bound audit round 7 requires. The qualification document states the recovered end-state rules and keeps the head section unmeasured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the restack onto #250's round 8 harness, close-provider-recover's end state is judged by the round 8 rules where they apply: - PR 2's four launches and the interrupted one are projected from every entry of the attached native roots as pinned native-vm payloads with the release's stub and manifest hashes, and their version 2 receipts must repeat that manifest hash. Any other entry fails the launch verdict. - PR 2's sessions must meet the fixture exchange contract, with served objects equal to the fake upstream's bytes at PR 2's head. The recovery process's backends must have checked exactly PR 2's artifact sessions, none held, so PR 1's interrupted session never runs again; its retained receipt must count exactly its one held inference, forwarded and never answered. - PR 2's source attempt, the only one this process acquired, must pass the lifecycle order from the journal sampled at its upload-pack. PR 1's retained attempt, whose order close-provider judged, is judged by its final record only. Both records must carry the release's stub hash. The review fixture now serves PR n's own file, the recovered fixture carries the held receipt's counters, the retained record's stub hash and invocation, and TestE2ERecoveredVerdicts names the target check of each new refusal. The qualification document states the rules for recovery, points the unmeasured head section at the b876290 run, and notes that run predates operator recovery and version 2 receipts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebased onto the qualification branch's round 9, close-provider-recover now meets those rules where they apply: - PR 2's stored reports and confirmed document are judged against the canned backend reports and production's renderer through the shared reviewed-PR verdicts; PR 1 has no artifact and no document. The recovered-verdict test refuses an altered PR 2 report or a document the intent and remote agree on but the canned reports do not produce. - The quiesced unit sampler applies unchanged: the case samples the observer only before finish, whose stop joins the periodic sampler. - The settled re-read covers the recovery process's own fake servers, which hold only PR 2's acquisition, sessions and review POST. A new test refuses a late rerun of PR 1's interrupted session, a late PR 1 review or refetch, and late PR 2 requests. - Auth dispositions are exact. Codex is fenced before recovery and after the refused attempts. An unfenced store with no state is now "retired", exactly what production's AuthProfile.Retire leaves (no state, no lease), and recovery must leave Codex retired with Claude usable. The re-provisioned seed is a fresh synthetic profile carrying the original synthetic tokens, the only credential the canned backend accepts; the store must hold exactly those bytes, and finish requires both stores usable. An empty store was previously judged invalid. The documentation keeps the daac99c measurement, noting it predates operator recovery and version 2 receipts, and points the not yet measured placeholder at it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebased onto the qualification branch's round 10, close-provider-recover now meets those rules where they apply. The rebase itself carried the version 2 receipts through: the audit mirror E2EReadNativeAudit reads persistAudit's version 2 encoding with AttemptID, the launch receipts test confirms that production writes it naming the phase's attempt, and every launch verdict binds the named attempt. - PR 1's recovered job must equal the retained job with only its state, lease, deadline, reservation and pending charge released and exactly one recovery receipt appended, so its attempt set is exactly the leased attempt with no source attempt. The receipt must carry a recovery time. - The interrupted launch's audit must still bind the recovered attempt after recovery and at the end: its derived request, the zero report, no source receipt, stage result and the auth-return failure. - PR 2's job holds exactly its four artifact attempts and no recovery receipt, and no completed job of any case may carry one. The end state also applies the shared attempt-set verdict to both jobs. - The operator candidate view must equal the leased attempt's candidate field for field. - The recovered fixture is built from the round 10 retained and review fixtures, with this boot's source records and the job's packet hash on each export. The recovered-verdict test names the target check of each new refusal, and the final-verdict recovery fixture carries whole stops. The record inventory lists the recovered job, the recovery receipt, the candidate view and the auth stores. The not yet measured section names the round 10 rules and points at the 89382c7 run, which predates operator recovery and version 2 receipts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
close-provider-recover no longer compiled after the rebase onto #250's round 11 and judged neither job against the harness's inputs. The attached job is now the one e2eAttachedJob derives under the harness configuration. PR 1's recovered job must be exactly the job admission derives from its retained open request, the fake upstream's snapshot, the configuration at version 1 and the current release, held in the retained job with only the released fields and one recovery receipt changed. PR 2's job is judged by e2eReviewedVerdicts against the job derived from the fake upstream's owner comment, and its retained request must carry that comment's pull and comment IDs. Sessions, source records, launches and the one extra interrupted Codex R1 session all derive from the expected jobs, and PR 2's remote review is bound to the fake upstream's head. Recovery uses no GitHub route, so the recover process must serve exactly the publishing per-role route set, matching the tally of the last real run at bb6d865; close-provider before it keeps the qualified set. Auth keeps retired (no state, lease, profile or pending journal) distinct from unavailable, with fenced only on ErrAuthRecoveryRequired. The fixtures gain per-PR origins and consistently rebuilt variants (e2eRetainedFixtureOf, e2eRecoveredFixtureOf), and the verdict tests refuse wrong origins, configurations, IDs, requesters, retained requests, remote review commits and route sets for the recovery case. The record inventory and recovery steps describe the round 11 rules, and the Not yet measured placeholder links the 13251e3 run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d92ee65 to
ec737a3
Compare
|
Rebased onto Validation at @codex review |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Part of #174. Based on
main(which includes #250's end-to-end qualification and the fixes #252, #253 and #254 it surfaced). It fixes the liveness gap #250's end-to-end qualification found. Closing the service during a provider session left the job leased and the Codex auth fenced, and after a restart nothing could ever be reviewed again.Design
The recovery is an explicit operator action. It never runs automatically: there is no timer or polling, and no GitHub, PR or model input can reach it.
GitReviewerService.ProviderRecoveryCandidates(ctx): a read-only, nonsecret list of leased jobs with their attempt, provider, round, session hash, whether stopped evidence is recorded, and matching launch directories.GitReviewerService.RecoverStoppedProvider(ctx, ProviderRecoveryRequest{JobID, AttemptID, LaunchID, RetireUnreturnedAuth}).Recovery refuses without any durable change unless all of the following hold:
ErrLiveBlocked);Auth then follows the existing stopped-worker rules:
provider_unavailableprovider_unavailableRetireUnreturnedAuth; otherwise refuseauth_failedauth_failedAn exported seed is never reused without its durable return.
The lease is released in one store transaction under the same lease token, which rechecks the attempt, provider, round and evidence digest and appends a
ProviderRecoveryReceipt. The session stays charged, no artifact is created and the job is never marked reviewed; only the existingResumeUnavailablecan requeue it. Without an operator call, behaviour is unchanged.Review fixes
native-audit.jsonis version 2 and records the durable attempt ID; recovery accepts only a version 2 receipt naming the job's outstanding attempt, so an older launch with the same session hash cannot be recovered (Codex P1).unexported, notreturned, in the result and the durable receipt (Codex P2).cancelled/stale), not a pre-fence state (Codex P2).close-provider-recoverE2E case follows every rule Qualify the reviewer end to end on a real host with fakes #250's qualification gained in its review rounds: production readers and validators for every durable record, exact attempt and request sets, the recovered job equal to the retained job but for its released fields and one recovery receipt, the exact operator candidate view, expected jobs from harness inputs, exact per-role GitHub route sets, and exact auth dispositions (fenced→retired→usable).Validation (exact head
d1b5d72, measured; docs-onlyec737a3on top)git_service_recovery_linux_test.go, fake manager, kernel and auth): the retire-and-release path; finishing a durable return; an unexported lease, including an interrupted commit; 13+ refusal cases each leaving lease and fence untouched, including another attempt's launch and a version 1 receipt; refusal while running or closing; fenced-job final state; the store release binding attempt, evidence and token.reviewer.testbuilt fromd1b5d72, VCS-stamped; one VM at a time behind a ≥2.5 GB available-memory gate; release hash1ec49cbb906e):In
close-provider-recover, Codex auth wasfencedbefore recovery,retiredafter it andusableat the end; PR 1 endedauth_failedwith exactly one recovery receipt; PR 2 was reviewed in exactly four sessions and published once.go -C reviewer vet ./...passes;go test -race ./...passes (reviewer package 261.902 s) on the same tree.GIT-STARTUP.md("Stopped provider recovery"),NATIVE-VM.md("Operator recovery of a stopped attempt") andE2E-QUALIFICATION.md(recovery steps, record inventory rows,Measured at d1b5d72).Limits
An implementation subagent (Claude Opus 5.5) wrote this. The orchestrating agent reviewed the service code and verified the race logs. If Codex review hangs, the fallback is a separate Claude Opus 5.5 instance (same provider, because of Codex quota).
🤖 Generated with Claude Code