Skip to content

Recover a stopped provider attempt through an explicit operator action - #251

Merged
HemSoft merged 19 commits into
mainfrom
feature/issue-174-provider-wedge-recovery
Oct 23, 2026
Merged

HemSoft merged 19 commits into
mainfrom
feature/issue-174-provider-wedge-recovery

Conversation

@HemSoft

@HemSoft HemSoft commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #174. Based on main (which includes #250's end-to-end qualification and the fixes #252, #253 and #254 it surfaced). It fixes the liveness gap #250's end-to-end qualification found. Closing the service during a provider session left the job leased and the Codex auth fenced, and after a restart nothing could ever be reviewed again.

Design

The recovery is an explicit operator action. It never runs automatically: there is no timer or polling, and no GitHub, PR or model input can reach it.

  • GitReviewerService.ProviderRecoveryCandidates(ctx): a read-only, nonsecret list of leased jobs with their attempt, provider, round, session hash, whether stopped evidence is recorded, and matching launch directories.
  • GitReviewerService.RecoverStoppedProvider(ctx, ProviderRecoveryRequest{JobID, AttemptID, LaunchID, RetireUnreturnedAuth}).

Recovery refuses without any durable change unless all of the following hold:

  • the service is started but has never run, is not closing and has no active controller work (otherwise ErrLiveBlocked);
  • the job's lease, outstanding attempt ID and recorded stopped provider evidence all match;
  • the original launch's unit journal shows a native unit that was captured, aborted and stopped, and its private stopped audit binds the same session, manifest and evidence;
  • a fresh whole-unit stop proof succeeds inside every auth change.

Auth then follows the existing stopped-worker rules:

Auth state for the launch Action Job ends as
Never exported Commit the unchanged seed provider_unavailable
Return already durably saved Finish it provider_unavailable
Exported but return lost Retire it, only with RetireUnreturnedAuth; otherwise refuse auth_failed
Mid-retirement Finish retiring auth_failed

An exported seed is never reused without its durable return.

The lease is released in one store transaction under the same lease token, which rechecks the attempt, provider, round and evidence digest and appends a ProviderRecoveryReceipt. The session stays charged, no artifact is created and the job is never marked reviewed; only the existing ResumeUnavailable can requeue it. Without an operator call, behaviour is unchanged.

Review fixes

  • Each native launch's native-audit.json is version 2 and records the durable attempt ID; recovery accepts only a version 2 receipt naming the job's outstanding attempt, so an older launch with the same session hash cannot be recovered (Codex P1).
  • An interrupted commit of an unexported lease reports unexported, not returned, in the result and the durable receipt (Codex P2).
  • The reported and recorded job state is the final state after the PR fence (cancelled/stale), not a pre-fence state (Codex P2).
  • The close-provider-recover E2E case follows every rule Qualify the reviewer end to end on a real host with fakes #250's qualification gained in its review rounds: production readers and validators for every durable record, exact attempt and request sets, the recovered job equal to the retained job but for its released fields and one recovery receipt, the exact operator candidate view, expected jobs from harness inputs, exact per-role GitHub route sets, and exact auth dispositions (fenced → retired → usable).

Validation (exact head d1b5d72, measured; docs-only ec737a3 on top)

  • Unit tests (git_service_recovery_linux_test.go, fake manager, kernel and auth): the retire-and-release path; finishing a durable return; an unexported lease, including an interrupted commit; 13+ refusal cases each leaving lease and fence untouched, including another attempt's launch and a version 1 receipt; refusal while running or closing; fenced-job final state; the store release binding attempt, evidence and token.
  • End to end on mini (images, stub and reviewer.test built from d1b5d72, VCS-stamped; one VM at a time behind a ≥2.5 GB available-memory gate; release hash 1ec49cbb906e):
Case Seconds Checks Source / native units Result
review 22.5 52 1 / 4 pass
close-publish 22.5 59 1 / 4 pass
close-capture 21.4 64 1 / 4 pass
isolation 42.7 91 2 / 8 pass
close-provider + close-provider-reopen 12.1 + 38.6 34 + 29 1 / 1, 0 / 0 pass
close-provider + close-provider-recover 12.1 + 21.6 34 + 101 1 / 1, 1 / 4 pass

In close-provider-recover, Codex auth was fenced before recovery, retired after it and usable at the end; PR 1 ended auth_failed with exactly one recovery receipt; PR 2 was reviewed in exactly four sessions and published once.

  • Vet and race: go -C reviewer vet ./... passes; go test -race ./... passes (reviewer package 261.902 s) on the same tree.
  • Docs: GIT-STARTUP.md ("Stopped provider recovery"), NATIVE-VM.md ("Operator recovery of a stopped attempt") and E2E-QUALIFICATION.md (recovery steps, record inventory rows, Measured at d1b5d72).

Limits

  • An attempt with no stopped audit stays fenced and cannot be recovered. Examples: a host process killed mid-session, or a stop that could not be proved.
  • Provisioning new provider auth after retirement is an operator step outside the reviewer.
  • Recovery runs only in a freshly started process.
  • A grace period at shutdown was not implemented.

An implementation subagent (Claude Opus 5.5) wrote this. The orchestrating agent reviewed the service code and verified the race logs. If Codex review hangs, the fallback is a separate Claude Opus 5.5 instance (same provider, because of Codex quota).

🤖 Generated with Claude Code

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-23T22:31:12.127961Z ec737a3 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@HemSoft

HemSoft commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

Restacked at 34c9d96 on the rebased #250 (f0a4bc2), which now sits on #252. No change to this branch's own commits.

With source and native images built from 34c9d96: close-provider then close-provider-recover pass (40 checks, 0 failures). Vet and go test -race ./... (236.969 s) pass.

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 34c9d961fc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread reviewer/git_service_recovery_linux.go Resolved
Comment thread reviewer/git_service_recovery_linux.go Resolved
@HemSoft
HemSoft force-pushed the feature/issue-174-provider-wedge-recovery branch from 34c9d96 to d92ee65 Compare October 23, 2026 06:17
@HemSoft

HemSoft commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: d92ee65f96

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@HemSoft
HemSoft force-pushed the feature/issue-174-e2e-qualification branch 2 times, most recently from 6be4696 to 08fd121 Compare October 23, 2026 17:37
@HemSoft
HemSoft changed the base branch from feature/issue-174-e2e-qualification to main October 23, 2026 19:05
HemSoft and others added 15 commits October 23, 2026 15:17
Refs #174

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #174

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #174

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #174

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #174

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #174

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A resumed job's retry can repeat the session hash and stopped provider
evidence, so an earlier attempt's complete-looking launch under the same
provider root passed recovery and released the current lease in the
no-auth-journal case. DurableReviewer now passes the attempt ID to the
native reviewer, the version 2 stopped audit receipt records it, and
recovery and the candidate scan accept only a receipt bound to the
outstanding attempt. Receipts without the ID fail closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A recovery retried after Complete advanced a never-exported lease to
committing or committed reported returned auth, and the durable recovery
receipt kept that false credential history. Settlement now splits the
commit stages by transfer stage: an empty transfer is unexported and a
returned transfer is returned, each refused if the stopped receipt
contradicts it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A cancellation or replacement can fence a job while its stopped attempt
still holds the lease. Recovery reported and recorded auth_failed or
provider_unavailable although the release transaction then restored
cancelled or stale. The store release now returns the job's final state
after the PR fence, and both the result and the recovery receipt use it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
close-provider-recover no longer compiled after the rebase onto the
tightened harness, and it had no declared end state. It now expects
one source unit and four native units from PR 2's review, both auth
stores usable at the end, and two owner journal records: PR 1's
retained attempt and PR 2's, each with one staged export. A sample of
the unit observer and the launch directories after recovery proves
that startup and recovery started no unit. The verdict unit test
covers the attached recovery end state and the ways it can drift.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebased harness still accepted only version 1 stopped audit
receipts, but this branch writes version 2 with the durable attempt ID,
so every real launch would have counted as unaudited. The projection
now requires version 2 and a well-formed attempt ID, and the session
verdict binds each launch to its job: a completed launch must name the
attempt the job recorded for that artifact's session and repeat its
provider evidence, and an interrupted launch must name a recorded
attempt of its own session that no artifact used. close-provider,
close-provider-reopen and close-provider-recover also check that the
interrupted launch names the leased attempt, and recovery judges PR 2's
launches together with PR 1's recovered job. The pre-rebase recovery
runs are marked as history: their receipts are version 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the rebase onto the tightened qualification harness, an attached
root no longer treats its native launches as pre-existing by default, so
close-provider-recover would have counted the interrupted launch as its
own. It now proves the retained launch with e2eRetainedLaunches before
excluding it, as close-provider-reopen does. That rule now also binds the
version 2 receipt: it must name the leased attempt's durable ID and repeat
the attempt's recorded stopped evidence, so a version 1 receipt, which
names no attempt, is refused. The reopen case's separate attempt check is
folded into it.

close-provider-recover also checks exact requests per PR: PR 1 retains
only its admitted opened request when attached, after recovery (which
releases the job but not its request) and at the end; PR 2 ends with only
the owner comment request, admitted as its reviewed job's request; no
other PR has one. The final projection is recorded as Requests. Store
close and evidence replacement already run through the shared paths.

The qualification document describes both rules for the recovery case,
notes that the e144bc5 measurement predates recovery and version 2
receipts, and adds the pending measurement section its history links to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
close-provider-recover now takes its retained job through e2eAttachedJob,
as close-provider-reopen does, so a root retained by another revision's
close-provider is refused before recovery; the job must also still have an
outstanding attempt. Its request checks use the shared helpers: PR 1 is
judged with e2eOpenedRequestVerdicts against the leased job when attached
and after recovery, and the end state with the new
e2eRecoveredRequestVerdicts, whose job request IDs go through e2eRequestOf
so a missing job or a zero request ID matches nothing. The after-recovery
and final projections are recorded as Requests through checkRequests.

TestE2ERetainedLaunches projects the retained launch under the release
policy and session limits from an interrupted receipt that passes
production validation, and refuses one that fails it even when the job
records the same evidence. TestE2ERequestVerdicts covers the recovered
helper with missing jobs and zero request IDs.

The qualification document applies the attached-release rule to recovery,
states the source export binding for its two records, and keeps the
unmeasured head section, noting the e144bc5 run predates operator recovery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the restack onto #250's round 7 harness, close-provider-recover
still called the removed assertReviewed and assertNativeSessions helpers
and the two-argument e2eAttachedJob, so the package did not build. It now
judges its end state with the completed-case rules, with PR 1 recovered
instead of reviewed:

- Exactly one job per PR, no PR 1 intent, one PR 2 intent and no other
  write, at attach, after recovery and at the end.
- PR 1's job is the retained job itself, auth_failed without lease or
  outstanding attempt, charged for its one session with no artifact or
  source access, recording only the leased attempt with the interrupted
  launch's evidence and one recovery receipt bound to that attempt, round,
  launch and evidence digest with the auth retired. No PR 1 POST or review.
- PR 2 passes e2eReviewedVerdicts: exact R1/R2 rounds, backend checks,
  intent binding and the remote review matched through E2ERemoteReview.
- Two source records, PR 1's retained one and PR 2's, each bound to its own
  job's snapshot and the current release and runtime hashes.
- The interrupted launch is still present with its receipt unchanged and
  names PR 1's recorded attempt; PR 2's four launches name their artifacts'
  attempts; nothing else launched.
- PR 2's acquisition is the only upstream exchange (checkUploads).

The attached job goes through e2eAttachedJob with PR 1's fake upstream
snapshot. e2eDurableCountVerdicts states exact per-PR counts and shares
its counting with e2eDurableSetVerdicts; e2eReviewFixture parametrizes the
completed fixture by PR for e2eRecoveredFixture and TestE2ERecoveredVerdicts,
which names the verdict each edit must fail. The final-verdict recovery
fixture's native units carry the bound audit round 7 requires.

The qualification document states the recovered end-state rules and keeps
the head section unmeasured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the restack onto #250's round 8 harness, close-provider-recover's
end state is judged by the round 8 rules where they apply:

- PR 2's four launches and the interrupted one are projected from every
  entry of the attached native roots as pinned native-vm payloads with
  the release's stub and manifest hashes, and their version 2 receipts
  must repeat that manifest hash. Any other entry fails the launch
  verdict.
- PR 2's sessions must meet the fixture exchange contract, with served
  objects equal to the fake upstream's bytes at PR 2's head. The
  recovery process's backends must have checked exactly PR 2's artifact
  sessions, none held, so PR 1's interrupted session never runs again;
  its retained receipt must count exactly its one held inference,
  forwarded and never answered.
- PR 2's source attempt, the only one this process acquired, must pass
  the lifecycle order from the journal sampled at its upload-pack. PR 1's
  retained attempt, whose order close-provider judged, is judged by its
  final record only. Both records must carry the release's stub hash.

The review fixture now serves PR n's own file, the recovered fixture
carries the held receipt's counters, the retained record's stub hash and
invocation, and TestE2ERecoveredVerdicts names the target check of each
new refusal. The qualification document states the rules for recovery,
points the unmeasured head section at the b876290 run, and notes that
run predates operator recovery and version 2 receipts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HemSoft and others added 4 commits October 23, 2026 15:17
Rebased onto the qualification branch's round 9, close-provider-recover
now meets those rules where they apply:

- PR 2's stored reports and confirmed document are judged against the
  canned backend reports and production's renderer through the shared
  reviewed-PR verdicts; PR 1 has no artifact and no document. The
  recovered-verdict test refuses an altered PR 2 report or a document the
  intent and remote agree on but the canned reports do not produce.
- The quiesced unit sampler applies unchanged: the case samples the
  observer only before finish, whose stop joins the periodic sampler.
- The settled re-read covers the recovery process's own fake servers,
  which hold only PR 2's acquisition, sessions and review POST. A new test
  refuses a late rerun of PR 1's interrupted session, a late PR 1 review
  or refetch, and late PR 2 requests.
- Auth dispositions are exact. Codex is fenced before recovery and after
  the refused attempts. An unfenced store with no state is now "retired",
  exactly what production's AuthProfile.Retire leaves (no state, no
  lease), and recovery must leave Codex retired with Claude usable. The
  re-provisioned seed is a fresh synthetic profile carrying the original
  synthetic tokens, the only credential the canned backend accepts; the
  store must hold exactly those bytes, and finish requires both stores
  usable. An empty store was previously judged invalid.

The documentation keeps the daac99c measurement, noting it predates
operator recovery and version 2 receipts, and points the not yet measured
placeholder at it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebased onto the qualification branch's round 10, close-provider-recover
now meets those rules where they apply. The rebase itself carried the
version 2 receipts through: the audit mirror E2EReadNativeAudit reads
persistAudit's version 2 encoding with AttemptID, the launch receipts
test confirms that production writes it naming the phase's attempt, and
every launch verdict binds the named attempt.

- PR 1's recovered job must equal the retained job with only its state,
  lease, deadline, reservation and pending charge released and exactly
  one recovery receipt appended, so its attempt set is exactly the leased
  attempt with no source attempt. The receipt must carry a recovery time.
- The interrupted launch's audit must still bind the recovered attempt
  after recovery and at the end: its derived request, the zero report, no
  source receipt, stage result and the auth-return failure.
- PR 2's job holds exactly its four artifact attempts and no recovery
  receipt, and no completed job of any case may carry one. The end state
  also applies the shared attempt-set verdict to both jobs.
- The operator candidate view must equal the leased attempt's candidate
  field for field.
- The recovered fixture is built from the round 10 retained and review
  fixtures, with this boot's source records and the job's packet hash on
  each export. The recovered-verdict test names the target check of each
  new refusal, and the final-verdict recovery fixture carries whole stops.

The record inventory lists the recovered job, the recovery receipt, the
candidate view and the auth stores. The not yet measured section names
the round 10 rules and points at the 89382c7 run, which predates operator
recovery and version 2 receipts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
close-provider-recover no longer compiled after the rebase onto #250's
round 11 and judged neither job against the harness's inputs. The
attached job is now the one e2eAttachedJob derives under the harness
configuration. PR 1's recovered job must be exactly the job admission
derives from its retained open request, the fake upstream's snapshot,
the configuration at version 1 and the current release, held in the
retained job with only the released fields and one recovery receipt
changed. PR 2's job is judged by e2eReviewedVerdicts against the job
derived from the fake upstream's owner comment, and its retained request
must carry that comment's pull and comment IDs. Sessions, source
records, launches and the one extra interrupted Codex R1 session all
derive from the expected jobs, and PR 2's remote review is bound to the
fake upstream's head.

Recovery uses no GitHub route, so the recover process must serve exactly
the publishing per-role route set, matching the tally of the last real
run at bb6d865; close-provider before it keeps the qualified set. Auth
keeps retired (no state, lease, profile or pending journal) distinct from
unavailable, with fenced only on ErrAuthRecoveryRequired.

The fixtures gain per-PR origins and consistently rebuilt variants
(e2eRetainedFixtureOf, e2eRecoveredFixtureOf), and the verdict tests
refuse wrong origins, configurations, IDs, requesters, retained requests,
remote review commits and route sets for the recovery case. The record
inventory and recovery steps describe the round 11 rules, and the
Not yet measured placeholder links the 13251e3 run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@HemSoft
HemSoft force-pushed the feature/issue-174-provider-wedge-recovery branch from d92ee65 to ec737a3 Compare October 23, 2026 22:29
@HemSoft

HemSoft commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto main after #250 merged (main also includes #249, #252, #253, #254); now ec737a3, base main. The earlier clean review at d92ee65 predates the restacks: since then this branch has absorbed #250's eleven review rounds into the close-provider-recover case (production readers and validators, exact sets, expected jobs, route sets, auth dispositions) without changing the recovery service code reviewed then.

Validation at d1b5d72 (docs-only ec737a3 on top): every E2E case passes, including close-provider followed by close-provider-recover (101 checks); vet and go test -race ./... (261.902 s) pass. Details are in the PR description and E2E-QUALIFICATION.md ("Measured at d1b5d72").

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: ec737a37ec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant