Skip to content

Give an exported provider session a bounded shutdown grace and end unfinishable jobs - #272

Merged
HemSoft merged 4 commits into
mainfrom
feature/issue-264-shutdown-grace
Oct 24, 2026
Merged

HemSoft merged 4 commits into
mainfrom
feature/issue-264-shutdown-grace

Conversation

@HemSoft

@HemSoft HemSoft commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator

Part of #264 (the bounded shutdown grace) and #176 (acceptance item 2). Recovery of a lease whose stopped receipt is missing is not implemented: it needs the owner decision recorded in #264 and stays fenced as before.

Shutdown grace

  • Each active controller job carries a private shutdown drain in its context. A native session registers with it just before exporting auth; once Close starts the drain, no session exports.
  • With no exported auth (capture, admission, restore, host check, boot, publication), Close cancels at once, as before.
  • With exported auth, that session closes its host provider bridge: the private provider stream and any in-flight request close, and the guard records the aborted request as failed. The guest's client proxy already stops its CLI when a private exchange fails, then returns auth and the source audit in its normal result record — no guest change.
  • Bound: providerShutdownGrace is 10 s or the caller's earlier deadline, only for the first Close (a retry cancels at once), and never extends the session's own deadline. The guest's post-client cleanup is capped at 12 s and normally under 1 s; 10 s plus the host's 15 s stop path fits inside GitReviewerService.Close's 30 s ceiling. PollingController.Run's deferred Close budget becomes the grace plus 10 s.
  • Unchanged: the host still proves the whole unit stopped before any auth step and commits auth only through the durable stopped return; a refresh-uncertain return stays fenced. If the guest hangs, the grace expires and the existing path runs (stop proved, auth fenced, lease kept, ErrWorkerStopPending, recoverable through Recover a stopped provider attempt through an explicit operator action #251).
  • A session that fails after the drain reports cancelled, so the job requeues with its session charged; a session that completes during the grace keeps its artifact.

Session budget rule

A requeued job keeps its charged session, and in the default schedule the budget equals the phases, so it can never finish; before this change it reran three more real sessions and expired. sessionBudgetFits (engine.go) requires remaining sessions ≥ remaining phases. An unfinishable job is ended as incomplete_expired before restore, runtime build, claim, lease or any session (Store.expireUnfinishable, claimJob); Engine.Run and ResumeUnavailable refuse it with ErrSessionBudgetExhausted. Its status names the cause ("the captured session budget is exhausted…"), and auth_failed/provider_unavailable statuses of an unfinishable job say it cannot be resumed. Behaviour change: after any charged failure in the default four-session schedule, a job ends instead of being resumable — the resume could never complete. A fresh owner request still creates a new job. Before-work refunds keep the budget whole, so quota_wait and refunded attempts still fit.

Evidence

  • Controller tests with a fake timer (grace succeeds; completion during grace; guest hangs until the bound then stop-pending with no second grace; caller deadline bounds the wait; no grace outside provider sessions); native host tests (withheld inference ends, the real guest proxy stops its client, drained session commits returned auth as cancelled, refusals keep their class, a missing return stays fenced); budget tests (rule vs schedule, restarted controller ends the job with zero restores/builds/sessions and one INCOMPLETE status, unfinishable quota_wait ended at claim, ResumeUnavailable refusal). Fail-before: six grace behaviour tests and ten budget tests fail without the change.
  • Measured at 3dc6a4d (this branch on main with Record permanent capture refusals as coverage limits #270) on mini, from images, a source stub and reviewer.test built from that commit, one VM at a time behind a ≥2.5 GB available-memory gate: exhaustion 12/12; hostile 19 pass + the contained codex-metadata client failure; every E2E case passes. close-provider now takes the grace branch (58 checks, 7.6 s): after Close the job is queued and unleased with 1 consumed session, the interrupted audit shows auth exported and returned, both auth stores are usable, and on reopen the job ends incomplete_expired with no further session and exactly one confirmed INCOMPLETE status (1 source / 1 native unit). The same results held at 89c36a1 on the previous base.
  • go vet ./... and go test -race ./... (reviewer package 267.174 s) pass.

Coverage change

Because the guest now returns within the grace, the fenced branch is reached only when the guest hangs, so close-provider-reopen and close-provider-recover no longer run on a host. #251's recovery keeps its unit and fault coverage; restoring an E2E path to the fenced branch is #271.

Host-side behaviour only (service and reviewer.test); every image and binary was rebuilt from the release commit for the measurement.

🤖 Generated with Claude Code

HemSoft and others added 4 commits October 24, 2026 00:26
PollingController.Close now signals a private shutdown drain carried in the
active invocation's context. A native launch holds that drain just before it
exports auth, and no launch exports once the drain has begun. Only when a
launch holds an export does the first Close wait, at most ten seconds
(providerShutdownGrace) or its own earlier deadline, for the invocation to
end before cancelling it as before. Every other phase is still cancelled at
once, and a retried Close gets no second grace.

During the grace the held launch closes its provider bridge, which closes the
private provider stream and any in-flight request. The guest's existing client
proxy stops its CLI on the failed exchange, and guest PID1 returns auth and the
source audit in its ordinary result record. The host still proves whole-unit
stop before any auth step and commits auth only through the durable stopped
return. A failure after the drain reports cancelled, unless the guard latched a
refusal or exhaustion, so the stopped job is requeued with its consumed budget.
The engine starts no further phase once the drain began. A guest that does
not return in time takes the existing stop-pending, fenced path.

PollingController.Run's deferred Close budget becomes the grace plus its
earlier ten seconds, inside GitReviewerService.Close's 30-second ceiling.

The E2E close-provider branch where Close proves completion now expects the
requeued job to rerun three phases and expire its captured budget, since the
interrupted session stays charged, instead of a published review.

Recovery of a lease whose stopped receipt is missing is unchanged and stays
fenced pending the owner decision in #264.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GIT-STARTUP.md gains a shutdown grace section: what holds the drain, what
Close waits for and for how long, why ten seconds fits the existing guest
cleanup, native stop path and 30-second Close ceiling, and that stop proof
and auth rules are unchanged. NATIVE-VM.md, POLLING.md and CONTROLLER.md
describe the host revocation and controller behaviour.

E2E-QUALIFICATION.md now names the Close-proves-completion branch of
close-provider as the expected one, with the requeued job expiring its
captured budget after three reruns, and keeps the fenced branch for a guest
that does not return in time. Recovery of a lease whose stopped receipt is
missing remains an open owner decision in #264 and stays fenced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…264)

A job's captured session budget equals its schedule, one session per
provider and round, so every charged session without a validated artifact
leaves it one session short. After a shutdown grace the requeued job kept
its interrupted session's charge and then reran three real sessions before
expiring. sessionBudgetFits now states the rule from the job's configuration
and checkpoint, as the engine plans phases, and such a job spends nothing:

- the polling controller ends a retained queued or quota-waiting job that
  does not fit as incomplete_expired before restoring evidence, building a
  runtime or claiming it;
- the claim ends it the same way before any lease, quota wait or session;
- the engine starts no phase of a schedule that does not fit
  (ErrSessionBudgetExhausted);
- ResumeUnavailable refuses to requeue it with ErrSessionBudgetExhausted,
  covering charged auth or provider failures and operator recovery.

Its status names the cause: an incomplete_expired status reports the
exhausted budget with its charged and resultless sessions, and an
auth_failed or provider_unavailable status no longer promises a resume.

The E2E close-provider branch where Close proves completion now expects the
reopened job to end incomplete with no further launch and exactly one
confirmed INCOMPLETE status matched to its remote COMMENT. Tests that relied
on resuming a short budget now require the refusal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ome (#264)

CONTROLLER.md states when a job's budget can no longer finish its schedule
and every point that ends or refuses it, and the status wording that names
the cause. GIT-STARTUP.md, POLLING.md and E2E-QUALIFICATION.md describe the
requeued job after a shutdown grace ending incomplete with one status, and
recovery's resume refusal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-24T04:51:44.709171Z 3dc6a4d Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@HemSoft

HemSoft commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 3dc6a4da9a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant