Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 31 additions & 5 deletions reviewer/CONTROLLER.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,10 @@ sessions begin, including jobs queued on earlier dates.

Only one lease may exist. Expiration does not authorize stealing it. A separately
qualified supervisor must prove the worker has stopped before `ReleaseStopped`.
Service shutdown gives a provider session that already exported auth a bounded
grace to stop and return it before cancellation
([POLLING.md](POLLING.md#shutdown-and-recovery)). The lease and stop rules are
unchanged, and a session that does not return in time keeps its lease.
Drivers supplied to `RunStopped` must return only after worker termination.

Broker-equipped drivers also return trusted per-attempt source receipts, kept
Expand Down Expand Up @@ -140,6 +144,28 @@ budget. It does no inference, is never called by configuration comments, and
rejects expired/exhausted, cancelled, superseded or paused jobs. Unknown provider
work remains charged; this recovery cannot create additional session capacity.

A job's captured session budget equals its schedule: one session per provider
and round. Every charged session without a validated artifact (an
interrupted, failed or shutdown-ended phase) therefore leaves the schedule one
session short, and the job can never complete. `sessionBudgetFits` states that
rule from the job's configuration and checkpoint exactly as the engine plans
phases. Such a job spends no further provider session:

- The polling controller ends a retained queued or quota-waiting job that does
not fit as `incomplete_expired` before it restores evidence, builds a
runtime or claims it.
- The claim ends it the same way before any lease, quota wait or session, so
a direct `RunStopped` caller cannot spend one either.
- The engine refuses to start a phase of a schedule that does not fit
(`ErrSessionBudgetExhausted`).
- `ResumeUnavailable` refuses to requeue it with `ErrSessionBudgetExhausted`
and leaves its state unchanged. This covers an `auth_failed` or
`provider_unavailable` job after a charged failure and the job that
[operator recovery](GIT-STARTUP.md#stopped-provider-recovery) releases.

A before-work refund keeps the budget whole, so `quota_wait` and other
refunded attempts still fit.

A lease claim returns its captured job/checkpoint from the claiming transaction.
There is no fallible post-commit read that can hide the committed lease nonce.
`RunStopped` uses that atomic result directly. Superseded/cancelled generations
Expand Down Expand Up @@ -173,15 +199,15 @@ published states and their leading words are:
| Job state | Status |
| --- | --- |
| `quota_wait` | INCOMPLETE: daily quota exhausted; resumes by itself until the configured quota wait expires |
| `auth_failed` | INCOMPLETE: provider authentication unavailable; stopped until an operator repairs it and resumes the job |
| `provider_unavailable` | INCOMPLETE: a selected provider is unavailable; stopped until an operator resumes the job |
| `auth_failed` | INCOMPLETE: provider authentication unavailable; stopped until an operator repairs it and resumes the job, or, when its budget no longer fits, cannot be resumed |
| `provider_unavailable` | INCOMPLETE: a selected provider is unavailable; stopped until an operator resumes the job, or, when its budget no longer fits, cannot be resumed |
| `invalid_output` | INCOMPLETE: provider output failed validation; ended without a review |
| `incomplete_expired` | INCOMPLETE: expired before every required phase completed |
| `incomplete_expired` | INCOMPLETE: expired before every required phase completed, or the captured session budget is exhausted, naming the charged sessions without a result |
| `cancelled` | CANCELLED: the request was cancelled before the job completed |
| `capture_refused` | INCOMPLETE: missing history, ambiguous merge base or over the 128-file inventory limit; the request ended without a review |

The text is fixed controller wording keyed by the durable state and the
captured configuration. It passes through `safeText` and known-secret
The text is fixed controller wording keyed by the durable state, the captured
configuration and the checkpoint's session counts. It passes through `safeText` and known-secret
redaction like every document, names the reviewed revisions and the completed
phase count, and never includes a finding. `queued` and `running` are
transient. `blocked` is the operator qualification gate, not a review
Expand Down
54 changes: 39 additions & 15 deletions reviewer/E2E-QUALIFICATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,19 +94,38 @@ ledger, and a restore probe.
| `review` | PR 1 automatic request through capture, four sessions and one confirmed COMMENT. | 1 / 4 | 1 | both usable |
| `isolation` | PR 1 (automatic) and PR 2 (owner comment) pending together. | 2 / 8 | 2 | both usable |
| `close-capture` | Close the whole service after capture and admission, inside the first session's host check, before any VM launch. Reopen and finish. | 1 / 4 | 1 | both usable |
| `close-provider` | Close the whole service while the first Codex session waits for a withheld inference response. | 1 / 1 | lifecycle order only | Codex fenced, Claude usable |
| `close-provider` | Close the whole service while the first Codex session waits for a withheld inference response. With the shutdown grace, reopen: the requeued job ends incomplete and publishes one status. | 1 / 1 | 1 (grace) or lifecycle order only (fenced) | both usable (grace), or Codex fenced and Claude usable |
| `close-provider-reopen` | A second process attached to the root retained by `close-provider` (`SFL_E2E_ATTACH`), as a restarted service. | 0 / 0 | 1 (retained) | Codex fenced, Claude usable |
| `close-provider-recover` | Instead of `close-provider-reopen`: a restarted process recovers the retained attempt through the operator API, publishes PR 1's `auth_failed` status, then reviews and publishes the next request. | 1 / 4 | 2 (1 retained) | both usable (Codex re-provisioned) |
| `close-publish` | The fake GitHub commits the review POST but withholds its response; close the whole service; reopen and reconcile. | 1 / 4 | 1 | both usable |

If `close-provider`'s `Close` proves completion instead of reporting
`ErrWorkerStopPending`, the case reopens and expects one source unit, one
native unit for the interrupted launch plus one per artifact of the reopened
job, and both auth stores usable. The job must record the interrupted
launch's complete stopped audit evidence. After the reopen it runs the same
completed-review checks as the other successful cases (see Evidence bindings),
with the interrupted launch, stopped, audited and incomplete, as the only
other launch. No host run has taken this branch.
Since the [shutdown grace](GIT-STARTUP.md#shutdown-grace) ([#264](https://github.com/hemsoft-dev/set-it-free-loop/issues/264)),
the expected `close-provider` branch is a `Close` that proves completion.
`Close` revokes the withheld session's provider intake, the guest's client
proxy stops the CLI, and the guest returns auth in its result record within
the ten-second grace. The case then requires a queued, unleased job with one
charged session and no artifact, and Codex auth usable. The stopped audit must
record the auth as exported and returned, end at the result record's
validation and fail as `cancelled`. The job must record that launch as its
attempt, with its complete stopped evidence and the source receipt its result
returned. The case then reopens. The requeued job keeps that charged session
without a result, so its captured four-session budget can no longer finish
the schedule. The reopened controller ends it `incomplete_expired` before
restoring evidence or starting a session, unleased, with one charged session
and no artifact. It publishes exactly one status for PR 1, which must be
confirmed, bound to the job and checkpoint, carry the document production
renders for that state and checkpoint (the exhausted-budget wording) and
match its one inline-free remote COMMENT on the head (`e2eStatusVerdicts`).
There is no review intent and no other write. The case therefore expects one
source unit, one native unit (the interrupted launch, the only launch), both
auth stores usable, and the publishing route set: the qualified routes plus
the publisher's review POST and receipt read. No host run has taken this
branch. Its earlier forms expected a published review, then three reruns
that expired, which spent sessions on a certain incomplete.

If the guest does not return within the grace, `Close` cancels as before and
the case takes the fenced branch below. Only that branch retains a root for
`close-provider-reopen` and `close-provider-recover`.

Every case also observes `systemctl --user list-units 'sfl-alpha-*'` every 200
ms, excluding units that existed before the run. At the end it first stops
Expand Down Expand Up @@ -533,10 +552,11 @@ flock -o ~/.cache/sfl-vm.lock env GOMAXPROCS=2 \
timeout 600 nice -n 10 "$OUT/reviewer.test" -test.run '^TestE2EQualification$' -test.v -test.timeout 590s
```

For `close-provider`, run `close-provider-reopen` next in a new process with
`SFL_E2E_ATTACH` set to the `Root` recorded in `close-provider.json`, inside
the same lock hold. State lives under a fresh `/tmp/sfl-e2e-<nonce>` root,
which a passing run removes. `SFL_E2E_KEEP_FAILED=1` retains a failed root.
When `close-provider` took the fenced branch and retained its root, run
`close-provider-reopen` next in a new process with `SFL_E2E_ATTACH` set to the
`Root` recorded in `close-provider.json`, inside the same lock hold. State
lives under a fresh `/tmp/sfl-e2e-<nonce>` root, which a passing run removes.
`SFL_E2E_KEEP_FAILED=1` retains a failed root.

## Measured at `13251e3`

Expand Down Expand Up @@ -716,7 +736,8 @@ and return auth. Both change security-relevant recovery behavior, so neither
was implemented in the commits measured above. The operator path was added
later and is qualified in
[Operator recovery after a provider-session close](#operator-recovery-after-a-provider-session-close).
The bounded shutdown grace is not implemented.
The bounded [shutdown grace](GIT-STARTUP.md#shutdown-grace) was added after
that, and its `close-provider` branch has not run on a host.

A shutdown before a session's work, while the host check runs, is
different. The native reviewer reports a trusted before-work failure, the
Expand Down Expand Up @@ -879,10 +900,13 @@ now waits for admission.
[GIT-VM-SOURCE.md](GIT-VM-SOURCE.md), and native supervisor process loss by
[NATIVE-VM-CRASH.md](NATIVE-VM-CRASH.md). A `SIGKILL` between session
reservation and stopped receipt would leave the same durable fence as
`close-provider`, without the in-process stop proof.
`close-provider`, without the in-process stop proof. The shutdown grace
applies only to `Close`; a process kill gets none.
- Operator recovery covers a provider attempt stopped with a durable
stopped audit receipt (see the recovery section). An attempt without that
receipt, such as after a host process kill mid-session, stays fenced.
Whether it may ever be recovered, and on what alternative evidence, is an
open owner decision in [#264](https://github.com/hemsoft-dev/set-it-free-loop/issues/264).
- Isolation is between sequential jobs because admission is serial. Both jobs
share the per-provider auth stores by design; each session exports and
returns them, and the case checks returns but cannot observe a concurrent
Expand Down
71 changes: 62 additions & 9 deletions reviewer/GIT-STARTUP.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,14 +128,61 @@ Operators must retain and investigate such artifacts; automatic crash-artifact
collection is not implemented. This bound and retained artifacts belong in the
full host disk/resource qualification.

## Shutdown grace

`Close` gives an in-flight native provider session a bounded chance to return
its auth before the session is cancelled
([#264](https://github.com/hemsoft-dev/set-it-free-loop/issues/264)).
The controller's active invocation carries a private shutdown drain. A native launch holds it
immediately before exporting auth, and no launch exports once the drain has
begun. When `Close` begins the drain:

- Without a held export, `Close` cancels at once, exactly as before. This
covers capture, admission, restoration, the host check, boot before export,
publication and work outside a controller invocation.
- With a held export, the launch closes its provider intake: the private
provider stream and any in-flight request close. The guest's client proxy
already stops its owned CLI when a private exchange fails, so the client
ends, and guest PID1 stops its roles and returns auth and the source audit
in its ordinary result record. The first `Close` waits for the invocation
to end for at most `providerShutdownGrace` (ten seconds) or its own earlier
deadline, then cancels as before. The grace is granted once; a retried
`Close` cancels at once. The session's own deadline is never extended.

The stop proof and the auth rules do not change. The host still reads the
result record only under the session deadline and the later cancellation,
proves whole-unit stop before any auth step, and commits auth only through the
durable stopped return. A refresh-uncertain return is still fenced. A session
that fails after the drain reports `cancelled`, so the stopped job is requeued
with its artifacts and consumed budget; a latched route or response refusal,
or exhaustion, keeps its own classification. A session that completes within
the grace keeps its artifact, and the engine starts no further phase. A
requeued job whose interrupted session left no artifact can no longer finish
its schedule, so the next controller ends it `incomplete_expired` without
another session and publishes one INCOMPLETE status naming the exhausted
budget ([CONTROLLER.md](CONTROLLER.md)).

The ten seconds cover the guest's bounded cleanup after its client ends (at
most twelve seconds, normally well under one) and its result record. If the
grace expires, cancellation interrupts the result read and the native host's
fifteen-second stop path runs, so grace and stop stay inside the 30-second
`Close` ceiling. The deferred `Close` in `PollingController.Run` has the grace
plus its earlier ten seconds. When the guest does not return in time, the
outcome is the one described below: the unit is still proved stopped, the
stopped receipt records the unreturned export, the lease stays and `Close`
reports `ErrWorkerStopPending`.

The grace applies only to `Close`. A host process kill gets none.

## Stopped provider recovery

A shutdown or failure during a native provider session can leave the durable
job leased: the unit is proven stopped, but the guest never returned its
result record, so the source audit is missing and an exported provider auth
stays fenced. The controller then admits nothing, and `Close` keeps reporting
`ErrWorkerStopPending`. Nothing releases that lease automatically, on a timer
or from polling, GitHub or model input.
A failure during a native provider session, a shutdown whose session does not
return within the [grace](#shutdown-grace), or a host process kill can leave
the durable job leased: the unit is proven stopped, but the guest never
returned its result record, so the source audit is missing and an exported
provider auth stays fenced. The controller then admits nothing, and `Close`
keeps reporting `ErrWorkerStopPending`. Nothing releases that lease
automatically, on a timer or from polling, GitHub or model input.

`RecoverStoppedProvider` is the explicit operator path. Call it after a
process restart, on a service that was opened but not yet refreshed or run:
Expand Down Expand Up @@ -165,7 +212,10 @@ It refuses, without changing the job or auth, unless all of these hold:
and evidence, so the attempt ID is what rejects an earlier attempt's
launch. A receipt without an attempt ID refuses. A host process kill
before that receipt was written therefore cannot be recovered through
this path.
this path. Whether such a lease may ever be recovered, and on what
alternative evidence, is an open owner decision in
[#264](https://github.com/hemsoft-dev/set-it-free-loop/issues/264).
Until it is decided, that lease stays fenced.
- The auth journal of that provider either belongs to this launch or does
not exist, and is consistent with the receipt (`ErrAuthRecoveryRequired`).

Expand All @@ -181,8 +231,11 @@ becomes `auth_failed` when the exported auth was lost and retired. It becomes
that a cancellation or newer request fenced while it held the lease stays
`cancelled` or `stale`; the result and the receipt report that final state.
None of these states is a review. Only an explicit `ResumeUnavailable` can
requeue the same unfenced request, within its remaining session budget.
Other requests proceed immediately.
requeue the same unfenced request, and only while its remaining session
budget can still finish the schedule. The recovered session stays charged
without a result, so for a job with no spare session, which the default
configuration is, it refuses with `ErrSessionBudgetExhausted` and the status
says the job cannot be resumed. Other requests proceed immediately.

A retired provider has no canonical auth. Its sessions fail before work as
`auth_failed` until the operator establishes new auth outside the reviewer.
Expand Down
18 changes: 18 additions & 0 deletions reviewer/NATIVE-VM.md
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,24 @@ then stops the entire unit and keeps uncertain exported auth held. A blocked
source read exposed a seventy-second wait for the original deadline; a stopped
source channel alone did not wake the host's guest-result read.

A controller shutdown drains a launch that already exported auth before it
cancels it (see the [shutdown grace](GIT-STARTUP.md#shutdown-grace)). The
launch holds the drain from just before export until its stopped receipt is
written, and exports nothing once the drain has begun. The drain closes the
host provider bridge's HTTP server, which closes the private provider stream
and any in-flight request; the guard records an aborted request as failed.
The guest protocol is unchanged. Its client proxy already stops the owned CLI
when a private exchange fails (`provider-transport`), and guest PID1 then
returns auth and the source audit in its finished record. The host reads that
record under the session deadline until the controller's grace expires and
cancels it. Whole-unit stop, auth import, the conservative refresh rule and
the stopped receipt are unchanged; the receipt of a drained session that
failed reports `cancelled`. Offline tests cover the host bridge, the guest
proxy handler and the host result handling with fakes. Whether the guest sees
the closed virtio port as end of stream promptly is a host property they do
not exercise; if it does not, the grace expires into the cancellation path
above.

## Operator recovery of a stopped attempt

`GitReviewerService.RecoverStoppedProvider` (see
Expand Down
Loading