fix(prover-node): do not abort in-flight proving jobs on a clean shutdown - #24579
Merged
PhilWindle merged 5 commits intoJul 13, 2026
Merged
Conversation
AztecBot
added
ci-draft
AztecBot
force-pushed
the
cb/prover-node-keep-jobs-on-shutdown-v5
branch
from
July 7, 2026 11:07
640550b to
41a8f66
Compare
Base automatically changed from
cb/prover-broker-abort-reenqueue-v5
to
merge-train/spartan-v5
July 8, 2026 12:25
PhilWindle
force-pushed
the
cb/prover-node-keep-jobs-on-shutdown-v5
branch
from
July 8, 2026 14:56
2c5c77e to
676c432
Compare
PhilWindle
marked this pull request as ready for review
July 8, 2026 14:56
…down On a clean shutdown SessionManager cancels sessions with abortJobs=false, so the in-flight top-tree broker jobs are left in the broker for the restarted node to reuse instead of being aborted and re-proven. Reorg/supersede/deadline cancels still abort (their inputs are stale).
Adds a single-node/proving e2e test that gates an epoch's top-tree proving so its jobs sit unproven at a shared broker, cleanly stops the prover node, and restarts it against the same broker. Asserts the clean shutdown issues zero cancelProvingJob calls and leaves the jobs in-queue (not aborted), and that the restarted node resumes proving and lands the epoch on L1 by reusing those jobs. The broker is a test-owned object injected into both prover-node incarnations via ProverNodeDeps.broker (the production external-broker topology), so the in-flight jobs survive the restart. Threads proverNodeDeps through the createProverNode / createAndSyncProverNode test helpers to allow the injection.
PhilWindle
force-pushed
the
cb/prover-node-keep-jobs-on-shutdown-v5
branch
from
July 8, 2026 15:00
676c432 to
a53e218
Compare
Splits the prover-restart e2e into two cases (now a .parallel file): - in-flight top-tree jobs: gates the top tree so its jobs are in flight at a clean shutdown, and asserts they are preserved (zero cancels) and reused on restart. Discriminates the clean-shutdown abortJobs:false fix. - in-flight transaction proofs: starves agents from the start so real tx base-rollup proofs sit unproven at the broker, then asserts those non-parity proofs survive a clean shutdown and are re-requested/reused on restart rather than the leaf parity jobs the top-tree case would otherwise dominate.
The top-tree case previously only caught leaf parity jobs in flight, which are sub-tree proofs preserved by the pre-existing cancelJobsOnStop default. Adds a type-selective starve to the broker proxy (withhold specific job types from agents via allow-list exclusion so their dependencies still prove) and uses it to hold the checkpoint-root proofs (CHECKPOINT_ROOT_ROLLUP / CHECKPOINT_ROOT_SINGLE_BLOCK_ROLLUP) in flight. Those are enqueued by the top-tree orchestrator, so the case now exercises the abortJobs:false top-tree cancel path against real non-parity proofs.
fcarreiro
approved these changes
Jul 9, 2026
PhilWindle
added a commit
that referenced
this pull request
Jul 21, 2026
…down (#24579) v5 version of #24562 (based on `merge-train/spartan-v5`). Stacked on #24578 — review/merge that first. Note: this is a fresh implementation, not a port — the v5 prover-node architecture differs from `next` (it uses `EpochSession` / `TopTreeJob` rather than `EpochProvingJob` / `ProvingOrchestrator`), which is the structure actually running on testnet. ## Problem On a clean shutdown, `SessionManager.stop()` cancels every live session with reason `'prover-node stopping'`. That flows `EpochSession.cancel` → `TopTreeJob.cancel()` → `topTree.cancel({ abortJobs: true })` — the `abortJobs: true` was **hardcoded**, so a deploy/restart aborted the in-flight top-tree broker jobs. Aborting on a clean restart is wasteful: the proofs are still valid, agents are mid-flight, and a restarted node re-orchestrating the same epoch produces the same deterministically-hashed job ids, so it re-proves from scratch. ## Fix Thread an `abortJobs` decision from the cancel reason down to the broker: - `SessionManager.stop()` cancels with `{ abortJobs: false }` — a clean shutdown preserves the jobs. - `EpochSession.cancel(reason, { abortJobs })` forwards it to `TopTreeJob.cancel(abortJobs)` → `topTree.cancel({ abortJobs })`. - All other cancels (reorg / supersede / deadline) keep the default `abortJobs: true`, since their inputs are stale. Composes with #24578: a clean redeploy mid-epoch neither cancels the in-flight jobs nor (if one ever were aborted) poisons them, so the epoch keeps proving across the restart. ## Tests - `session-manager.test.ts`: `stop()` cancels every session with `abortJobs: false`. - `epoch-session.test.ts`: a normal cancel forwards `abortJobs: true` to the top-tree orchestrator; a clean-shutdown cancel forwards `abortJobs: false`. Note: not run locally in this session — the prover-node suite needs a full noir/wasm bootstrap that wasn't available here — so relied on CI. --- *Created by [claudebox](https://claudebox.work/v2/sessions/6fa5e242ed9ceae7) · group: `slackbot`* --------- Co-authored-by: Phil Windle <philip.windle@gmail.com> (cherry picked from commit c44e74e)
PhilWindle
added a commit
that referenced
this pull request
Jul 21, 2026
…down (#24579) v5 version of #24562 (based on `merge-train/spartan-v5`). Stacked on #24578 — review/merge that first. Note: this is a fresh implementation, not a port — the v5 prover-node architecture differs from `next` (it uses `EpochSession` / `TopTreeJob` rather than `EpochProvingJob` / `ProvingOrchestrator`), which is the structure actually running on testnet. ## Problem On a clean shutdown, `SessionManager.stop()` cancels every live session with reason `'prover-node stopping'`. That flows `EpochSession.cancel` → `TopTreeJob.cancel()` → `topTree.cancel({ abortJobs: true })` — the `abortJobs: true` was **hardcoded**, so a deploy/restart aborted the in-flight top-tree broker jobs. Aborting on a clean restart is wasteful: the proofs are still valid, agents are mid-flight, and a restarted node re-orchestrating the same epoch produces the same deterministically-hashed job ids, so it re-proves from scratch. ## Fix Thread an `abortJobs` decision from the cancel reason down to the broker: - `SessionManager.stop()` cancels with `{ abortJobs: false }` — a clean shutdown preserves the jobs. - `EpochSession.cancel(reason, { abortJobs })` forwards it to `TopTreeJob.cancel(abortJobs)` → `topTree.cancel({ abortJobs })`. - All other cancels (reorg / supersede / deadline) keep the default `abortJobs: true`, since their inputs are stale. Composes with #24578: a clean redeploy mid-epoch neither cancels the in-flight jobs nor (if one ever were aborted) poisons them, so the epoch keeps proving across the restart. ## Tests - `session-manager.test.ts`: `stop()` cancels every session with `abortJobs: false`. - `epoch-session.test.ts`: a normal cancel forwards `abortJobs: true` to the top-tree orchestrator; a clean-shutdown cancel forwards `abortJobs: false`. Note: not run locally in this session — the prover-node suite needs a full noir/wasm bootstrap that wasn't available here — so relied on CI. --- *Created by [claudebox](https://claudebox.work/v2/sessions/6fa5e242ed9ceae7) · group: `slackbot`* --------- Co-authored-by: Phil Windle <philip.windle@gmail.com> (cherry picked from commit c44e74e)
3 tasks
PhilWindle
added a commit
that referenced
this pull request
Jul 24, 2026
…down (#24579) v5 version of #24562 (based on `merge-train/spartan-v5`). Stacked on #24578 — review/merge that first. Note: this is a fresh implementation, not a port — the v5 prover-node architecture differs from `next` (it uses `EpochSession` / `TopTreeJob` rather than `EpochProvingJob` / `ProvingOrchestrator`), which is the structure actually running on testnet. ## Problem On a clean shutdown, `SessionManager.stop()` cancels every live session with reason `'prover-node stopping'`. That flows `EpochSession.cancel` → `TopTreeJob.cancel()` → `topTree.cancel({ abortJobs: true })` — the `abortJobs: true` was **hardcoded**, so a deploy/restart aborted the in-flight top-tree broker jobs. Aborting on a clean restart is wasteful: the proofs are still valid, agents are mid-flight, and a restarted node re-orchestrating the same epoch produces the same deterministically-hashed job ids, so it re-proves from scratch. ## Fix Thread an `abortJobs` decision from the cancel reason down to the broker: - `SessionManager.stop()` cancels with `{ abortJobs: false }` — a clean shutdown preserves the jobs. - `EpochSession.cancel(reason, { abortJobs })` forwards it to `TopTreeJob.cancel(abortJobs)` → `topTree.cancel({ abortJobs })`. - All other cancels (reorg / supersede / deadline) keep the default `abortJobs: true`, since their inputs are stale. Composes with #24578: a clean redeploy mid-epoch neither cancels the in-flight jobs nor (if one ever were aborted) poisons them, so the epoch keeps proving across the restart. ## Tests - `session-manager.test.ts`: `stop()` cancels every session with `abortJobs: false`. - `epoch-session.test.ts`: a normal cancel forwards `abortJobs: true` to the top-tree orchestrator; a clean-shutdown cancel forwards `abortJobs: false`. Note: not run locally in this session — the prover-node suite needs a full noir/wasm bootstrap that wasn't available here — so relied on CI. --- *Created by [claudebox](https://claudebox.work/v2/sessions/6fa5e242ed9ceae7) · group: `slackbot`* --------- Co-authored-by: Phil Windle <philip.windle@gmail.com> (cherry picked from commit c44e74e)
rangozd
pushed a commit
to rangozd/aztec-packages
that referenced
this pull request
Aug 5, 2026
…o next (AztecProtocol#24933) Forward-ports the **prover** and **node / world-state / barretenberg** slices of the v5-next → next backlog (work merged to `v5-next` after the ~2026-07-08 cut that reshaped `next`). ## Applied (clean cherry-picks, chronological) - feat(prover): revive cancelled proving jobs from a persisted aborted state (AztecProtocol#24578) - fix(prover-node): rebuild pruned checkpoint provers and recover the epoch (AztecProtocol#24436) - feat: exported in-process testing network (AztecProtocol#24629) - fix(validator): sync world state before forking in checkpoint proposal validation (AztecProtocol#24694) ##⚠️ Needs owner conflict-resolution (conflict against reshaped `next`; not included here) Cherry-pick onto this branch and resolve: - [ ] `git cherry-pick -x a4e3a44` — feat(world-state): support prefilled nullifiers in genesis state (AztecProtocol#24567) **[RESHAPE — v6 subsystem rewritten; re-implement, do not merge]** - [ ] `git cherry-pick -x c44e74e` — fix(prover-node): do not abort in-flight proving jobs on a clean shutdown (AztecProtocol#24579) - [ ] `git cherry-pick -x 1039d38` — feat: allow custom proof submission target address (AztecProtocol#24270) Part of the manual v5-next → next backlog sweep. Draft until conflicts are resolved and CI is green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v5 version of #24562 (based on
merge-train/spartan-v5). Stacked on #24578 — review/merge that first.Note: this is a fresh implementation, not a port — the v5 prover-node architecture differs from
next(it usesEpochSession/TopTreeJobrather thanEpochProvingJob/ProvingOrchestrator), which is the structure actually running on testnet.Problem
On a clean shutdown,
SessionManager.stop()cancels every live session with reason'prover-node stopping'. That flowsEpochSession.cancel→TopTreeJob.cancel()→topTree.cancel({ abortJobs: true })— theabortJobs: truewas hardcoded, so a deploy/restart aborted the in-flight top-tree broker jobs. Aborting on a clean restart is wasteful: the proofs are still valid, agents are mid-flight, and a restarted node re-orchestrating the same epoch produces the same deterministically-hashed job ids, so it re-proves from scratch.Fix
Thread an
abortJobsdecision from the cancel reason down to the broker:SessionManager.stop()cancels with{ abortJobs: false }— a clean shutdown preserves the jobs.EpochSession.cancel(reason, { abortJobs })forwards it toTopTreeJob.cancel(abortJobs)→topTree.cancel({ abortJobs }).abortJobs: true, since their inputs are stale.Composes with #24578: a clean redeploy mid-epoch neither cancels the in-flight jobs nor (if one ever were aborted) poisons them, so the epoch keeps proving across the restart.
Tests
session-manager.test.ts:stop()cancels every session withabortJobs: false.epoch-session.test.ts: a normal cancel forwardsabortJobs: trueto the top-tree orchestrator; a clean-shutdown cancel forwardsabortJobs: false.Note: not run locally in this session — the prover-node suite needs a full noir/wasm bootstrap that wasn't available here — so relied on CI.
Created by claudebox · group:
slackbot