chore(port): forward-port v5-next prover + node/world-state backlog to next - #24933
Merged
Conversation
…state (#24578) v5 version of #24558 (same change, based on `merge-train/spartan-v5`). ## Problem When the prover broker cancelled a proving job it settled the job as a cached `{ status: 'rejected', reason: 'Aborted' }` result — in memory and in the database. Because a job's id is a deterministic hash of its inputs, any later enqueue of the same job returned that cached abort (`Cached proving job … Not enqueuing again`) and the facade failed the job with `Aborted`. So once a proof was aborted it stayed aborted, and a retry — a fresh epoch attempt after a reorg/failure, or a re-orchestration after a restart — got the cached abort instead of re-proving. A planned mid-epoch redeploy hit this and turned into a testnet epoch prune. ## Fix Cancellation is now its own first-class, **revivable** state: - A new `aborted` proving-job status (`ProvingJobStatus` / `ProvingJobSettledResult`). `cancelProvingJob` records it, notifies the current waiter, and **persists** it (`ProvingBrokerDatabase.setProvingJobAborted`). - `enqueueProvingJob` **revives** an aborted job: when the producer re-requests it, the broker clears the aborted state — in memory and, via the new `deleteProvingJobResult`, in the database — and re-enqueues it as if new. So a restart mid-revival keeps the job pending rather than resurrecting the abort. Genuine agent errors and timeouts still settle as terminal `rejected` results. Net effect: cancelling frees the job cleanly and records why, but re-requesting the same proof always brings it back to life — in the same process or after a restart — so an abort can never permanently block an epoch from proving. The stacked PR (prover-node clean-shutdown, v5) stops a clean prover-node shutdown from cancelling its in-flight jobs in the first place. ## Tests `proving_broker.test.ts` (both DB variants): cancel leaves a job `aborted`; re-request revives it and it completes; the aborted state persists across a restart; and a revived job survives a restart mid-revival as pending (not aborted). Note: not run locally in this session — the prover-client suite needs a full noir/wasm bootstrap that wasn't available here — so relied on CI. --- *Created by [claudebox](https://claudebox.work/v2/sessions/6fa5e242ed9ceae7) · group: `slackbot`* --------- Co-authored-by: Phil Windle <philip.windle@gmail.com> (cherry picked from commit a4e88ca)
…poch (#24436) Resolves A-1290. ## Problem A `CheckpointProver`'s sub-tree work forks world-state per block. When an L1 reorg prunes a base block, those fork reads fault and the prover permanently rejects its one-shot `blockProofs` promise. The store keeps the prover alive across a prune (`markPruned`) so a re-add of identical content can reuse its in-flight work. But a prune that removes the checkpoint also breaks the fork reads, so that reuse only ever hands back a poisoned prover: a re-add or a full `EpochSession` recreate re-references the same rejected `blockProofs` and fails immediately. The affected node then silently abandons the epoch until a process restart (a healthy prover elsewhere still proves it, so it's not a network stall). ## Fix **Drop the reuse machinery — a pruned prover cannot survive, so rebuild instead of reuse.** - `CheckpointStore.cancelAndRemoveAboveBlock` cancels and **removes** orphaned provers (replacing `markPrunedAboveBlock`); a re-add builds a fresh prover with a fresh `blockProofs`. - Removed the `pruned` flag / `markPruned` / `markCanonical` from `CheckpointProver`, and the `SlotWatcher` that only reaped lingering pruned provers. - `listCanonical` / `addOrUpdate` simplified — every prover in the store is now canonical. **Recover the epoch by rebuilding on re-add, rather than trying to prevent the terminal state.** A prune unwinds world-state as soon as it's detected — before the prover-node cancels the affected session — so a checkpoint prover mid-fork faults while still live and its session goes terminal `failed`. That race is inherent (the world-state rejection and the prover being cancelled are unordered), so instead of trying to reclassify it as a cancellation, we make the terminal state harmless: - `SessionManager.openFullSessionIfReady` no longer treats a terminal session as "already open" — it drops it and rebuilds. So re-adding an epoch's checkpoints reopens a fresh live session over fresh provers, via the checkpoint/prune triggers (ungated by the tick high-water mark). This reopen depends on the store no longer holding the poisoned prover: with the reuse machinery gone the rebuilt session gets a fresh prover with a healthy `blockProofs`, so it can actually prove rather than re-inheriting the rejected one. - `runSession` skips the failure-upload when the session's checkpoints no longer match canonical content — a prune-invalidated failure isn't a genuine proving failure, so it no longer emits a spurious post-mortem upload / alert. The only non-recovered case is "content pruned and never re-added" (chain stops publishing that epoch's checkpoints) — there's nothing to prove on this node then, so abandoning the epoch is correct. **Cancel the pruned prover's in-flight work promptly.** `CheckpointProver` now threads its abort signal into `PublicProcessor.process`, so a prune-driven cancel stops the current block's public execution immediately instead of running it to completion before the next `signal.aborted` check. This is independent of the recovery logic above — it just avoids wasting CPU re-executing a block whose checkpoint is being discarded. The in-flight block has not yet enqueued a broker proof, so aborting it loses no reusable work. ### Note: broker-level proof reuse is unaffected Removing the `CheckpointProver`-level reuse does not waste proving work. The expensive part — the SNARK proofs — is cached in the proving broker, content-addressed by job id, independent of world-state. `cancelJobsOnStop` defaults to `false`, so cancelling a prover does not abort its broker jobs; on an identical re-add the fresh prover regenerates identical inputs, and the broker dedups against the cached jobs/results (bounded by the broker's per-epoch cleanup). The reuse we deleted was witness-generation reuse, whose world-state forks cannot survive a prune anyway. ## Testing - `session-manager.test.ts`: pins the retry/recovery invariant — the periodic tick does not re-attempt a failed epoch (gated by `lastTickEpoch`), but a checkpoint re-add recovers it (ungated); a `failed` session's epoch is reopened once its checkpoints are re-added; a genuine proving failure still uploads a post-mortem; a failure coinciding with a canonical content change (prune) skips the upload. (Red/green verified for both the gate and the upload suppression.) - `checkpoint-store.test.ts` / `prover-node.test.ts`: pruned provers are cancelled and removed (not flagged); a re-add builds a fresh prover. - `checkpoint-prover.test.ts`: cancelling a prover mid-block aborts the signal its public execution is running under (red/green verified). - Full prover-node suite passes (166 tests); package typechecks clean; lint OK. - Updated `optimistic.parallel.test.ts` (the e2e that spawned this issue) to the cancelled+removed semantics. ## Notes - No operator/contract-dev-facing config or API changed, so no changelog entry. - The terminal `failed` state is not prevented, only recovered from: the world-state unwind and the prover cancellation are unordered, so the failure can't be reliably reclassified from the session side. Recovering on re-add is deterministic and needs no such race to be won. (cherry picked from commit c1e6229)
Upstreaming the utilities build for `aztec-kit` so external projects can benefit from our e2e scaffolding. Essentially the same thing we already use, but pure ts rather than the unpublishable .sh script. Verified with `ci-full-no-test-cache` (cherry picked from commit 7e09008)
…l validation (#24694) ## Problem On a live mainnet node, checkpoint-proposal validation could throw an uncaught error out of the gossipsub message handler whenever it ran while the world-state synchronizer trailed the archiver (block source) by a block: ``` ERROR world-state:database Call CREATE_FORK failed: Error: Unable to initialize from future block: 117860 unfinalizedBlockHeight: 117859. Tree name: NullifierTree ERROR p2p:libp2p_service Error handling gossipsub message: Error: Unable to initialize from future block: 117860 unfinalizedBlockHeight: 117859. Tree name: NullifierTree ``` The archiver already had block N (so the block lookups inside checkpoint validation succeeded), but world state's `unfinalizedBlockHeight` was still N-1. When `validateCheckpointProposal` forked world state at the parent block before it had been applied, the raw tree error escaped as an uncaught ERROR-level gossipsub message. The node lost that checkpoint validation/attestation for the round and spammed error logs; it self-healed the next tick, but it should not happen. ## Root cause The two world-state fork sites were asymmetric. The block re-execution path (`reexecuteTransactions`) syncs world state *before* forking. The checkpoint validation path (`validateCheckpointProposal`) forked via `checkpointsBuilder.getFork` *without* a preceding sync. The checkpoint path does sync the block source (archiver) earlier, but that does not advance the world state, so the gap remained. ## Fix - Move the sync-before-fork invariant into `FullNodeCheckpointsBuilder.getFork`: it now calls `worldState.syncImmediate(blockNumber, blockHash)` before `worldState.fork(blockNumber)`, so the single fork helper owns the invariant rather than each caller. `syncImmediate` blocks until world state reaches the block, or throws a typed error if it genuinely cannot, instead of leaking the raw tree error. - In `validateCheckpointProposal`, look up the parent block's hash from the block source and pass it through so the sync re-syncs on a world-state reorg. Wrap the fork in a try/catch that maps any failure to a clean, non-slashable `world_state_not_synced` result, so a sync/fork failure never surfaces as an uncaught ERROR-level gossipsub message. - As a second, independent guard, check that the forked world state's archive root matches the proposal's `lastArchiveRoot` before rebuilding the checkpoint. A mismatch means world state forked from a different chain than the proposal was built on, so we fail fast with a clean, non-slashable `initial_archive_mismatch` result — mirroring the block-proposal re-execution check — instead of a confusing downstream header/archive mismatch. Both new reasons record an `unvalidated` outcome and are non-slashable, matching the other "we couldn't validate right now" / local-state-divergence reasons. Other checkpoint-builder fork sites were reviewed and need no change: the proposer's `checkpoint_proposal_job` forks at its own already-synced, coherence-checked world-state tip, and the block-proposal re-execution path already syncs before forking and performs the equivalent archive-root check. ## Tests - `checkpoint_builder.test.ts`: `getFork` syncs world state to the block before forking, and propagates a sync failure without forking. - `proposal_handler.test.ts`: forks at the parent block number passing its block hash; a fork failure returns `world_state_not_synced`; and a fork whose archive root diverges from the proposal returns `initial_archive_mismatch`. ## Notes The same bug exists on the v4 line — the v4 site is the same function, forking via `this.worldState.fork(parentBlockNumber)` directly without a preceding `syncImmediate`. This fix should be ported there as well. Fixes A-1421 (cherry picked from commit b7a7d1b)
…down (#24579) v5 version of #24562 (based on `merge-train/spartan-v5`). Stacked on #24578 — review/merge that first. Note: this is a fresh implementation, not a port — the v5 prover-node architecture differs from `next` (it uses `EpochSession` / `TopTreeJob` rather than `EpochProvingJob` / `ProvingOrchestrator`), which is the structure actually running on testnet. ## Problem On a clean shutdown, `SessionManager.stop()` cancels every live session with reason `'prover-node stopping'`. That flows `EpochSession.cancel` → `TopTreeJob.cancel()` → `topTree.cancel({ abortJobs: true })` — the `abortJobs: true` was **hardcoded**, so a deploy/restart aborted the in-flight top-tree broker jobs. Aborting on a clean restart is wasteful: the proofs are still valid, agents are mid-flight, and a restarted node re-orchestrating the same epoch produces the same deterministically-hashed job ids, so it re-proves from scratch. ## Fix Thread an `abortJobs` decision from the cancel reason down to the broker: - `SessionManager.stop()` cancels with `{ abortJobs: false }` — a clean shutdown preserves the jobs. - `EpochSession.cancel(reason, { abortJobs })` forwards it to `TopTreeJob.cancel(abortJobs)` → `topTree.cancel({ abortJobs })`. - All other cancels (reorg / supersede / deadline) keep the default `abortJobs: true`, since their inputs are stale. Composes with #24578: a clean redeploy mid-epoch neither cancels the in-flight jobs nor (if one ever were aborted) poisons them, so the epoch keeps proving across the restart. ## Tests - `session-manager.test.ts`: `stop()` cancels every session with `abortJobs: false`. - `epoch-session.test.ts`: a normal cancel forwards `abortJobs: true` to the top-tree orchestrator; a clean-shutdown cancel forwards `abortJobs: false`. Note: not run locally in this session — the prover-node suite needs a full noir/wasm bootstrap that wasn't available here — so relied on CI. --- *Created by [claudebox](https://claudebox.work/v2/sessions/6fa5e242ed9ceae7) · group: `slackbot`* --------- Co-authored-by: Phil Windle <philip.windle@gmail.com> (cherry picked from commit c44e74e)
This pr adds an optional prover-node config, `PROVER_NODE_PROOF_SUBMISSION_TARGET_ADDRESS`, that lets a prover send the `submitEpochRootProof` tx to a contract other than the rollup. Everything else is unchanged: all rollup state is still read from the real rollup, only the destination of the submit tx is redirected. When unset, it defaults to the rollup address, so existing provers are unaffected. Use case: A prover that wants to claim prover tips from oxide-styled withdrawal txs must register itself in the Pool contract as the first prover to submit a proof of a given length L. To do that, it deploys a wrapper contract that calls the Pool before (check proven length = L-T) and after (check proven length = L) calling `submitEpochRootProof` on the rollup, then points this config at the wrapper. Closes GK-752 (cherry picked from commit 1039d38)
PhilWindle
marked this pull request as ready for review
July 24, 2026 10:22
…ults (#24678) Follow-up to #24436 (A-1290). That PR fixed the immediate prune-induced failure but left a class of race conditions around L1-reorg prunes: every "was this failure a prune or a genuine failure?" predicate samples control-plane state that lags the data-plane world-state unwind, so it is racy by construction. The fix pushes the fact down to where it is unambiguous — **decouple "a checkpoint prover failed" (a fact about the prover) from "the epoch failed" (a decision)**, and never treat a *cancel* (prune / reap / shutdown) as a failure. A genuine failure is then told apart from a prune with no control-plane sampling. - **`CheckpointProver`** — gains `isFailed()`, set when its block proofs reject for a non-cancel reason (a sub-tree fault or a prune-induced fork fault). On failure it fires an `onFailed` callback so the owner can upload that single checkpoint's post-mortem. A cancel leaves `isFailed()` false. - **`EpochSession`** — a fault now settles in one of two terminal states, decided by the provers: - `stopped` — a `CheckpointProver` under it failed. May be a prune, so it is **not** uploaded and the session is dropped; the epoch recovers when a prune/re-add installs a fresh prover. - `failed` (`hasFailed()`) — the session's **own** top-tree/submit work failed while *every* prover was healthy. Healthy provers rule out a prune, so this is a genuine, race-free failure. - **`SessionManager`** — removed `lastTickEpoch`. Two mechanisms stop a doomed epoch being re-proved every tick: `openFullSessionIfReady` refuses to build over a set containing a failed prover, and `recreateInvalidSessions` **retains** a genuinely-`failed` full session as a do-not-re-prove marker, replacing it only when its canonical content changes (a re-add). A `failed` session fires `onSessionFailed` (session-level upload). Recovery from prune/reorg flows through the ungated `checkpoint`/`prune` triggers, reusing already-completed sub-proofs from the content-addressed broker. - **`ProverNode`** — uploads post-mortems at **two levels**: per checkpoint (`tryUploadCheckpointFailure`, wired to `CheckpointProver.onFailed` — fires for prune-induced faults too, on purpose) and per session (`tryUploadEpochFailure`, wired to `onSessionFailed`). `expireEpoch` only releases the chonk cache and reaps — it does **not** upload, since a missed-window epoch's provers may already be pruned by then. - **Re-proving** — add `rerunCheckpointProvingJob` to re-prove a single downloaded checkpoint in isolation (sub-tree only; no epoch top-tree or L1 submit), sharing offline setup with `rerunEpochProvingJob`. `downloadEpochProvingJob` is unchanged (same serialized format). - `epoch-session.test.ts` — a fault ends `stopped` (a prover failed) vs `failed` (top-tree/submit with healthy provers). - `checkpoint-prover.test.ts` — a mid-proof fork fault sets `isFailed()` and fires `onFailed` once; a cancel does neither. - `session-manager.test.ts` — skips building over a failed prover; retains + uploads a `failed` session and replaces it on content change; does not upload on `stopped`; data-plane fault + re-add (identical and different content) recovers and completes. - `prover-node.test.ts` — a registered checkpoint prover failure routes to `tryUploadCheckpointFailure`; `expireEpoch` reaps without uploading. - `upload_failed_proof.test.ts` (e2e) — the existing epoch upload/rerun test, plus a new checkpoint test: force a sub-tree failure via the `checkpointProveOverride` hook, capture the per-checkpoint upload, and re-prove it via `rerunCheckpointProvingJob`. - prover-node README rewritten to the failed-prover / two-level-upload model, with the "how a genuine failure is told apart from a prune" rationale. prover-node unit suite green; full `yarn build`; format + lint clean. Resolves A-1418. (cherry picked from commit 6795246)
PhilWindle
enabled auto-merge
July 24, 2026 11:46
spalladino
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Forward-ports the prover and node / world-state / barretenberg slices of the v5-next → next backlog (work merged to
v5-nextafter the ~2026-07-08 cut that reshapednext).Applied (clean cherry-picks, chronological)
next; not included here)Cherry-pick onto this branch and resolve:
git cherry-pick -x a4e3a445ee— feat(world-state): support prefilled nullifiers in genesis state (feat(world-state): support prefilled nullifiers in genesis state #24567) [RESHAPE — v6 subsystem rewritten; re-implement, do not merge]git cherry-pick -x c44e74e5ae— fix(prover-node): do not abort in-flight proving jobs on a clean shutdown (fix(prover-node): do not abort in-flight proving jobs on a clean shutdown #24579)git cherry-pick -x 1039d38148— feat: allow custom proof submission target address (feat: allow custom proof submission target address #24270)Part of the manual v5-next → next backlog sweep. Draft until conflicts are resolved and CI is green.