feat(prover-node): stop caching checkpoint txs; re-fetch from the pool for failure upload (A-1216) by PhilWindle · Pull Request #24983 · AztecProtocol/aztec-packages

PhilWindle · 2026-07-25T21:57:15Z

What

Stops the prover-node from holding every checkpoint's Tx objects in memory for the whole proof-submission window.

Drops CheckpointProver.txs (an instance map that was populated at gather and never cleared). executeCheckpoint now consumes the gathered txs locally and lets them go out of scope.
The only reader of the cached txs after execution was failure upload. SessionManager.buildProvingData now re-fetches each checkpoint's txs from the tx pool (via a new CheckpointProver.getTxsForUpload, concurrently across checkpoints) instead of reading the cached map. It becomes async, and the two prover-node upload call sites await it.

Why

A live heap snapshot of a prover-node under load showed thousands of retained input Tx objects (client proofs + public inputs) — one set per registered CheckpointProver, held until the 100-epoch submission window expired. The tx pool already stores these txs durably on disk (kept until L1 finality, with A-1274's retention margin covering a lagging prover), so the in-memory copy is pure duplication of data the pool holds.

Re-fetch is safe: at failure time the txs were used for proving moments ago, so they are almost always still resident in the local pool (reqresp covers the rest); the A-1274 margin guarantees the durable case. Re-fetch is best-effort — a tx the pool can no longer supply is logged and omitted rather than failing the post-mortem, since a partial snapshot is still useful.

Note the rerun/replay path (rerun-epoch-proving-job.ts) is unaffected — it rebuilds from the already-uploaded jobData.txs, not from live prover state.

Testing

yarn build, full @aztec/prover-node suite (184/184).
New tests: a CheckpointProver caches no txs and re-fetches from the pool on demand; failure upload rebuilds complete EpochProvingJobData by re-fetching.

Depends on

A-1274 (tx-pool retention margin behind finality) — guarantees the pool keeps txs long enough for a lagging prover's failure-upload re-fetch.

…l for upload (A-1216) Each CheckpointProver cached its checkpoint's full Tx objects (client proofs + public inputs) in an instance map that was never cleared, and the prover stays registered in the CheckpointStore until the proof-submission window expires — so those txs sat on the heap for the whole window. A live heap snapshot showed thousands of retained input Tx objects. The tx pool already holds these txs durably on disk (until L1 finality, with A-1274's retention margin covering a lagging prover), so the in-memory copy is pure duplication. - Drop CheckpointProver.txs: executeCheckpoint consumes the gathered txs locally and lets them go out of scope. - The only post-execution reader was failure upload. SessionManager.buildProvingData now re-fetches each checkpoint's txs from the pool (via a new CheckpointProver.getTxsForUpload, concurrently across checkpoints) instead of reading the cached map; it becomes async and the two prover-node upload call sites await it. Re-fetch is best-effort — a tx the pool can no longer supply is logged and omitted rather than failing the post-mortem. Verified: yarn build, full prover-node suite (184/184), plus tests asserting the prover caches nothing and failure upload rebuilds complete EpochProvingJobData by re-fetching. Depends on A-1274 (tx-pool retention margin behind finality).

spalladino

Do we have a sense of how much memory (in %) we cut down with this change? In other words, do we know if this is enough to keep prover memory under control?

…l for failure upload (A-1216) (#24983) ## What Stops the prover-node from holding every checkpoint's `Tx` objects in memory for the whole proof-submission window. - Drops `CheckpointProver.txs` (an instance map that was populated at gather and never cleared). `executeCheckpoint` now consumes the gathered txs locally and lets them go out of scope. - The only reader of the cached txs *after* execution was failure upload. `SessionManager.buildProvingData` now re-fetches each checkpoint's txs from the tx pool (via a new `CheckpointProver.getTxsForUpload`, concurrently across checkpoints) instead of reading the cached map. It becomes `async`, and the two `prover-node` upload call sites `await` it. ## Why A live heap snapshot of a prover-node under load showed thousands of retained input `Tx` objects (client proofs + public inputs) — one set per registered `CheckpointProver`, held until the 100-epoch submission window expired. The tx pool already stores these txs durably on disk (kept until L1 finality, with A-1274's retention margin covering a lagging prover), so the in-memory copy is pure duplication of data the pool holds. Re-fetch is safe: at failure time the txs were used for proving moments ago, so they are almost always still resident in the local pool (reqresp covers the rest); the A-1274 margin guarantees the durable case. Re-fetch is best-effort — a tx the pool can no longer supply is logged and omitted rather than failing the post-mortem, since a partial snapshot is still useful. Note the rerun/replay path (`rerun-epoch-proving-job.ts`) is unaffected — it rebuilds from the already-uploaded `jobData.txs`, not from live prover state. ## Testing - `yarn build`, full `@aztec/prover-node` suite (184/184). - New tests: a `CheckpointProver` caches no txs and re-fetches from the pool on demand; failure upload rebuilds complete `EpochProvingJobData` by re-fetching. ## Depends on A-1274 (tx-pool retention margin behind finality) — guarantees the pool keeps txs long enough for a lagging prover's failure-upload re-fetch.

AztecBot · 2026-07-27T21:12:48Z

✅ Successfully ported to port-to-next-staging #24964.

PhilWindle · 2026-07-28T09:36:47Z

Do we have a sense of how much memory (in %) we cut down with this change? In other words, do we know if this is enough to keep prover memory under control?

Percentage is fairly low. Will have bigger gains in further PRs.

BEGIN_COMMIT_OVERRIDE fix(foundation): always include a result key in json-rpc responses (AztecProtocol#24840) fix(pxe): cap fresh secret pending tag indexes to the probed window (port AztecProtocol#24667) (AztecProtocol#24977) fix(archiver): resolve L2-to-L1 witness from a single store snapshot (AztecProtocol#24754) fix(sequencer): stop signalling already-executed governance payloads (AztecProtocol#24764) fix(archiver): clean up removed blocks from raw rows and ownership-check tx-effect deletes (AztecProtocol#24765) fix(node): warm KZG trusted setup at startup (AztecProtocol#24775) feat(prover-node): stop caching checkpoint txs; re-fetch from the pool for failure upload (A-1216) (AztecProtocol#24983) fix(sequencer): log tx failure reason at warn when dropping from mempool (AztecProtocol#25000) feat(prover-client): stop duplicating broker job inputs/results in memory (A-1215) (AztecProtocol#24990) fix(prover-client): don't retain inline job inputs in the facade without a failed-proof store (A-1517) (AztecProtocol#25027) END_COMMIT_OVERRIDE

Promotes `v5-next` onto `v5` for the **v5.2.0** release. Frozen at `ee5d2d367e` — the `v5-next` tip at cut time. Both `merge-train/spartan-v5` and `merge-train/fairies-v5` are drained to that same commit, so nothing staged is left behind. ### Testnet validation This branch was cut at `a4db216abf`, which is byte-identical to the `v5.2.0-nightly.20260815` tag — same commit, same tree (`0b22572eace2419ede6bbab173514d6965d4e0d0`). That nightly's CI3 run is green, its artifacts are published to npm and Docker Hub, and it has been **running healthily on testnet since 2026-08-15**: | Component | Image / reported version | |---|---| | validators, prover node, prover broker, prover agents (`testnet` ns) | `aztecprotocol/aztec:5.2.0-nightly.20260815` | | `v5.testnet.rpc.aztec-labs.com` | `nodeVersion=5.2.0-nightly.20260815` | | `canonical.testnet.rpc.aztec-labs.com` | `nodeVersion=5.2.0-nightly.20260815` | Chain advancing normally, no prunes or reorgs, no WARN/ERROR across node pods since rollout. The branch has since been fast-forwarded to `ee5d2d367e` to pick up #25242 (configurable RPC server HTTP timeouts and CORS allowed-headers). That is the only delta from the soaked tree — 8 files, +131/−10, no nightly covers it yet. Its config defaults were checked against Node's built-ins (`keepAliveTimeout` 5000 ms, `headersTimeout` 60000 ms) and match exactly, and the CORS default path resolves to the same `cors()` call as before, so a node that sets none of the new env vars behaves identically. It touches no circuits, protocol contracts, or generated constants. ### Manifest `.release-please-manifest.json` reads `5.2.0` on this branch, which is the released version — correct as-is, no change needed in this PR. `v5-next` moves to `5.3.0` separately in #25240; this branch is frozen and cannot pick that up, so the two can merge in either order. `v5` is a strict ancestor of `v5-next` this cycle, so there was no manifest conflict to pre-resolve. ### Scope 139 commits (86 non-merge, 23 PR-level) spanning 2026-07-14 to 2026-08-17. | Area | Theme | |---|---| | `prover-node` / `prover-client` | Epoch-proving robustness: retry-to-converge with failure declared only at submission-window expiry, ticker-driven expiry, per-checkpoint post-mortem upload, checkpoint-only re-proving, prune-induced fault handling (#24678, #24982, #24983, #24990, #25027) | | `pxe` | Sync performance: hash-pinned node read cache (#24969), anchor-bounded tag log caching (#25074), note/event validation from cached tx data (#25076), constrained tag sync (#24275), sender tagging finalization from log blocks (#25045) | | `p2p` / `validator-client` | Gossip tx validation no longer stalls behind tx-pool finalization (#25148); startup fails when p2p fails to start (#25177); slashing/proposing/health/`sendTx` gated on p2p connectivity (#25185); duplicate time-sensitive proposal validation removed (#25207); `ValidatedProposal` branding (#25222) | | `ethereum` / `aztec-node` | L1 watchers poll `getLogs` instead of `eth_newFilter` (#25176); block stream sync no longer resolves against an earlier pass (#25206) | | `stdlib` / `foundation` | Deserialization bounds hardening (#25026, #25028, #25029, #25109); checkpoint block-shape and block-count validation (#25229); JSON-RPC cookies (#25231) | | `archiver` | Removed-block cleanup and ownership-checked tx-effect deletes (#24765); L2→L1 witness from a single store snapshot (#24754) | | `aztec` | Declarative deployment framework at `@aztec/aztec/deploy` (#24685) | | `slasher` | Own-validator slash-target warnings and metrics (#25058) | | telemetry | JSON-RPC metrics (#25159) | | JSON-RPC server | Configurable HTTP keep-alive / headers timeouts and CORS allowed-headers, defaults preserving current behaviour (#25242) | | toolchain | Noir bumped to `v1.0.0-beta.25` (#24907) | **One breaking change**, inherited from the Noir bump: note types declared directly inside a `contract` module must now be `pub`. Everything else is additive or internal. ### Protocol constants Built from source and compared against `v5` — a cache-free rebuild of every protocol circuit with each ref's own nargo and bb, then regeneration of `vk_tree.ts` and `protocol_contract_data.ts`: - `vkTreeRoot` = `0x2b3b6ea4412b9c8f6457a37f91a2870306f8641e07e16a49b68bda6f8bc02892` — unchanged from v5.1.0 - `protocolContractsHash` = `0x2c075866eafc88a1f6f9addc7e337c6e64e45d1cb7fd7c0d612ebcec72aab2ca` — unchanged from v5.1.0 The Noir beta.24 → beta.25 bump does not reach the circuits: the release build consumes the committed `pinned-build.tar.gz`, which is bit-identical between v5.1.0 and this commit (blob `3bedcb1fd1…`), so the protocol-circuit bytecode is frozen rather than recompiled. The 47 verification keys were recomputed locally from that pinned bytecode with the cache disabled, and all 47 `check_pinned_vk` checks passed. `protocolContractsHash` is likewise backed by the new `pinned-protocol-contracts.tar.gz`, whose three artifacts were byte-compared against the build. Both values were also confirmed inside the published `@aztec/protocol-contracts` and `@aztec/noir-protocol-circuits-types` packages for `5.2.0-nightly.20260815`. That makes v5.2.0 a drop-in upgrade against the current rollup rather than a coordinated one. ✅ **Gate re-run against `ee5d2d367e` (the current head) and passed** — both values reproduced exactly from a build at this commit, with `vkTreeRoot` numerically evaluated rather than inferred. `29556326ce..ee5d2d3` touches no `noir-projects/**`, `l1-contracts/src/**`, `ConstantsGen.sol`, `constants.gen.ts` or `constants.nr`. Scope of the check: it verifies that the pinned circuit bytecode plus locally recomputed VKs agree with the pin — not that a from-source recompile reproduces v5.1.0's bytecode. <details> <summary>Commits added after the original constants check at 2955632</summary> `#25207` `#25222` `#25206` `#25185` `#25176` `#25163` `#25229` `#25159` `#25162` `#25231` `#25224` `#25228` `#25230` `#25242` plus their merge commits. </details> ### Known gap Migration notes carry entries under `## TBD` that arrived with the docs baseline backport (#25017) and describe `next`-line changes not present on the v5 line — protocol contracts removed from `@aztec/noir-contracts.js`, and the `at(wallet)` → `withWallet(wallet)` deprecation. Neither exists on this branch. There is also no `## 5.1.0` heading, and the entries that ship in v5.2.0 sit under `## TBD` rather than a version heading. This ships in the release docs, so it is worth correcting on `v5-next` and backporting to `v5` before the tag, rather than after.

…5041) **Umbrella / consolidated PR for the whole Fast Inbox (AZIP-22) stack.** This PR spans the entire stack — from the topmost branch down to `merge-train/spartan` — so **CI runs once** over the combined diff and reviewers get a single consolidated view. The four part PRs below are the review surface; this one exists to consolidate CI and provide the full-stack diff. ## Structure The stack was previously **20 per-issue PRs plus 2 umbrellas**. It is now **4 area PRs plus this umbrella**, each area PR holding **one squashed commit per issue** with the original PR description preserved in the commit body, plus follow-up commits at each area's end. | Part | Milestone | Commits | |---|---|---| | #25036 | circuits + L1 | 13 (A-1372, A-1373, A-1374, A-1375, A-1427, A-1432, A-1377, A-1378, A-1538, A-1539, sample regen, + cutoff-from-config fix, `num_msgs` removal) | | #25037 | node + flip | 16 (A-1379, A-1380, A-1381, A-1382, A-1383, A-1384, A-1385, comment/label follow-ups, A-1539 validator cap check, 256-cap sample regen, msgs-only block test, + checkpoint-backlog abort, TS cutoff/min-age mirror, validator check reordering, blob comment fix, per-head sample regen) | | #25038 | cleanup | 8 (A-1386, A-1387, A-1388, A-1434, A-1435, fmt reflow, + cutoff-API follow-through, per-head sample regen) | | #25067 | rolling-hash domain separation | 3 (A-1431, + rolling-hash comment fix, per-head sample regen) | ## What the stack does Fast Inbox replaces the per-checkpoint frontier-tree L1-to-L2 message path with a **streaming** one: `Inbox.sendL2Message` folds each message into a rolling sha256 hash and snapshots `{rollingHash, cumulativeTotal, key}` into a per-L1-block bucket in a ring. A proposer names a bucket; `propose` validates the consumption against the previous checkpoint's record; the circuits absorb each block's own message bundle and prove one variable-size `InboxParity<S>` per checkpoint. The result is message latency measured in a block or two rather than a full checkpoint. - **Circuits** — message-bundle components in `rollup-lib`; `inboxRollingHash` threaded end-to-end; per-block bundles with parity at the checkpoint root; a msgs-only block-root circuit for no-tx blocks carrying messages; one variable-size `InboxParity<S>` per checkpoint replacing the parity base ×4 + root fan-in; same-block consumption for non-first blocks. - **L1** — rolling-hash buckets in the `Inbox` with rollover-on-overflow and a 512 ring floor; streaming propose-time consumption validation; the censorship cap-escape branch. - **Node** — archiver bucket sync and chain validation; per-block world-state message insertion with per-block unwind; bucket references on block proposals; sequencer bucket selection mirroring the L1 predicate; the validator's four acceptance checks plus the censorship rule. - **Flip** — `Rollup.propose` cuts over to streaming validation, the per-block cap drops 1024 → 256, every block carries its own L1-to-L2 root in the blob, and the `streamingInbox` flag is removed. - **Cleanup** — the legacy frontier trees, `consume()`, `LAG`, `AZTEC_INBOX_LAG`, the checkpoint header's `inHash`, the 128-bit keccak rolling hash, and the node's legacy per-checkpoint paths are all deleted. - **Domain separation** — every rolling-hash link is tagged (`sha256ToField(DOM_SEP__INBOX_ROLLING_HASH || h || leaf)`), so a chain link can no longer be reinterpreted as an untagged two-field sha256 hash such as an `out_hash` merkle node. ## Rebase notes Currently based on `merge-train/spartan` at `cec2d754d2`. The original rebase (onto `63aa610685`) absorbed **#25007 (split noir-projects into `fnd/` and `labs/`)** — every circuit path moved to `noir-projects/fnd/noir-protocol-circuits/`. Conflicts of substance, resolved against that base: - `ProvingBroker.#getProvingJob` — this stack's oldest-epoch-first selection across queues (so `InboxParity` is not starved by sustained block production) now runs over the base's claim-then-read-inputs-from-database protocol (#24990, A-1215). The selection re-runs if the claimed winner has no inputs; losers return to their queues while the claimed job stays popped, so it cannot be re-selected. - `CheckpointProver` — kept the base's removal of the in-memory `txs` cache (#24983, A-1216) alongside this stack's `CheckpointSubTreeProofs` result type. - The broker duplicate-job-id test keeps the base's differing proof type for its second enqueue; the base's identity check is metadata-level, so re-enqueuing with the same type would no longer throw. This stack deletes `pinned-build.tar.gz`, so it builds circuits from source and runs gates the pinned path skips (`nargo fmt --check`, reset-cost staleness, full VK generation). ## Merge notes - **The 2026-08-04 adversarial review's five findings (1 Medium, 4 Lows) are all fixed in-stack.** The sequencer now aborts a checkpoint whose mandatory Inbox backlog cannot fit its remaining blocks — before the last block or the checkpoint proposal is gossiped — with a `maxBlocksPerCheckpoint >= MIN_BLOCKS_FOR_INBOX_CATCHUP` config floor derived from the message caps (7 for 1024/checkpoint and 256/block) rejecting networks that an adversarial bucket pattern could halt permanently. The censorship cutoff and the minimum bucket age now derive from the configured Ethereum slot duration (L1 gains `TimeLib.getBuildFrameStart`) instead of a hardcoded 12s. Validators run the cheap O(1) bucket checks before collecting proposal txs, so a malicious proposer can no longer burn the validation window on a proposal a map lookup rejects. The dead `num_msgs` parity public input is gone. Stale comments (rolling-hash link formula, empty-block blob claim) are corrected. - **Needs explicit sign-off: the devnet block duration moves 6s → 3s** (`spartan/environments/network-defaults.yml`), taking it from 4 to 9 blocks per checkpoint. Devnet's 36s slot at 6s blocks derived 4, below the new floor, which would have made it a haltable network. Testnet and mainnet already derive 10 and are unchanged. - **Consensus-format changes**, all release-boundary (fresh networks, no migrations — nodes resync): the checkpoint header drops `inHash`; the block-proposal p2p format drops its zeroed `inHash`; the per-block blob encoding always carries the L1-to-L2 root; the per-block message cap is 256; the rolling-hash link is domain-tagged; `RollupConfigInput` and `TimeStorage` carry `ethereumSlotDuration`, which shifts the deploy config version. `ARCHIVER_DB_VERSION` and the p2p attestation store are bumped. - **A-1431 (domain separation) is resolved: adopted, implemented in #25067 at the top of the stack.** The earlier note here said a separator would have to be inserted below the flip commit; landing it at the stack top is equivalent in practice because the whole stack merges together before any network deploys the streaming inbox — the encoding becomes consensus at deployment, not at merge order. The flip must not ship to a network without #25067. - **Accepted residuals**, all in the hardening milestone: unconsumed-bucket overwrite protection (A-1390), L1-reorg bucket invalidation (A-1389), the validator L1-sync race (A-1393). An independent review raised A-1390's severity: if consumption stalls long enough that every retained bucket exceeds the per-checkpoint cap while every bucket under the cap has been overwritten, there is no proposable cursor — a cap-vs-window deadlock rather than the self-healing the notes assumed. Worth an explicit test in A-1390. - A failed final-block build can still leave a checkpoint below the censorship floor and cost that slot, but the proposer now detects it and abandons the checkpoint instead of spending the propose tx on something validators refuse and L1 reverts. The remaining gap is the build failure itself, to be tightened with the A-1392 boundary matrix. - **Known first-run flakes**: if `streaming_inbox.test.ts` tests 1–2 (mid-checkpoint inclusion, latency bound) flake early, add tight `error_regex` entries to `.test_patterns.yml` for the specific assertion rather than broad skips. None are pre-added. - **Gas** (fresh base vs fresh head, regenerated after the 2026-08-15 review fixes): `sendL2Message` mean 53,020 → 46,585 (−12%; includes +1,753 from `MessageSent` now carrying the full message and +36 from the domain tag), `Inbox` bytecode −19%; `propose` +~4% from `bucketHint` calldata and the consumption validation; `submitEpochRootProof` 1,000,225 → 991,032 without validators and 1,581,276 → 1,572,081 with — *cheaper* than the pre-Fast-Inbox base. The earlier committed baseline that made it look like a near-doubling predated #24190 (fee entries → full rehashed headers); #25237 refreshes it on the base branch (pending one approval). Known quirk for future regens: three `Rollup.t.sol` tests fail only under `FORGE_GAS_REPORT=true`, a Foundry `vm.expectRevert` call-depth interaction with the `IInbox.getBucket()` staticcall in `validateInboxConsumption`; invisible to normal CI. ## Branch note This umbrella lives on its own branch (`spl/fast-inbox-umbrella`), kept at the same commit as #25067's `spl/fast-inbox-4-domain-sep`. They are deliberately *not* the same branch: CI3 derives its spot-fleet name from the branch (`aztec-packages_<branch>_amd64_x-fast`), so two PRs sharing a head branch make each run terminate the other's build instance, and both fail with `SSM ... statusDetails=Undeliverable`. Keep the two branches in sync when force-pushing.

AztecBot added the port-to-next Forward-port this merged PR into next label Jul 25, 2026

PhilWindle added the ci-full Run all master checks. label Jul 26, 2026

alexghr approved these changes Jul 27, 2026

View reviewed changes

spalladino approved these changes Jul 27, 2026

View reviewed changes

spalladino merged commit 17578de into merge-train/spartan-v5 Jul 27, 2026
36 of 40 checks passed

spalladino deleted the phil/a-1216-stop-caching-checkpoint-txs-in-the-prover-node-read-from-the branch July 27, 2026 21:11

This was referenced Jul 27, 2026

chore: Accumulated ports to next #24964

Merged

feat: merge-train/spartan-v5 #25019

Merged

This was referenced Jul 28, 2026

feat(fast-inbox): circuits and L1 for the streaming inbox (part 1/3) #25036

Closed

feat(fast-inbox): streaming L1-to-L2 messages, AZIP-22 (umbrella) #25039

Closed

feat(fast-inbox): streaming L1-to-L2 messages, AZIP-22 (umbrella) #25041

Merged

aminsammara mentioned this pull request Aug 16, 2026

chore(release): 5.2.0 #25225

Merged

Navigation Menu

feat(prover-node): stop caching checkpoint txs; re-fetch from the pool for failure upload (A-1216) - #24983

feat(prover-node): stop caching checkpoint txs; re-fetch from the pool for failure upload (A-1216)#24983
spalladino merged 1 commit into
merge-train/spartan-v5from
phil/a-1216-stop-caching-checkpoint-txs-in-the-prover-node-read-from-the

PhilWindle commented Jul 25, 2026

Uh oh!

spalladino left a comment

Uh oh!

Uh oh!

AztecBot commented Jul 27, 2026

Uh oh!

PhilWindle commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

PhilWindle commented Jul 25, 2026

What

Why

Testing

Depends on

Uh oh!

spalladino left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AztecBot commented Jul 27, 2026

Uh oh!

PhilWindle commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants