test(e2e): anchor multi_proof on a fresh epoch instead of hardcoding epoch 0 - #24372
Merged
spalladino merged 1 commit intoJun 29, 2026
Merged
Conversation
…epoch 0 The 'submits proofs from multiple prover-nodes' test hardcoded epoch 0 (slots 0-5): after waitUntilEpochStarts(1) it filtered blocks to epoch 0 and read .at(-1). On a cold-started chain under CI load the single-node sequencer can come up after the chain has advanced past epoch 0's early slots, so its first block lands at slot 6 (epoch 1) and epoch 0 has zero blocks -> firstEpochBlocks empty -> .at(-1) throws TypeError (CI run 78c83e244fe732ef). The product is correct (it skips slots it is too late to propose for); the test's epoch-0 assumption is the flake. Anchor on a freshly-started epoch with the provers already warm (the idiom the sibling optimistic.parallel.test.ts uses), let it fully elapse, then snapshot its checkpoints behind a complete-view gate: read the authoritative L1 checkpoint tip via monitor.run(true) and wait until the archiver has indexed every checkpoint up to it before filtering to the anchored epoch, so a partial archiver view can't snapshot a prefix. Pass the checkpoint count (not block count) to waitForAllProversToSubmit and derive the expected proven tip from the last checkpoint's startBlock + blockCount - 1; getHasSubmittedProof keys on checkpoint count, which previously worked only because epoch 0 happened to have one block per checkpoint.
spalladino
force-pushed
the
spl/deflake-multi-proof-epoch-anchor
branch
from
June 29, 2026 16:58
5730b9b to
e3fcb02
Compare
spalladino
enabled auto-merge (squash)
June 29, 2026 17:22
fcarreiro
approved these changes
Jun 29, 2026
spalladino
added a commit
that referenced
this pull request
Jun 29, 2026
… prune predicate (#24379) ## Flake `single-node/recovery/sync_after_reorg.test.ts` → "new node can sync world-state after unpruned reorg" was flagged FLAKED on `merge-train/spartan-v5` (CI hash `32e4c65996158955`, group `e2e-p2p-epoch-flakes`). The first attempt failed; the retry passed. First-attempt failure: ``` expect(received).toEqual(expected) // deep equality Expected: 0 Received: 5 > 54 | expect(await node.getBlockNumber()).toEqual(0); ``` The fresh node synced all 5 unproven blocks instead of pruning them back to genesis. ## Root cause The test stops the sequencer, opens a reorg window, then asserts a fresh node prunes the unproven checkpoints back to block 0 on initial sync. The fresh archiver only prunes when `rollup.canPruneAtTime` is true (`l1_synchronizer.ts` `handleEpochPrune`). The contract allows pruning once the current epoch reaches `getEpochForCheckpoint(proven + 1) + proofSubmissionEpochs + 1` — with `proofSubmissionEpochs = 1` that is `oldestPendingEpoch + 2`, where `oldestPendingEpoch` is the epoch of the first (oldest pending) checkpoint, since nothing is ever proven (no prover node). The test hardcoded `waitUntilEpochStarts(2)`, which is only correct when the first checkpoint lands in epoch 0. On a freshly-deployed chain the L1-contract deploy warps the sim clock (~17×12s) and the sequencer comes up late, so the first checkpoint can land in epoch 0 or epoch 1 depending on cold-start timing: - Passing CI run: first checkpoint at L2 slot 3 → epoch 0 → deadline epoch 2 → waiting for epoch 2 made it prunable → "Removed 5 checkpoints after checkpoint 0 due to predicted reorg" → block 0. - Failing CI run: first checkpoint at L2 slot 6 → epoch 1 → deadline epoch 3 → waiting only for epoch 2 left proofs still acceptable → `canPruneAtTime` false → no prune → fresh node kept all 5 blocks. This is the same cold-start clock-warp family as the already-merged proving-test fixes (#24364, #24372), but this reorg-recovery test is not covered by them. ## Fix Replace the hardcoded `waitUntilEpochStarts(2)` with a poll of the rollup's own prune predicate (`canPruneAtTime` at the current L1 block timestamp) before creating the fresh node. This is robust to where the cold-start warp lands the first checkpoint, and tests the exact precondition the fresh archiver's `handleEpochPrune` checks. Evaluating at the current block timestamp is stricter than the archiver's check (which looks one L1 block further ahead), so once the wait returns the archiver's prune is guaranteed to fire. The behavioral assertion (fresh node prunes to block 0 during a prunable reorg window) is preserved. ## Verification - Full `yarn build` passes. - Ran the test locally — passes (the fresh archiver waits on the new poll, prunes the 5 checkpoints, and reaches block 0). The failing path requires the specific cold-start timing that only manifests under CI load (here the first checkpoint landed in epoch 0), so the red case is reasoned from the two CI logs and the contract math rather than reproduced locally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deflakes
single-node/proving/multi_proof.test.ts › "submits proofs from multiple prover-nodes", which intermittently fails on the v5 line (e.g. CI run78c83e244fe732ef).Root cause
The test hardcoded epoch 0 (slots 0-5,
aztecEpochDuration=6): afterwaitUntilEpochStarts(1)it filtered blocks to epoch 0 and read.at(-1). On a cold-started chain under CI load, the single-node sequencer can come up after the L1 clock has already advanced past epoch 0's early slots — its first proposal then targets slot 6 (epoch 1), so epoch 0 ends up with zero blocks.firstEpochBlocksis empty,.at(-1)!.numberthrowsTypeError: Cannot read properties of undefined (reading 'number'), and the run short-circuits (~58s vs ~93-164s for a healthy run).This is the same cold-start startup-timing race as the optimistic-proving flakes. The product behaves correctly (the sequencer skips slots it is too late to propose for); the test's assumption that epoch 0 is always populated is what flakes.
Fix
waitUntilNextEpochStarts()+waitUntilEpochStarts(epoch + 1)), instead of hardcoding epoch 0. The anchored epoch's full slot range is guaranteed ahead of us with the sequencer warm — the same idiom the siblingoptimistic.parallel.test.tsuses.monitor.run(true), thenretryUntilthe archiver has indexed every checkpoint up to that tip before filtering to the anchored epoch. Alength > 0poll would race a partial archiver view and snapshot only a prefix of the epoch.getHasSubmittedProofis keyed by the number of checkpoints the epoch-root proof covers, so pass the checkpoint count towaitForAllProversToSubmitand derive the expected proven tip from the last checkpoint'sstartBlock + blockCount - 1. This also fixes a latent block-vs-checkpoint conflation that previously only worked because epoch 0 happened to have one block per checkpoint.The behavioral assertion (
provenBlockNumber === the anchored epoch's last block) is preserved.Notes
merge-train/spartan-v5: this test is v5-line-only.RewardLib.handleRewardsAndFees).tsgo --noEmitpasses; the e2e test was not run locally (non-deterministic startup race needing the CI harness).multi_proof.test.tsalso suffers a distinctbeforeEachdeploy hang (600s hook timeout) rooted inl1-contracts/scripts/forge_broadcast.js(a regression from test(e2e): 10x faster deploy L1 contracts via on-demand mining under anvil automine #24316). That is a different file/failure mode and is tracked separately; both need to land for this test to be fully reliable.