test(e2e): deflake optimistic-proving by anchoring epoch and asserting on checkpoint header slot - #24364
Merged
PhilWindle merged 1 commit intoJun 29, 2026
Conversation
…g on checkpoint header slot The 'proves an epoch via checkpoint-driven flow' test asserts the prover-node started proving an epoch mid-epoch (a CheckpointProver registered before the epoch's last slot). On a cold-started chain, epoch 0 begins at genesis but the L1-contract deploy phase warps the simulated clock forward several slots while the sequencer is still starting up; on a slow CI box the sequencer only goes live near slot 4-5, so epoch 0 gets its only checkpoint at the last slot and there is nothing to prove mid-epoch. This surfaced after the deploy-speedup work shifted setup timing (CI run 901b2552713b08a6: 'start slots by epoch: [[0,5]]', lastSlot 5). - Anchor both happy-path tests on a freshly-started epoch via waitUntilNextEpochStarts() (the pattern the reorg tests already use) so the sequencer is warm and the epoch under test gets checkpoints from its start. - Reframe the sampler to record the lowest checkpoint *header* slot per epoch (prover.slotNumber) instead of the wall-clock slot at observation time, and assert that header slot < lastSlot. The header slot is the deterministic property the assertion actually cares about and does not race the 100ms sampler tick.
Collaborator
Flakey Tests🤖 says: This CI run detected 1 tests that failed, but were tolerated due to a .test_patterns.yml entry. |
spalladino
commented
Jun 29, 2026
| // until slot 4-5 of epoch 0, so its first — and only — checkpoint lands at the epoch's last | ||
| // slot and there is nothing to prove mid-epoch. Anchoring guarantees a full epoch with | ||
| // checkpoints produced from its start, which is the premise the optimistic check needs. | ||
| await test.waitUntilNextEpochStarts(); |
Contributor
Author
There was a problem hiding this comment.
I'll probably change this to a warp instead of wait in an upcoming PR
PhilWindle
approved these changes
Jun 29, 2026
spalladino
added a commit
that referenced
this pull request
Jun 29, 2026
… prune predicate (#24379) ## Flake `single-node/recovery/sync_after_reorg.test.ts` → "new node can sync world-state after unpruned reorg" was flagged FLAKED on `merge-train/spartan-v5` (CI hash `32e4c65996158955`, group `e2e-p2p-epoch-flakes`). The first attempt failed; the retry passed. First-attempt failure: ``` expect(received).toEqual(expected) // deep equality Expected: 0 Received: 5 > 54 | expect(await node.getBlockNumber()).toEqual(0); ``` The fresh node synced all 5 unproven blocks instead of pruning them back to genesis. ## Root cause The test stops the sequencer, opens a reorg window, then asserts a fresh node prunes the unproven checkpoints back to block 0 on initial sync. The fresh archiver only prunes when `rollup.canPruneAtTime` is true (`l1_synchronizer.ts` `handleEpochPrune`). The contract allows pruning once the current epoch reaches `getEpochForCheckpoint(proven + 1) + proofSubmissionEpochs + 1` — with `proofSubmissionEpochs = 1` that is `oldestPendingEpoch + 2`, where `oldestPendingEpoch` is the epoch of the first (oldest pending) checkpoint, since nothing is ever proven (no prover node). The test hardcoded `waitUntilEpochStarts(2)`, which is only correct when the first checkpoint lands in epoch 0. On a freshly-deployed chain the L1-contract deploy warps the sim clock (~17×12s) and the sequencer comes up late, so the first checkpoint can land in epoch 0 or epoch 1 depending on cold-start timing: - Passing CI run: first checkpoint at L2 slot 3 → epoch 0 → deadline epoch 2 → waiting for epoch 2 made it prunable → "Removed 5 checkpoints after checkpoint 0 due to predicted reorg" → block 0. - Failing CI run: first checkpoint at L2 slot 6 → epoch 1 → deadline epoch 3 → waiting only for epoch 2 left proofs still acceptable → `canPruneAtTime` false → no prune → fresh node kept all 5 blocks. This is the same cold-start clock-warp family as the already-merged proving-test fixes (#24364, #24372), but this reorg-recovery test is not covered by them. ## Fix Replace the hardcoded `waitUntilEpochStarts(2)` with a poll of the rollup's own prune predicate (`canPruneAtTime` at the current L1 block timestamp) before creating the fresh node. This is robust to where the cold-start warp lands the first checkpoint, and tests the exact precondition the fresh archiver's `handleEpochPrune` checks. Evaluating at the current block timestamp is stricter than the archiver's check (which looks one L1 block further ahead), so once the wait returns the archiver's prune is guaranteed to fire. The behavioral assertion (fresh node prunes to block 0 during a prunable reorg window) is preserved. ## Verification - Full `yarn build` passes. - Ran the test locally — passes (the fresh archiver waits on the new poll, prunes the 5 checkpoints, and reaches block 0). The failing path requires the specific cold-start timing that only manifests under CI load (here the first checkpoint landed in epoch 0), so the red case is reasoned from the two CI logs and the contract math rather than reproduced locally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deflakes the
single-node/proving/optimistic › happy path › proves an epoch via checkpoint-driven flowe2e test, which intermittently fails on the v5 line (e.g. CI run901b2552713b08a6,Optimistic proving start slots by epoch: [[0,5]]→Expected: < 5 / Received: 5).Root cause
The test asserts the prover-node began proving the epoch optimistically — that a
CheckpointProverfor the epoch was registered for a checkpoint before the epoch's last L2 slot. ACheckpointProveronly appears in theCheckpointStorewhen the sequencer publishes a checkpoint for that epoch on chain, so the observed slot is the slot of the epoch's first checkpoint.On a cold-started chain, epoch 0 begins the instant genesis is set, but
setup()mines ~25 L1-deploy blocks that each warp the simulated clock forward by one ethereum slot (12s) while the sequencer is still starting and syncing. On a slow CI box the sequencer only becomes active around simulated slot 4-5, so epoch 0 gets its only checkpoint at slot 5 (the epoch's last slot) and there is nothing to prove mid-epoch. The recent deploy-speedup work (#24316) shifted this setup timing, which is why the flake started surfacing on this line. The prover behaved correctly — this is a test-harness flake, not a product regression.A secondary fragility: the sampler recorded the wall-clock slot observed at the 100ms tick rather than the prover's own checkpoint header slot, so even with a warm epoch a checkpoint with header slot N could first be observed in wall-clock slot N+1.
Fix
await test.waitUntilNextEpochStarts()(the same pattern the reorgdescribeblocks already use) before starting the sampler and landing the tx. This guarantees the sequencer is warm and the epoch under test gets checkpoints from near its start — the premise the optimistic check requires.CheckpointProver(prover.slotNumber, i.e.checkpoint.header.slotNumber) and assert that header slot< lastSlot. The sampler still runs live (provers are reaped once the proof window closes), but it now keys off the deterministic property the assertion actually cares about, eliminating the 100ms-tick race.This preserves the behavioral intent (proving begins mid-epoch, not after the epoch ends) rather than relaxing it.
Notes
merge-train/spartan-v5: this test is v5-line-only (it does not exist onnext) and lives at itssingle-node/proving/path only after the e2e-consolidation refactor, which is on this line.