test: fix proof_boundary startup race - #24671
Merged
PhilWindle merged 1 commit intoJul 14, 2026
Merged
Conversation
…ribe The proof_boundary multi-node scenarios created their three validator nodes sequentially and let each sequencer start proposing at node construction. When the first-created validator drew the first slot's proposer, it broadcast the inaugural block proposal before the other committee members had joined the mock gossip bus (which has no replay), so the proposal was lost, the 3/3 quorum was missed, and the chain never produced its first checkpoint — timing out in computeBoundarySlot's waitUntilCheckpointNumber(1). Create the validator nodes with dontStartSequencer and start all sequencers together once every node has subscribed to gossip, matching the pattern already used by first_slot.test.ts and block-production/setup.ts.
spalladino
changed the title
PhilWindle
approved these changes
Jul 14, 2026
PhilWindle
pushed a commit
that referenced
this pull request
Jul 21, 2026
## What Fixes a low-frequency (~1 in 140) startup-race flake in the `proof_boundary` multi-node e2e suite (CI hash `c06563e7d2b7e067`, group `e2e-p2p-epoch-flakes`). All five scenarios time out in the shared `computeBoundarySlot()` helper at `waitUntilCheckpointNumber(1)` — the 3-validator committee never produces its first checkpoint. ## Why `setupTest` created the three validator nodes sequentially (`asyncMap`) and started each sequencer at node construction. A node subscribes to the gossip topics when its P2P client starts (unconditionally, at node creation), but the in-memory mock gossip bus has no message replay. So when the first-created validator happened to draw the first slot's proposer, it broadcast the inaugural block proposal before the other two committee members had been created and subscribed — the proposal was lost, the 3-of-3 quorum was missed, and the chain forked onto genesis and never produced checkpoint 1. It's randao-dependent (fatal only when the first-created node is the first proposer), hence the low frequency. ## Fix Create the validator nodes with `dontStartSequencer: true`, then `startSequencers(nodes)` once all three exist. All peers subscribe to gossip at creation, so no proposal is broadcast before the whole committee is on the bus. This matches the pattern already used by the non-flaky sibling tests `first_slot.test.ts` and `block-production/setup.ts`. ## Scope Test-only change. Related to A-1419, which tracks the underlying product-level recovery race (competing block-1 re-executions surfacing as `BlockNumberNotSequentialError`). This PR implements that issue's "keep the test startup barrier" acceptance criterion but does **not** resolve A-1419 — the product-side race is unchanged. (cherry picked from commit 54edf7a)
PhilWindle
pushed a commit
that referenced
this pull request
Jul 21, 2026
## What Fixes a low-frequency (~1 in 140) startup-race flake in the `proof_boundary` multi-node e2e suite (CI hash `c06563e7d2b7e067`, group `e2e-p2p-epoch-flakes`). All five scenarios time out in the shared `computeBoundarySlot()` helper at `waitUntilCheckpointNumber(1)` — the 3-validator committee never produces its first checkpoint. ## Why `setupTest` created the three validator nodes sequentially (`asyncMap`) and started each sequencer at node construction. A node subscribes to the gossip topics when its P2P client starts (unconditionally, at node creation), but the in-memory mock gossip bus has no message replay. So when the first-created validator happened to draw the first slot's proposer, it broadcast the inaugural block proposal before the other two committee members had been created and subscribed — the proposal was lost, the 3-of-3 quorum was missed, and the chain forked onto genesis and never produced checkpoint 1. It's randao-dependent (fatal only when the first-created node is the first proposer), hence the low frequency. ## Fix Create the validator nodes with `dontStartSequencer: true`, then `startSequencers(nodes)` once all three exist. All peers subscribe to gossip at creation, so no proposal is broadcast before the whole committee is on the bus. This matches the pattern already used by the non-flaky sibling tests `first_slot.test.ts` and `block-production/setup.ts`. ## Scope Test-only change. Related to A-1419, which tracks the underlying product-level recovery race (competing block-1 re-executions surfacing as `BlockNumberNotSequentialError`). This PR implements that issue's "keep the test startup barrier" acceptance criterion but does **not** resolve A-1419 — the product-side race is unchanged. (cherry picked from commit 54edf7a)
2 tasks
rangozd
pushed a commit
to rangozd/aztec-packages
that referenced
this pull request
Aug 5, 2026
## What Fixes a low-frequency (~1 in 140) startup-race flake in the `proof_boundary` multi-node e2e suite (CI hash `c06563e7d2b7e067`, group `e2e-p2p-epoch-flakes`). All five scenarios time out in the shared `computeBoundarySlot()` helper at `waitUntilCheckpointNumber(1)` — the 3-validator committee never produces its first checkpoint. ## Why `setupTest` created the three validator nodes sequentially (`asyncMap`) and started each sequencer at node construction. A node subscribes to the gossip topics when its P2P client starts (unconditionally, at node creation), but the in-memory mock gossip bus has no message replay. So when the first-created validator happened to draw the first slot's proposer, it broadcast the inaugural block proposal before the other two committee members had been created and subscribed — the proposal was lost, the 3-of-3 quorum was missed, and the chain forked onto genesis and never produced checkpoint 1. It's randao-dependent (fatal only when the first-created node is the first proposer), hence the low frequency. ## Fix Create the validator nodes with `dontStartSequencer: true`, then `startSequencers(nodes)` once all three exist. All peers subscribe to gossip at creation, so no proposal is broadcast before the whole committee is on the bus. This matches the pattern already used by the non-flaky sibling tests `first_slot.test.ts` and `block-production/setup.ts`. ## Scope Test-only change. Related to A-1419, which tracks the underlying product-level recovery race (competing block-1 re-executions surfacing as `BlockNumberNotSequentialError`). This PR implements that issue's "keep the test startup barrier" acceptance criterion but does **not** resolve A-1419 — the product-side race is unchanged. (cherry picked from commit 54edf7a)
rangozd
pushed a commit
to rangozd/aztec-packages
that referenced
this pull request
Aug 5, 2026
…rotocol#24930) Forward-ports the **end-to-end / testing** slice of the v5-next → next backlog (work merged to `v5-next` after the ~2026-07-08 cut that reshaped `next`). ## Applied (clean cherry-picks, chronological) - AztecProtocol#24518 docs(e2e): update READMEs for the consolidated suite layout - AztecProtocol#24534 test(e2e): instrument and diagnose bot suite setup cost - AztecProtocol#24566 perf(e2e): warp dead waits in multi-node recovery and proving tests - AztecProtocol#24570 perf(e2e): shrink e2e slot times - AztecProtocol#24564 perf(e2e): seed BananaFPC fee juice at genesis instead of bridging - AztecProtocol#24568 perf(e2e): seed standard contracts at genesis - AztecProtocol#24597 chore: add writing-e2e-tests skill - AztecProtocol#24590 feat(e2e): interactive handshake e2e - AztecProtocol#24671 test: fix proof_boundary startup race ##⚠️ Needs owner conflict-resolution (conflict against reshaped `next`; not included here) Cherry-pick onto this branch and resolve: - [ ] AztecProtocol#24569 `git cherry-pick -x 1d280af` — perf(e2e): overlap and batch setup txs in e2e harnesses - [ ] AztecProtocol#24479 `git cherry-pick -x da9ac1c` — feat: assert non revertible phase when setting fee payer Part of the manual v5-next backlog sweep. Draft until owners resolve the conflicts above and CI is green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes a low-frequency (~1 in 140) startup-race flake in the
proof_boundarymulti-node e2e suite (CI hashc06563e7d2b7e067, groupe2e-p2p-epoch-flakes). All five scenarios time out in the sharedcomputeBoundarySlot()helper atwaitUntilCheckpointNumber(1)— the 3-validator committee never produces its first checkpoint.Why
setupTestcreated the three validator nodes sequentially (asyncMap) and started each sequencer at node construction. A node subscribes to the gossip topics when its P2P client starts (unconditionally, at node creation), but the in-memory mock gossip bus has no message replay. So when the first-created validator happened to draw the first slot's proposer, it broadcast the inaugural block proposal before the other two committee members had been created and subscribed — the proposal was lost, the 3-of-3 quorum was missed, and the chain forked onto genesis and never produced checkpoint 1. It's randao-dependent (fatal only when the first-created node is the first proposer), hence the low frequency.Fix
Create the validator nodes with
dontStartSequencer: true, thenstartSequencers(nodes)once all three exist. All peers subscribe to gossip at creation, so no proposal is broadcast before the whole committee is on the bus. This matches the pattern already used by the non-flaky sibling testsfirst_slot.test.tsandblock-production/setup.ts.Scope
Test-only change. Related to A-1419, which tracks the underlying product-level recovery race (competing block-1 re-executions surfacing as
BlockNumberNotSequentialError). This PR implements that issue's "keep the test startup barrier" acceptance criterion but does not resolve A-1419 — the product-side race is unchanged.