refactor(e2e): consolidate the multi-node and single-node test categories - #24201
Merged
Conversation
spalladino
force-pushed
the
spl/e2e-consolidation
branch
from
June 23, 2026 20:36
6fbb112 to
c2cd99f
Compare
spalladino
force-pushed
the
spl/e2e-consolidation
branch
from
June 24, 2026 12:57
e47bed6 to
4afe5c0
Compare
spalladino
force-pushed
the
spl/e2e-consolidation
branch
from
June 25, 2026 17:32
e5d33c2 to
df1c814
Compare
PhilWindle
pushed a commit
that referenced
this pull request
Jun 26, 2026
## What Adds per-test timing capture to the e2e suite (A-1178, Phase 1), distinguishing **function-level** time inside our `setup()`/`teardown()` from **hook-level** time across jest's before/after hooks, and uploads one JSONL file per test to S3 with a `ci.sh` command to pull a run's timings back. This is the PR-independent subset that does not depend on the e2e consolidation reorg (#24201). The reorg-gated pieces (`wait:*`/compute body split, multi-node/prover-node start attribution) are deliberately deferred — those files don't exist on this base. Fixes A-1180 Fixes A-1181 ## How - **`yarn-project/end-to-end/src/shared/timing_env.mjs`** (new) — a `TimingEnvironment` subclass of foundation's `CustomEnvironment`. Calls `await super.handleTestEvent()` first (preserves the base env's unhandled-rejection patching), then times jest-circus events (`hook_start`/`hook_success`/`hook_failure` by hook type, `test_fn_*` for the body, `test_start`/`test_done` for total). Installs a collector on `this.global` and flushes JSONL on env `teardown`. Entirely gated on `E2E_TIMING_FILE`; a pure delegate otherwise. - **`package.json`** — `jest.testEnvironment` → `./shared/timing_env.mjs`. - **`fixtures/setup.ts`** — wraps the exported `setup()` and the inner `teardown` closure (both `ctx.teardown()` and the exported `teardown(context)` funnel through it) in try/finally, pushing their durations to `globalThis.__e2eTimings`, tagged with the running test (or `null` during `beforeAll`/`afterAll`). - **`ci3/exec_test`** — exports `E2E_TIMING_FILE` and, after the test, uploads the gzipped file via `cache_s3_transfer_to`. - **`ci3/run_test_cmd`** — passes the attempt's log id to `exec_test` as `E2E_LOG_ID` (tracks `rotate_log` on retries). - **`ci.sh`** — new `e2e-timings <ci_log_id> <folder>` command: `aws s3 cp --recursive` the job's prefix and gunzip each file to `<log_id>.jsonl`. - `tsconfig.json` / `eslint.config.js` — exclude/ignore the env `.mjs` (it imports foundation's env across packages and jest loads it from source), matching how foundation already ignores its own `src/jest/*.mjs`. ## JSONL schema (one line per test) ```jsonc { "suite":"e2e_fees", "name":"pays fee via private payment", "status":"passed", "commit":"abc1234", "branch":"merge-train/spartan-v5", "runId":"177...", "startedAt":"2026-06-24T12:00:02.000Z", "setupFnMs":42000, "beforeHooksMs":47000, "bodyMs":13000, "teardownFnMs":2500, "afterHooksMs":3000, "totalMs":63000 } ``` `beforeAll`/`afterAll` (once per describe) are emitted on a single suite-scoped line with `name: null`. ## Examples from this PR's CI run ```json5 // type:"test" — per-test, with beforeEach setup on the test line { "suite":"epochs_invalidate_block.parallel", "type":"test", "name":"e2e_epochs/epochs_invalidate_block proposer invalidates previous checkpoint…", "status":"passed", "setupFnMs":14330, "beforeHooksMs":19357, "bodyMs":179667, "teardownFnMs":732, "afterHooksMs":765, "totalMs":199797, "startedAt":"2026-06-25T12:58:34.643Z", "commit":"1925654a900…", "branch":"spl/a-1178-track-running-times", "runId":"28171484400" } ``` ```json5 // type:"suite" — the former name:null line (beforeAll/afterAll for the whole file) { "suite":"e2e_amm", "type":"suite", "name":null, "status":"passed", "setupFnMs":32591, "beforeHooksMs":61200, "teardownFnMs":102, "afterHooksMs":103, "totalMs":61303, "commit":"1925654a900…", "branch":"spl/a-1178-track-running-times", "runId":"28171484400" } ``` ## Storage & retrieval - S3 object: `s3://aztec-ci-artifacts/logs/e2e-timings/<CI_LOG_ID>/<LOG_ID>.log.gz` (gzipped JSONL). - `CI_LOG_ID` = the job's top-level log id (the decimal id in a job's `ci.aztec-labs.com/<id>` URL) — per-job, so it's collision-free under grind. - `LOG_ID` = the test's individual log id, the same id as the test's log at `ci.aztec-labs.com/<log_id>`, so a test log maps straight to its timing file. - Download: `./ci.sh e2e-timings <ci_log_id> <folder>` → `<folder>/<LOG_ID>.jsonl` per test (the `suite`/`name` fields inside identify the test). ## Notes - References A-1178 / A-1179 / A-1180 / A-1181 (Phase-1 subset; no auto-closing keywords). - No `ci-no-squash` label — single commit on merge.
PhilWindle
approved these changes
Jun 26, 2026
…ared wait-helpers First code commit of the e2e suite consolidation. Establishes the `multi-node` category (N validators on the in-memory mock-gossip bus) by promoting the epochs base class and extracting the highest-value shared wait-helpers. Base class: - Move end-to-end/src/e2e_epochs/epochs_test.ts -> end-to-end/src/multi-node/multi_node_test_context.ts and rename EpochsTestContext -> MultiNodeTestContext (EpochsTestOpts -> MultiNodeTestOpts). All 24 non-migrated epochs tests updated to import from the new location/name. Migrated pilot tests into end-to-end/src/multi-node/: - epochs_simple_block_building.test.ts (control) - epochs_missed_l1_publish.test.ts (helper-extraction target) - epochs_mbps.parallel.test.ts (stress + prover) Helpers introduced/extended: - fixtures/wait_helpers.ts: waitForBlockNumber / waitForProvenBlock, waitForTxs. - ChainMonitor.waitUntilCheckpointProven (event-driven, mirrors waitUntilCheckpoint). - MultiNodeTestContext.waitForAllNodes (+ waitForAllNodesToReachProvenCheckpoint / waitForAllNodesToReachBlockAtSlot): multi-node fan-out convergence. - MultiNodeTestContext.findSlotsWithProposers: extracts the EpochNotStable slot-search/warp loop. - MultiNodeTestContext.waitForSequencerEvent: one-shot sequencer-event waiter. The migration changes only setup wiring and async-waiting style; no test assertions were changed. Adds end-to-end/src/multi-node/README.md.
…s_ prefix Relocates all 24 e2e_epochs/*.test.ts files into the multi-node/ category folder established by the pilot commit, and drops the redundant epochs_ prefix from both the filenames and the describe titles (now multi-node/<name>). Also retroactively renames the 3 pilot files already in multi-node/ for consistency. e2e_epochs/ is now empty and removed. Rewires CI test discovery in end-to-end/bootstrap.sh: the epochs globs are replaced with src/multi-node/!(long_proving_time).test.ts (plus the long_proving_time special case), and the NAME derivation now strips the src/ prefix so it produces multi-node/ names for the new folder while leaving e2e_<dir>/<file> names unchanged. Re-points the four .test_patterns.yml entries that referenced e2e_epochs paths / epochs_ test names to the new multi-node/ paths. No test assertions change — only file location, titles, and import/CI wiring.
Extracts a ValidatorRegistrationHarness (multi-node/validator_registration_harness.ts) that composes MultiNodeTestContext: it registers a validator set on the in-memory mock-gossip bus via initialValidators (genesis staking + validator-set-lag advance, the mock-gossip replacement for P2PNetworkTest's MultiAdder/GSE flow), exposes the per-validator keys/addresses, spawns validator nodes on the mock bus, and resolves the slasher/slashing-proposer L1 contracts. Converts the duplicate_attestation_slash and duplicate_proposal_slash tests from P2PNetworkTest (which ran REAL libp2p despite mockGossipSubNetwork:true — the flag was inert because setup_p2p_test.createNode never passed a p2pServiceFactory) to genuine mock gossip, and relocates them into multi-node/e2e_slashing/. The generic offense/ proposer helpers in e2e_p2p/shared.ts are reused unchanged via cross-folder import. No test assertions change — only the gossip transport, registration wiring, and location. Wires src/multi-node/e2e_slashing/*.test.ts into CI discovery (end-to-end/bootstrap.sh) and re-points the duplicate_proposal_slash flaky entry in .test_patterns.yml to the new path.
…ests to intent-revealing helpers Add waitForNodeCheckpoint / waitForNodeProvenCheckpoint to fixtures/wait_helpers.ts, a single-node checkpoint-number convergence wait supporting eq/gte/gt/lte/lt comparisons (reorg/prune tests wait for the number to drop, not just rise). Convert l1_reorgs, optimistic_proving, manual_rollback, partial_proof, and proof_public_cross_chain to the new and existing helpers (waitForBlockNumber/waitForProvenBlock/waitUntilProvenCheckpointNumber), removing hand-rolled retryUntil checkpoint/block polls and local getCheckpointNumber helpers.
… setup presets Carve the prod-sequencer single-node lifecycle out of MultiNodeTestContext into a new SingleNodeTestContext parent (environment, node spawning, prover lifecycle, and the epoch/checkpoint/proof-window/reorg waiters), leaving MultiNodeTestContext with only the validator-node spawning and committee-convergence helpers. Set inboxLag: 2 as a base default (the intended value when pipelining). Add shared presets/helpers: buildMockGossipValidators, MOCK_GOSSIP_MULTI_VALIDATOR_OPTS, FAST_REORG_TIMING, defaultSlashingPenalties/withOnlyOffense. Rename multi-node/e2e_slashing -> slashing (git mv + describe titles + .test_patterns.yml + bootstrap.sh discovery). All 29 tests still import MultiNodeTestContext and pass unchanged.
…SingleNodeTestContext Relocate the 14 single-node-topology tests into multi-node/single-node/ and switch them onto SingleNodeTestContext. Apply the FAST_REORG_TIMING preset to l1_reorgs and the six optimistic_proving reorg blocks; drop now-redundant explicit inboxLag: 2. Merge the proving trio (multiple + empty_blocks_proof + long_proving_time) into single-node/proving.parallel.test.ts (per-it setup, keeping multiple's world-state- prune assertion as its own it). Co-locate partial_proof (its unique startProof path preserved) with partial_proof_multi_root, and manual_rollback with sync_after_reorg (node_reorg_recovery.test.ts, flattening the redundant nested describe). Update .test_patterns.yml + bootstrap.sh CI discovery for the new subfolder (proving.parallel keeps the 15m timeout the long-proving scenario needs).
…bfolders Create consensus/, prune/, ha/ under multi-node/ and relocate the multi-validator tests, adopting buildMockGossipValidators + MOCK_GOSSIP_MULTI_VALIDATOR_OPTS to drop the copy-pasted validator-builder block and the shared mock-gossip setup cluster. Merge simple_block_building + high_tps_block_building into consensus/block_building.parallel (per-it setup). Keep missed_l1_publish + orphan_block_prune as two prune/ files, reconciling orphan_block_prune's hand-rolled slot loop onto findSlotsWithProposers. Extract setupHaPairs for the two ha/ tests' shared pair wiring (ha_sync keeps its initial sequencer). Move equivocation + invalidate_block.parallel into slashing/. Add CI discovery globs for the new subfolders.
Merge duplicate_attestation_slash + duplicate_proposal_slash into
slashing/equivocation_slash.parallel.test.ts (shared harness opts, per-it setup),
preserving the proposal test's all-node offense poll + proposer-for-slot check verbatim.
Apply withOnlyOffense('slashDuplicateProposalPenalty') to equivocation (replacing its
~9-line manual penalty zero-out) and MOCK_GOSSIP_MULTI_VALIDATOR_OPTS +
buildMockGossipValidators to equivocation and invalidate_block (penalties left explicit
where the test relies on config defaults). Update .test_patterns.yml flake entry to the
merged file. The remaining ~10 e2e_p2p slashing/sentinel conversions are deferred: they
run real libp2p (P2PNetworkTest/createNodes) and the heaviest ones stub libp2p internals
the mock bus cannot reproduce — genuine work, not a folder move.
…lder map Document the SingleNodeTestContext -> MultiNodeTestContext split, the shared presets/helpers (FAST_REORG_TIMING, buildMockGossipValidators, MOCK_GOSSIP_MULTI_VALIDATOR_OPTS, defaultSlashingPenalties/withOnlyOffense, setupHaPairs), the single-node/consensus/prune/ha/ slashing subfolder map, the deferred top-level single-node category move, and that the MBPS dissolution is pending review. CI discovery for all subfolders was added in the prior commits; this confirms the mbps files stay discovered.
Dissolves the three MBPS test files in the multi-node category, relocating every unique assertion verbatim: - consensus/mbps.parallel.test.ts: the proposed-anchor monotonicity, L2->L1, L1->L2, non-validator re-exec/cold-sync, and deploy+call sub-slot ordering its from mbps.parallel; the proposer-pipelining offset + blob-promotion it from mbps.pipeline.parallel; and both redistribution its (budget redistribution kept verbatim, multiplier asymmetry) from mbps_redistribution. - prune/pipeline_prune.parallel.test.ts: the prune-on-skip-publish-under- pipelining it from mbps.pipeline.parallel. mbps.parallel it 1 (checkpointed-anchored MBPS block-count + proven) is dropped as redundant: the block-count and proven-checkpoint properties are already covered by consensus/block_building/high_tps. assertProposerPipelining is lifted onto SingleNodeTestContext (with a shared BlockProposedEvent type) so both relocated pipelining its reuse it. test_patterns flake entries for the prune and redistribution its move to their new paths. This is a behavior-preserving relocation, revertible as a unit.
…isk convergences
Introduce three named timing-only profiles in single_node_test_context.ts (re-exported
from multi_node_test_context.ts) to collapse the byte-identical timing clusters copied
across the multi-node category, plus a shared REORG_TIMING_BASE factored out of the reorg
profiles:
- REORG_TIMING_BASE = { aztecSlotDuration: 36, blockDurationMs: 8000, aztecEpochDuration: 4 }
- FAST_REORG_TIMING = base + { ethereumSlotDuration: 4, anvilSlotsInAnEpoch: 32 } (single-node reorg)
- MV_REORG_TIMING = base + { ethereumSlotDuration: 6, attestationPropagationTime: 0.5 } (MV prune/HA/equivocation)
- MV_CONSENSUS_TIMING = { ethereumSlotDuration: 12, aztecSlotDurationInL1Slots: 3, blockDurationMs: 6000 }
- MBPS_TIMING = { ethereumSlotDuration: 12, aztecSlotDuration: 72, blockDurationMs: 5500, aztecEpochDuration: 4, perBlockAllocationMultiplier: 8, aztecTargetCommitteeSize: 3 }
Profiles are spread BEFORE per-test overrides everywhere so per-test knobs win.
MV_CONSENSUS_TIMING keeps aztecSlotDurationInL1Slots:3 (not an explicit 36) to preserve
eth-coupling. MBPS_TIMING's JSDoc carries the A-914 rationale.
Reorg-profile unification attempt (FAST_REORG eth 4 -> 6 to share one L1 cadence with
MV_REORG): FELL BACK. Under eth=6 the single-node l1_reorgs suite times out on its
proof-removal and proof-restore reorg assertions (TimeoutError: checkpoint proven eq 0 /
checkpointed lte 1) because the longer L1 slot shifts the proof-submission window; 5/7
cases passed but those two failed. FAST_REORG stays at ethereumSlotDuration:4 (the shared
REORG_TIMING_BASE is kept either way). MV_REORG stays at eth=6.
Low-risk naming convergences (timing already equals the profile; byte-identical relocations):
- consensus/block_building simple + high_tps -> MV_CONSENSUS_TIMING (high_tps is 36s per Codex
correction; keeps fakeProcessingDelayPerTxMs + attestationPropagationTime:1)
- consensus/first_slot -> MV_CONSENSUS_TIMING (keeps epoch 32, committee, propagation, polling)
- consensus/proof_at_boundary -> MV_CONSENSUS_TIMING
- prune/orphan_block_prune + prune/missed_l1_publish -> MV_REORG_TIMING
- consensus/mbps setupMbps + setupPipeline -> MBPS_TIMING (redistribution KEPT on its own 4s/36s/6s timing)
- prune/pipeline_prune -> MBPS_TIMING
Verified-risk convergence in this commit:
- slashing/equivocation -> MV_REORG_TIMING (timing byte-identical; run because of slashing
offense-detection timing). HELD: test passed (DUPLICATE_PROPOSAL offense detected, chain healed).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…iles Verify-risk value convergences for the multi-node category. Every change alters a value (not pure naming); each affected test was run individually and sequentially. All HELD — no fallbacks. - ha/ha_sync: ethereumSlotDuration 4 -> 6 (adopt MV_REORG_TIMING). Keeps numberOfAccounts:1, min/maxTxs, pxeOpts.syncChainTip:'proposed', and the absence of skipInitialSequencer. HELD: 1/1 passed (23.7s). - ha/ha_checkpoint_handoff: adopt MV_REORG_TIMING + aztecEpochDuration 8 -> 4. The same-HA-pair consecutive-slot finder did not starve at epoch=4 (found slots 17/18 without excess EpochNotStable warping). HELD: 1/1 passed (95s). - single-node/missed_l1_slot: converge the 48s slot to 36 via ethereumSlotDuration:6 + aztecSlotDurationInL1Slots:6 (6*6=36), preserving the load-bearing 6-L1-slots-per-L2-slot invariant. Keeps perBlockAllocationMultiplier:8, useHardcodedAccount, mockGossipSubNetwork, blockDurationMs:8000. The bug-fix INITIALIZING_CHECKPOINT assertion and the account-fitting math hold at the 36s slot. HELD: 1/1 passed (77s). - slashing/invalidate_block: aztecSlotDuration 32 -> 36 (eth8/block6000 unchanged); keeps the 6-validator topology, anvil ports, slashing-round config, and committee-invalidation delay. The multi-block invalidation timetable holds. HELD: 9/9 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…I job No test files live directly under src/multi-node/ (all are in subfolders), so the src/multi-node/*.test.ts glob matched nothing. nullglob is not set in ci3/source_options, so the unmatched pattern leaked as a literal into the test command list, producing a job named multi-node/* that ran docker with --name 'multi-node_*' and failed with an invalid-container-name error (exit 125). The per-subfolder globs already cover every real multi-node test.
Both describes in block_building.parallel.test.ts had an
it('builds blocks without any errors'). CI's .parallel discovery
(bootstrap.sh extract_test_names) keys one isolated job per it title, so the
duplicate produced two byte-identical jobs sharing the same docker container
name; the second collided ("container is marked for removal and cannot be
started") and the test never ran. Rename to 'builds simple/high-tps blocks
without any errors' so each job has a unique NAME and its testNamePattern
selects exactly one describe.
The per-it CI job NAME for a .parallel.test.ts is derived from the test title, and becomes the docker container name via docker_isolate. The old sanitizer only replaced spaces, so a title with parentheses (e.g. the mbps multiplier-asymmetry it: "... (no fair-share re-execution)") produced an invalid container name and 'docker run --name' rejected it with exit 125 before the test could run, failing the whole fast CI run via fail-fast. Collapse every character outside docker's allowed set [a-zA-Z0-9_.-] to an underscore in both the simple and compat parallel-test command builders.
The proof-removal and proof-restore cases in l1_reorgs.parallel assert a transient post-prune node tip (proven == 0 / checkpointed <= 1) in the single-node self-building topology. After the proof is reorged out and the node prunes, its own sequencer rebuilds the pending chain immediately and a DELETE_FORK "Fork not found" world-state inconsistency disturbs the tip read, so under CI's L1 cadence the asserted value is never observed within the wait window. Both the original run and the flake retry time out, which makes the flaky mechanism hard-fail (it only tolerates fail-then-pass). The cases pass locally at the faster L1=4 cadence. This is pre-existing timing fragility, not introduced by the multi-node move: the file was previously skip:true (skip lifted in #23642) and FAST_REORG_TIMING resolves byte-identically to the prior eth4/slot36/block8000/epoch4/anvil32 config. Scoped per-it so the other five cases in the file remain covered.
…ce fix landed
The two proof-reorg cases ("prunes L2 blocks if a proof is removed due to an L1 reorg"
and "restores L2 blocks if a proof is added due to an L1 reorg") were skipped because
after an L1 reorg drove a single-node prune/rebuild, the node could settle on a stable
but non-canonical tip and never reach the awaited block, timing the waiter out.
8f03778 (fix(world-state): verify archive root in sync_block to reject divergent state)
targets exactly this failure class: it makes sync_block reject and roll back a divergent
archive root instead of silently committing an orphan block. The thrown error is caught
by the block stream's poll loop and retried, so the node can recover to the canonical
chain once the archiver re-syncs the reorged state rather than wedging on a wrong tip.
Un-skipping to let CI judge; revert this single commit if the cases still fail.
…divergence fix landed" This reverts commit 4b4d4a2.
…omment - Remove the "MBPS/pipelining dissolution — PENDING REVIEW" section from multi-node/README.md (the files were dissolved into consensus/mbps/ in prior PRs and sign-off is recorded). - Remove the stale top-level mbps row from the subfolder map. - Rewrite the single-node row and note that single-node is now a sibling category (done in §1). - Update the base-class hierarchy to reflect single_node_test_context.ts moving to ../single-node/. - Fix the stale "we will later promote" comment in SingleNodeTestContext's JSDoc. - Fix the stale "last block" comment in .test_patterns.yml redistribution entry to "last two blocks".
- git mv src/multi-node/single-node → src/single-node (all test files). - git mv src/multi-node/single_node_test_context.ts → src/single-node/single_node_test_context.ts. - Fix the two import sites in multi_node_test_context.ts: ./single_node_test_context.js → ../single-node/single_node_test_context.js. - Fix imports in all moved test files: ../../fixtures/ → ../fixtures/, ../single_node_test_context.js → ./single_node_test_context.js. - Update bootstrap.sh discovery glob from src/multi-node/single-node/*.test.ts to src/single-node/*.test.ts; re-point the 15m timeout case to single-node/proving/long_proving_time. - Re-point the three l1_reorgs entries in .test_patterns.yml (keep them skipped; add A-1266). - Add catch-all src/single-node/.*\.test\.ts owner entry in .test_patterns.yml. - Add src/single-node/README.md. - Update multi-node/README.md base-class hierarchy + subfolder map.
- Explode consensus/block_building.parallel.test.ts (2 describes) into:
consensus/block_building/simple.test.ts
consensus/block_building/high_tps.test.ts
with shared setup in consensus/block_building/setup.ts.
- Explode consensus/mbps.parallel.test.ts (7 describes) into:
consensus/mbps/proposed_anchor.test.ts
consensus/mbps/l2_to_l1.test.ts
consensus/mbps/l1_to_l2.test.ts
consensus/mbps/non_validator_sync.test.ts
consensus/mbps/deploy_and_call.test.ts
consensus/mbps/pipelining.test.ts
consensus/mbps/redistribution.parallel.test.ts (2 its — keeps .parallel)
with shared helpers in consensus/mbps/setup.ts.
- Update bootstrap.sh discovery to add consensus/block_building/*.test.ts
and consensus/mbps/*.test.ts globs.
- Re-point .test_patterns.yml redistribution entry to new path.
Splits proving.parallel, node_reorg_recovery, partial_proof_multi_root,
and equivocation_slash.parallel into per-describe subfolders:
- single-node/proving/{multiple,empty_blocks,long_proving_time}.test.ts
- single-node/reorg-recovery/{manual_rollback,sync_after_reorg}.test.ts
- single-node/partial-proofs/{multi_root,single_root}.test.ts
- multi-node/slashing/equivocation/{duplicate_proposal,duplicate_attestation}.test.ts
Each subfolder has a setup.ts for jest.setTimeout and shared re-exports.
Updates bootstrap.sh globs and .test_patterns.yml flake path accordingly.
…te explosions - single-node/*.ts: correct ../../shared/ and ../../test-wallet/ to ../shared/ and ../test-wallet/ - mbps/pipelining.test.ts: remove unused AztecNodeService, EndToEndContext, RegisteredValidator, TrackedSequencerEvent imports - mbps/redistribution.parallel.test.ts: remove unused AztecNodeService and MBPS_TIMING imports - equivocation/duplicate_*.test.ts: remove unused jest import (no jest.restoreAllMocks needed) - proving/multiple.test.ts: remove unused EndToEndContext import
…bps pieces The mbps explosion dropped the MBPS_TIMING import in pipelining.test.ts (re-exported by mbps/setup.ts) and degraded the archiver cast in redistribution.parallel.test.ts from 'as Archiver' to 'as any', which forced four downstream lambdas to acquire ': any' annotations. Restore the Archiver type import and the original untyped lambdas so the redistribution body is byte-identical to the pre-explosion source.
…categories
Move epochs_prune_when_cannot_build into multi-node/prune/, update its
describe title and base class from EpochsTestContext to MultiNodeTestContext.
The new it('proposer invalidates a non-broadcast checkpoint whose blob is
withheld') was already carried into invalidate_block.parallel.test.ts by the
rebase. Import consolidations for missed_l1_publish and orphan_block_prune
were already applied by earlier commits in this branch.
… instead of awaiting-checkpoints jobs The 'checkpoint reorg during proving' test gated top-tree proving by looking for a job in 'awaiting-checkpoints', but EpochSession.beforeProve flips the state to 'awaiting-root' *before* awaiting the beforeTopTreeProve hook. By the time the hook ran, no session was in 'awaiting-checkpoints', so the gate's early-return fired and proving was never actually blocked. The reorg then raced the live prove: in the losing ordering the session touched pruned block data, went terminal 'failed', and SessionManager dropped it (terminal sessions are not recreated), so the test's wait for a recreated smaller session timed out. Identify the gating session directly: scan the session manager for the live full session in 'awaiting-root' with >= 2 checkpoints (the one calling the hook), capture it, and only reorg once it has parked on the gate. This keeps the original session non-terminal across the prune so the production recreate-with-survivors path runs as intended. Test-only; no production change.
waitForNodeCheckpoint and waitForBlockNumber returned the matched numeric
value straight into retryUntil, whose contract is "truthy result = done". When
the correct answer is 0 (e.g. proven checkpoint 'eq' 0 after a proof is reorged
out, or checkpointed 'lte' 1 when the node prunes to 0), retryUntil reads 0 as
falsy and keeps polling until it times out.
This is the actual cause of the two deterministic CI failures in
single-node/l1_reorgs.parallel.test.ts ("prunes L2 blocks if a proof is
removed" times out on 'proven eq 0'; "restores L2 blocks if a proof is added"
times out on 'checkpointed lte 1'). The node prunes correctly in both cases;
only the test waiter was broken. The DELETE_FORK "Fork not found" log is a
benign artifact of the prune path, not the cause.
Wrap the matched value in an object so retryUntil always sees a truthy result,
then unwrap. retryUntil itself is left unchanged: many callers deliberately
return a falsy value (0, false) to mean "keep retrying".
Removes the two A-1266 skip entries now that the cases pass.
…plit slashing/recovery
…e current structure Describe each category by node topology and the primary behavior under test, with a per-subfolder line covering what each test exercises and which base context it uses. Fold the slashing sub-folder description into the multi-node README and remove its standalone file.
…e tests Pull the recurring hand-rolled steps in the single-node and multi-node e2e suites up into intent-revealing helpers on the test contexts and shared fixtures, so test bodies read as a sequence of high-level steps. Test contexts: - buildWindowTimestampForSlot / warpToBuildWindowForSlot / waitForBuildWindowForSlot: the pipelining "build window" warp/wait (one L1 block before an L2 slot), replacing the getTimestampForSlot(slot) - L1_BLOCK_TIME + warp math copied across the consensus/recovery/HA tests. - getSequencers / startSequencers, and watchNodeSequencerEvents: start and watch a committee of node sequencers without the .map(getSequencer) boilerplate. - waitForAllNodesToReachCheckpoint and waitForOffenseOnNodes: multi-node fan-out polls for a checkpointed tip and for slash-offense convergence. - waitForAllProversToSubmit and waitForSequencerState (idle-safe). - assertMultipleBlocksPerSlot now owns both wait modes (some-checkpoint-has-N-blocks and checkpointed-tip-reaches-block-X), collapsing the three divergent copies. Shared fixtures: - test-wallet/utils.ts: proveTxs / proveAndSendTxs (pre-prove a batch and send NO_WAIT) and startMempoolFeeder. - wait_helpers.ts: waitForBlocksAtSlots, waitForL2ToL1Witness; waitForNodeCheckpoint now takes a comparator lambda instead of a string-keyed comparator table. - ChainMonitor.waitForCheckpoint(predicate). Adopts these across the suites (dropping the matching REFACTOR markers) and lifts the shared block-production beforeEach into setupBlockProduction.
…ValidatorCluster - setupBlockProduction → setupSimpleBlockProduction (BlockProductionFixture → SimpleBlockProductionFixture) - setupMbps → setupBlockProductionWithProver (MbpsFixture → BlockProductionWithProverFixture) - MBPS_TIMING → WIDE_SLOT_TIMING - MULTI_VALIDATOR_CONSENSUS_TIMING → MULTI_VALIDATOR_BLOCK_PRODUCTION_TIMING - setupPipeline → setupBlobPromotion in blob_promotion.test.ts - Extract shared buildValidatorCluster helper in setup.ts - Update stale prose: "consensus timing" → "block-production timing", "consensus / prune" → "block-production / recovery", README MBPS/consensus → wide-slot / block-production
…NodeTestContext The upstream refactor (#24283) replaced AztecNodeService.createAndSync with the new createAztecNodeService factory in epochs_test.ts (the source that SingleNodeTestContext was extracted from in this PR). Port the same rename to the relocated file.
spalladino
force-pushed
the
spl/e2e-consolidation
branch
from
June 26, 2026 12:13
df1c814 to
8733749
Compare
spalladino
enabled auto-merge (squash)
June 26, 2026 12:21
PhilWindle
pushed a commit
that referenced
this pull request
Jun 29, 2026
…egories (#24310) Continues the e2e test consolidation building on the now-merged #24201. This PR **completes the `single-node` and `multi-node` categories** (no single-node- or multi-node-topology test remains outside them), settles `e2e_p2p/` down to transport-only, and adds a small `infra/` category. ## Context Leftovers from #24201's consolidation. `e2e_p2p/` had become a catch-all: offense-detection (slashing), sentinel, and governance tests sat alongside the genuine transport tests, and `e2e_slashing/` duplicated the slashing topology. Most of those aren't about real networking — they exercise proposals/attestations/offense detection that the in-memory `MockGossipSubNetwork` reproduces faithfully. Separately, a long tail of production-sequencer, prover, fee, cross-chain, and validator tests still lived as loose `e2e_*` files outside the categories the previous PR established. A note on `mockGossipSubNetwork`: the flag is inert on the `P2PNetworkTest` path — `setup_p2p_test.ts`'s `createNode` never wires the mock `p2pServiceFactory`, so tests that set it were in fact on real libp2p despite their comments. `MultiNodeTestContext` wires the mock bus genuinely, so each conversion moves the test onto a real in-memory bus. ## Approach - Move the slashing + sentinel tests into `multi-node/slashing/` and the governance tests into `multi-node/governance/`, converting each from `P2PNetworkTest` to `MultiNodeTestContext` (mock gossip). - Rename `e2e_p2p/` → `p2p/` once only transport tests remain, adding explicit CI globs. - Fold the single-sequencer production and prover tests into `single-node/`, over named setup factories on a none/fake/real prover axis, and make `FullProverTest` a `SingleNodeTestContext` sub-base. - Convert the remaining domain harnesses — `FeesTest`, `CrossChainMessagingTest`, and the multi-validator node test — onto the category bases, and move the public-testnet smoke into a new `infra/` category. After this, `single-node` and `multi-node` are complete. ## Hierarchy before → after - `e2e_p2p/{broadcasted_invalid_block_proposal_slash,data_withholding_slash,slash_veto_demo,inactivity_slash,inactivity_slash_with_consecutive_epochs,validators_sentinel,multiple_validators_sentinel,sentinel_status_slash}` and `e2e_slashing/{attested_invalid_proposal,broadcasted_invalid_checkpoint_proposal_slash}` → `multi-node/slashing/` (sentinel folds in here — it exists to drive slashing) - `e2e_p2p/{add_rollup,upgrade_governance_proposer}` → `multi-node/governance/` - `e2e_multi_validator/e2e_multi_validator_node` → `multi-node/block-production/multi_validator_node.parallel` (one node hosting multiple validator keys; converted to `MultiNodeTestContext`) - `e2e_p2p/` → `p2p/` (transport tests only; `reex.test.ts` deleted) - `e2e_slashing/` removed (now empty) - `e2e_block_building`, `e2e_debug_trace`, `e2e_multiple_blobs` → `single-node/block-building/` - `e2e_synching` → `single-node/sync/` - `e2e_sequencer/*`, plus `e2e_multi_eoa`, `e2e_publisher_funding_multi`, `e2e_sequencer_config` → `single-node/sequencer/` - `e2e_prover/{full,client}` → `single-node/prover/` - `e2e_fees/*` (incl. `e2e_fee_asset_price_oracle`) → `single-node/fees/` - `e2e_cross_chain_messaging/*` → `single-node/cross-chain/` - `e2e_bot` → `single-node/bot/` - `e2e_public_testnet/*` → `infra/` (new category) - `e2e_simple` → folded into `single-node/misc/node_block_api` (block-data-API check; the redundant deploy-and-prove `it` dropped) - `e2e_l1_with_wall_time` deleted (its multi-tx-block assertion is covered by `single-node/block-building/block_building`) ## Helpers introduced - **`single-node/setup.ts`** — two factories over `SingleNodeTestContext.setup`: `setupWithProver` (fake in-process prover — the single-node default) and `setupBlockProducer` (no prover; raises `aztecProofSubmissionEpochs` to `1024` so unproven blocks aren't pruned, PXE on `syncChainTip: 'proposed'`). - **`FullProverTest`, `FeesTest`, `CrossChainMessagingTest` now `extend SingleNodeTestContext`** via a `protected hydrateFromContext(context)` split out of `setup`, so each reuses the base's rollup/epoch-cache/chain-monitor/proven-chain-waiter/node-tracking/teardown machinery while keeping its own domain setup (real-BB env / FPC + gas state / `CrossChainTestHarness`). - The slashing shared helpers (`advanceToEpochBeforeProposer`, `awaitCommitteeExists`, `awaitOffenseDetected`, `awaitCommitteeKicked`, `findUpcomingProposerSlot`, `awaitProposalExecution`) move from `e2e_p2p/shared.ts` into `multi-node/slashing/setup.ts`, plus `submitTxsThroughNode`. - `multi-node/slashing/inactivity_setup.ts` (`InactivityTest`) and `multi-node/governance/setup.ts` (`GOVERNANCE_TIMING`). ## Semantic changes - Slashing/sentinel/governance/multi-validator tests move from the (effectively real-libp2p) `P2PNetworkTest` harness to the genuine in-memory mock-gossip bus of `MultiNodeTestContext`. Offense and assertion semantics are preserved verbatim; node creation maps `createNode(s)`/`createProverNode` onto `createValidatorNodeAt`/`createNonValidatorNode`/the context's fake prover; real-libp2p concerns (`waitForP2PMeshConnectivity`, bootstrap nodes, `removeInitialNode`) drop out. - `add_rollup` converts to `MultiNodeTestContext` too: its real-libp2p restart dance collapses to "stop old nodes/prover, warp, spawn new nodes/prover on the new rollup version" re-attaching to the shared in-memory bus; the new rollup is deployed with a genesis matching `context.genesis` so the fake prover can prove it. - The prod-seq tests run with **no prover** (`setupBlockProducer`); since the env default proof-submission window is `1`, the factory raises it to `1024` so blocks aren't pruned. Tests that depend on crossing the window (the `block_building` reorg cases, `synching`'s prune) pin `aztecProofSubmissionEpochs` explicitly. - `FeesTest` / `CrossChainMessagingTest` keep their bespoke prover-backed `setup()` opts byte-for-byte and preserve `catchUpProvenChain`/`advanceToEpochProven` exactly; only the shared environment machinery moves to the base. `FullProverTest` likewise keeps its real-BB env (L1 env epoch default, not the base's `6`). ## Test fixes - Several merged/converted files gain the `.parallel` suffix where they have >1 top-level `it`: `attested_invalid_proposal`, `broadcasted_invalid_checkpoint_proposal_slash`, `validators_sentinel`, `multi_validator_node`, and the fees/cross-chain files (`failures`, `gas_estimation`, `private_payments`; all five cross-chain files). - The stale `bootstrap.sh` per-test timeout case for `e2e_cross_chain_messaging/l1_to_l2` (which no longer matched the `.parallel` test name) is repointed to `single-node/cross-chain/l1_to_l2.parallel`. ## API changes Test-infra only. ## Open decisions (not blocking; for follow-up) - **`single-node/prover/client.test.ts` runs in no CI job.** Pre-existing — the old `e2e_prover/` dir was never globbed (only `full` had dedicated real/fake jobs); the move preserved that. Decide whether to start running it (e.g. add `src/single-node/prover/!(full).test.ts` to the CI globs). Tracked as A-1301. Part of A-1176
spalladino
added a commit
that referenced
this pull request
Jul 1, 2026
## Motivation A-1178 (#24281) measures each e2e test's wall-clock as four coarse buckets (`setupFnMs`, `beforeHooksMs`, `bodyMs`, `afterHooksMs`). That tells us a test is slow but not *where* it loses time. Most e2e tests spend their wall-clock in the same handful of operations — standing up the environment, waiting for a tx to be mined/checkpointed/proven, waiting for a committee to form or an offense to be detected, client-side proving of seed txs, and warp scans hunting for a proposer slot. This decomposes that time into **named spans** so a single aggregate query over a full CI run answers *"across the suite, how much total wall-clock goes into proving / spinning up nodes / waiting for checkpoints?"* — a ranked list of where to invest in speedups. It extends A-1178 and builds on the consolidation (#24201/#24310) and helpers (#24404) work that turned each repeated operation into one shared definition, so wrapping it once instruments every test with no per-test edits. This is **instrumentation / data-gathering only** — no test behavior or timing changes. ## Approach - A generic `testSpan(name, fn)` / `testSpanSync(name, fn)` wrapper (`fixtures/timing.ts`) records `{ owner, name, start, end }` into the shared collector installed by the timing environment. When `TEST_TIMING_FILE` is unset there is no collector, so `testSpan()` calls `fn()` directly — exactly zero-cost, with no clock reads. - At flush, the timing environment groups spans by owner then tag and computes four numbers per `(owner, tag)`: - `count` — multiplicity is itself a signal (waited for a checkpoint 14× points at a loop to batch). - `totalMs` — naive sum (correct for serial repeats). - `busyMs` — duration of the **union** of the spans' intervals; concurrency-correct, so a `Promise.all` of 12 concurrent 3s spans reads ~3s instead of ~36s. The `busyMs ≪ totalMs` signature flags work that is run serially where it could be parallel. - `maxMs` — longest single occurrence, to catch one pathological wait hiding in a cheap average. - The aggregates attach as an additive `spans` map on each `type:"test"` / `type:"suite"` JSONL line. `setupFnMs` / `teardownFnMs` are kept for back-compat, now derived from the `setup:env:<mode>` / `teardown:env` tags. - Tags follow a stable `category:label` taxonomy (`setup:`, `wait:`, `tx:`, `warp:`, `wallet:`, `deploy:`, `other:`), tagged **by concept, not by function** — e.g. every checkpoint waiter maps to `wait:checkpoint` — so a label is a forever aggregation key regardless of which helper a test happened to call. - **One clock.** All spans use `performance.now()`. I verified Node's `perf_hooks` clock is process-wide: a `performance.now()` taken inside a fresh `vm` context (the jest sandbox realm) shares the same monotonic origin as the host realm (delta ~0.002ms; a fresh `perf_hooks.performance` inside a separate context anchors to the same `timeOrigin`). Interval-merging for `busyMs` across the sandbox and host realms is therefore valid, so **`busyMs` is included** (no fallback to `totalMs`-only needed). - **Background loop.** `startMempoolFeeder` runs interleaved with arbitrary tests; without care its prove/send spans would smear onto whichever test was current when each round fired. Its production is run inside an `AsyncLocalStorage`-scoped owner override (`other:mempool-feeder`), which pins every span in its async call tree to a fixed owner — isolated from concurrent test-body spans. That owner matches no test/suite record, so the feeder's spans are cleanly excluded from the per-test view. ## Changes by phase - **Phase 0 — collector + `testSpan()` + flush aggregation.** New `fixtures/timing.ts` (`testSpan`/`testSpanSync`/`withTestSpanOwner`). `timing_env.mjs` collector generalized from `fnSpans` to a `spans` array; `finalizeAndFlush` groups by owner/tag, merges intervals for `busyMs`, attaches the `spans` map, and derives the back-compat fields. - **Phase 1 — shared waits + `setup:node`.** Wrapped the waiters in `fixtures/wait_helpers.ts` (`wait:proposed` / `wait:proven`, `wait:checkpoint` / `wait:proven-checkpoint`, `wait:tx-mined`, `wait:l2-to-l1-witness`, `wait:pending-tx`, `wait:sequencer-state`) and the wait methods on `SingleNodeTestContext` / `MultiNodeTestContext` (`wait:epoch`, `wait:slot`, `wait:proof-window`, `wait:node-sync`, `wait:proof-submitted`, `wait:offense`, the multi-node `wait:checkpoint`/`wait:proven-checkpoint`/`wait:block` convergence waiters). Wrapped `createNode`/`createProverNode` with `setup:node`. - **Phase 2 — decompose `setup:env`.** Cracked the opaque setup in `fixtures/setup.ts` into `setup:env:anvil`, `:l1-deploy`, `:sequencer-start`, `:prover-node`, `:pxe`, and `wallet:create`. The top-level `setup:env` is tagged with the prover mode (`setup:env:none` / `:fake` / `:real`) so the three factories are comparable. - **Phase 3 — tx leaf wraps.** `proveInteraction` → `tx:prove`, `ProvenTx.send` → `tx:send`; everything else (`proveTxs`/`proveAndSendTxs`/the submit helpers) aggregates through those leaves. `startMempoolFeeder` handled as above. - **Phase 4 — slashing/governance waiters.** Wrapped `awaitCommitteeExists` (`wait:committee`), `awaitOffenseDetected` (`wait:offense`), `awaitCommitteeKicked` (`wait:committee-kicked`), `awaitProposalExecution` (`wait:slash-execution`), and the `findUpcomingProposerSlot` / `advanceToEpochBeforeProposer` / `findSlotsWithProposers` scans (`warp:find-proposer`). - **Phase 5 — reporting.** The JSONL schema now carries the per-line `spans` map, and `TEST_TIMING_SPANS=1` emits one `type:"span"` line per occurrence (owner, name, ms) for deep dives. The repo now also ships a `track-e2e-times` skill (`yarn-project/.claude/skills/track-e2e-times/`) covering how to **collect and aggregate** the span timings locally: run the suite with `TEST_TIMING_FILE` set, find the per-worker JSONL, and print per-test sums plus the ranked span leaderboard (via the bundled `row.sh`). Only the step of publishing aggregate numbers to a running tracking log remains a separate out-of-band workflow. The leaderboard rollup is a small additive jq over the new `spans` maps: ``` jq -rs '[ .[] | select(.type=="test" or .type=="suite") | (.spans // {}) | to_entries[] ] | group_by(.key) | map({ tag: .[0].key, count: (map(.value.count) | add), busyMs: (map(.value.busyMs) | add), maxMs: (map(.value.maxMs) | max) }) | sort_by(-.busyMs) | .[] | "\(.busyMs)\t\(.count)\t\(.maxMs)\t\(.tag)"' ``` The existing `row.sh` is unaffected — it filters `type=="test"` and reads the unchanged `setupFnMs`/`bodyMs`/etc., ignoring the new `spans` key and `type:"span"` lines. ## Notes - `waitForEpoch`/`waitForSlot` (added to `@aztec/ethereum`'s `rollup_cheat_codes.ts` by #24404) are deliberately **not** instrumented in-package: that package should not depend on the e2e timing collector, and the `wait:epoch`/`wait:slot` concepts they cover are already captured at the e2e-context wrapper layer (`waitUntilEpochStarts`, the slot waiters). They have no e2e callers yet; a future call site picks them up via its context wrapper. - Spans do not partition `bodyMs` — a parent span includes its children, so `sum(spans) ≠ bodyMs`. Tagging is at the leaf wait/setup/tx level where additivity holds; the few intentional nests (e.g. `wait:committee-kicked` contains `wait:slash-execution`) carry distinct tags. Verified: `yarn build`, `yarn lint end-to-end`, `yarn format --check end-to-end` all pass. The flush aggregation (busyMs interval merge, back-compat derivation, feeder-owner exclusion, opt-in raw lines) is covered by unit tests in `end-to-end/src/shared/timing_env.test.ts`, which exercise the extracted `aggregateSpans` / `foldSpansInto` pure functions directly. Fixes A-1179 ## Example timing output Each e2e test run in CI emits one JSONL record per test and per suite, carrying a `spans` map keyed by `category:label` tag, where each value is `{count, totalMs, busyMs, maxMs}` (`busyMs` is the de-overlapped union duration, so concurrent occurrences of a span are counted once). An illustrative `type:"test"` line from CI run 28469687883 (commit de3635e): ```json { "suite": "optimistic.parallel", "type": "test", "name": "single-node/proving/optimistic happy path proves multiple epochs via checkpoint-driven flow", "status": "passed", "setupFnMs": 4707, "beforeHooksMs": 4717, "bodyMs": 327324, "teardownFnMs": 615, "afterHooksMs": 624, "totalMs": 332668, "startedAt": "2026-06-30T19:20:58.279Z", "spans": { "setup:env:anvil": { "count": 1, "totalMs": 82, "busyMs": 82, "maxMs": 82 }, "setup:env:l1-deploy": { "count": 1, "totalMs": 470, "busyMs": 470, "maxMs": 470 }, "setup:env:sequencer-start": { "count": 1, "totalMs": 3141, "busyMs": 3141, "maxMs": 3141 }, "setup:env:prover-node": { "count": 1, "totalMs": 139, "busyMs": 139, "maxMs": 139 }, "setup:env:pxe": { "count": 1, "totalMs": 391, "busyMs": 391, "maxMs": 391 }, "wallet:create": { "count": 1, "totalMs": 79, "busyMs": 79, "maxMs": 79 }, "setup:env:fake": { "count": 1, "totalMs": 4707, "busyMs": 4707, "maxMs": 4707 }, "wait:epoch": { "count": 5, "totalMs": 180210, "busyMs": 180210, "maxMs": 36067 }, "tx:prove": { "count": 4, "totalMs": 2123, "busyMs": 2123, "maxMs": 621 }, "tx:send": { "count": 4, "totalMs": 144305, "busyMs": 144305, "maxMs": 36124 }, "wait:proven-checkpoint": { "count": 4, "totalMs": 0, "busyMs": 0, "maxMs": 0 }, "wait:node-sync": { "count": 4, "totalMs": 401, "busyMs": 401, "maxMs": 101 }, "teardown:env": { "count": 1, "totalMs": 615, "busyMs": 615, "maxMs": 615 } }, "commit": "de3635e74eb89071b52939d1583d072ba2c3852c", "branch": "spl/a1179-track-common-spans", "runId": "28469687883" } ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The e2e suite under
end-to-end/src/had accumulated ad-hoc base classes with overlappingresponsibilities and large multi-
describefiles that were hard to navigate. The A-1175 survey settled onconsolidating the suite into a few setup/lifecycle categories, each backed by one base class that owns the
environment, with domain behavior composed on top. This PR delivers the
single-nodeandmulti-nodecategories, organizes each into behavior-named subfolders, and breaks the suite into small,one-
describefiles.Approach
single-nodeandmulti-nodeare sibling categories.SingleNodeTestContextowns the environment (onesequencer + optional fake prover, prover lifecycle, epoch/proof/reorg waiters);
MultiNodeTestContext extendsit, adding a validator committee, gossip helpers, and validator-registration sugar. Every file with more than
one top-level
describewas exploded into a subfolder of one-describefiles, with shared describe-levelsetup pulled into a co-located
setup.ts.Hierarchy: before → after
Before — one flat directory plus two strays:
After:
Criteria for the hierarchy
single-node(one sequencer) vsmulti-node(validator committeeis a flag, so a multi-validator test isn't a "proving" test just because a prover participates (hence
multi-node has no
proving/folder, single-node does).production traits, not toggleable features, so they no longer earn a folder or a filename.
helper/context ownership (e.g.
prune_when_cannot_buildstands up one solo sequencer, so it's a single-nodetest even though it came from the multi-node setup).
describeper file; shared setup in a co-locatedsetup.ts; the.parallelsuffix iff afile has more than one
it(preserving CI's per-itjob splitting without extra anvils).Tests removed / replaced
itcount 64 → 63): the MBPS case "builds multiple blocks per slot withtransactions anchored to checkpointed block", dropped as redundant with the retained "...anchored to
proposed blocks" variant (now
block-production/proposed_chain).e2e_p2pduplicate-slash tests were moved, not deleted, intomulti-node/slashing/(and switched to mock gossip). Several files were merged (combining files whilekeeping every distinct
it):mbps/l2_to_l1+l1_to_l2→cross_chain_messages.parallel;mbps/proposed_anchor+non_validator_sync→proposed_chain.parallel;prune/missed_l1_publish+orphan_block_prune→proposal_failure_recovery.parallel.l1_reorgs.parallelwas split along its
describes intol1-reorgs/blocks.parallel+messages.parallel.ittitles changed with their files: the twoblock_building"builds blocks without any errors"tests became "builds simple/high-tps blocks..."; "manually rolls back" → "...to an unfinalized block";
and the
pipeliningtest was renamedblob_promotion("promotion-disabled node fetches blobs while peersskip them, and the checkpoint proves"), trimmed to its unique blob-promotion + proving assertions (the
redundant MBPS/pipelining-offset re-checks are still asserted by
recovery/pipeline_prune).Helpers introduced
SingleNodeTestContext— base context (environment, prover lifecycle, epoch/proof/reorg waiters); ownsthe shared timing profiles
REORG_TIMING_BASE,FAST_REORG_TIMING,MULTI_VALIDATOR_REORG_TIMING,MULTI_VALIDATOR_BLOCK_PRODUCTION_TIMING, andWIDE_SLOT_TIMING(the 72s wide-slot cadence forprover-backed multi-block-per-slot tests).
MultiNodeTestContext— extends the above with the validator set, committee/gossip helpers, and thevalidator-registration API (
validatorAt/addressAt/privateKeyAt/createValidatorNodeAt,getSlashingContracts(), and aslasherEnabledsetup preset).fixtures/wait_helpers.ts— intent-revealing waiters:waitForBlockNumber,waitForProvenBlock,waitForNodeCheckpoint,waitForNodeProvenCheckpoint,waitForTxs.setup.tsper multi-file subfolder (proving,partial-proofs,l1-reorgs,recoveryunder single-node;
block-production,slashingunder multi-node). Theblock-productionsetup factors ashared
buildValidatorClusterspine into two presets:setupSimpleBlockProduction(lean, prover-lessliveness/throughput canary) and
setupBlockProductionWithProver(prover-backed, with a wallet + contract +fail-event tracking wired up for content/proving assertions).
A follow-up pass pulled the most-repeated hand-rolled steps up into higher-level helpers so test bodies
read as a sequence of named steps (the long tail of smaller helpers is omitted here):
warpToBuildWindowForSlot/waitForBuildWindowForSlot(on the context) — the pipelining "buildwindow" warp/wait (one L1 block before an L2 slot), replacing the
getTimestampForSlot(slot) - L1_BLOCK_TIME+ warp arithmetic copied across the consensus/recovery/HA tests.proveAndSendTxs(test-wallet/utils.ts) — pre-prove a batch of interactions and send themNO_WAIT, collapsing thetimesAsync(proveInteraction)+Promise.all(send)pair duplicated in ~13 tests.startSequencers/watchNodeSequencerEvents(on the context) — start and watch a committee of nodesequencers without the
nodes.map(getSequencer)boilerplate.waitForAllNodesToReachCheckpoint/waitForOffenseOnNodes(onMultiNodeTestContext) — themulti-node fan-out polls for a checkpointed tip and for slash-offense convergence.
assertMultipleBlocksPerSlotconsolidated onto one context method that owns both wait modes(some-checkpoint-has-N-blocks and checkpointed-tip-reaches-block-X), replacing three divergent copies.
Semantic changes (setups that differ)
of
e2e_p2p) — deterministic and faster.prune_when_cannot_buildmigrated fromMultiNodeTestContexttoSingleNodeTestContext— it alwaysstood up a single solo sequencer (zero extra validators), so it now uses only the base-class environment.
retryUntilcheckpoint / block-number polls replaced by the intent-revealing wait-helpersabove. (
retryUntilremains the primitive for tx-status and other custom predicates.)SingleNodeTestContext.Test fixes introduced
0fix (bbfbe69):waitForNodeCheckpoint/waitForBlockNumbernow wrap theirmatched value so a correct answer of
0is no longer read as falsy byretryUntil(which otherwise polls totimeout). This was the actual cause behind the
l1-reorgsproof-reorg cases that were temporarily skippedduring development (A-1266); they now run.
proving/optimistic.paralleldeflake: the top-tree-proving gate now captures the live prover sessiondirectly instead of polling for an
awaiting-checkpointsjob — the session flips toawaiting-rootbeforethe hook fires, so the old predicate never matched and the gate never engaged, letting proving race the
prune. Test-only.
API changes
Test-infrastructure only (no product/public API).
EpochsTestContext→MultiNodeTestContext(extends the newSingleNodeTestContext); the standaloneValidatorRegistrationHarnessis gone (folded intoMultiNodeTestContext);e2e_epochs/removed and theepochs_prefix dropped; all importers, the CItest-discovery globs in
bootstrap.sh, and the.test_patterns.ymlpath entries were updated to match.Part of A-1176 (consolidation roadmap) / A-1064.