Skip to content

refactor(e2e): consolidate the multi-node and single-node test categories - #24201

Merged
spalladino merged 38 commits into
merge-train/spartan-v5from
spl/e2e-consolidation
Jun 26, 2026
Merged

refactor(e2e): consolidate the multi-node and single-node test categories#24201
spalladino merged 38 commits into
merge-train/spartan-v5from
spl/e2e-consolidation

Conversation

@spalladino

@spalladino spalladino commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Motivation

The e2e suite under end-to-end/src/ had accumulated ad-hoc base classes with overlapping
responsibilities and large multi-describe files that were hard to navigate. The A-1175 survey settled on
consolidating the suite into a few setup/lifecycle categories, each backed by one base class that owns the
environment, with domain behavior composed on top. This PR delivers the single-node and
multi-node categories, organizes each into behavior-named subfolders, and breaks the suite into small,
one-describe files.

Approach

single-node and multi-node are sibling categories. SingleNodeTestContext owns the environment (one
sequencer + optional fake prover, prover lifecycle, epoch/proof/reorg waiters); MultiNodeTestContext extends
it, adding a validator committee, gossip helpers, and validator-registration sugar. Every file with more than
one top-level describe was exploded into a subfolder of one-describe files, with shared describe-level
setup pulled into a co-located setup.ts.

Hierarchy: before → after

Before — one flat directory plus two strays:

e2e_epochs/                 ~30 epochs_*.test.ts  +  epochs_test.ts (base context)
e2e_p2p/duplicate_attestation_slash.test.ts
e2e_p2p/duplicate_proposal_slash.test.ts

After:

single-node/                          # one sequencer (+ optional fake prover)
  single_node_test_context.ts         # base context + shared timing profiles
  proving/                            # epoch/proof lifecycle is the subject
    world_state_pruning  
    empty_blocks  
    long_proving_time  
    multi_proof
    optimistic.parallel  
    proof_fails.parallel  
    cross_chain_public_message  
    upload_failed_proof
  partial-proofs/                     # mid-epoch / multi-root / Outbox semantics
    single_root  
    multi_root
  l1-reorgs/                          # split along its describes
    blocks.parallel  
    messages.parallel
  recovery/                           # reorg + pending-chain recovery
    manual_rollback  
    sync_after_reorg  
    prune_when_cannot_build
  misc/
    missed_l1_slot                    # single-node sequencer sync/timetable regression

multi-node/                           # N validators on mock gossip
  multi_node_test_context.ts          # also owns validator registration (harness folded in)
  block-production/                   # happy-path committee production
    simple  
    high_tps  
    first_slot  
    proof_boundary.parallel  
    proposed_chain.parallel
    cross_chain_messages.parallel  
    deploy_and_call_ordering  
    blob_promotion  
    redistribution.parallel
  recovery/                           # detect a bad/withheld/conflicting proposal → recover
    proposal_failure_recovery.parallel  
    pipeline_prune  
    equivocation_recovery
  invalid-attestations/
    invalidate_block.parallel         # invalid checkpoints detected/invalidated, chain progresses
  high-availability/
    ha_sync  
    ha_checkpoint_handoff
  slashing/                           # pure offense detection
    duplicate_proposal  
    duplicate_attestation

Criteria for the hierarchy

  • Top level = topology / setup model: single-node (one sequencer) vs multi-node (validator committee
    • gossip) — it names the environment a test needs, never the feature it happens to touch.
  • Second level = the primary system behavior under test — not the shared setup, and not a flag. "Prover"
    is a flag, so a multi-validator test isn't a "proving" test just because a prover participates (hence
    multi-node has no proving/ folder, single-node does).
  • Third level only for behavior that is still genuinely exceptional. MBPS and pipelining are now default
    production traits, not toggleable features, so they no longer earn a folder or a filename.
  • Folder homogeneity: every test in a subfolder uses that subfolder's context — placement never follows
    helper/context ownership (e.g. prune_when_cannot_build stands up one solo sequencer, so it's a single-node
    test even though it came from the multi-node setup).
  • One top-level describe per file; shared setup in a co-located setup.ts; the .parallel suffix iff a
    file has more than one it (preserving CI's per-it job splitting without extra anvils).

Tests removed / replaced

  • One test removed (net it count 64 → 63): the MBPS case "builds multiple blocks per slot with
    transactions anchored to checkpointed block"
    , dropped as redundant with the retained "...anchored to
    proposed blocks"
    variant (now block-production/proposed_chain).
  • No other test removed. The two e2e_p2p duplicate-slash tests were moved, not deleted, into
    multi-node/slashing/ (and switched to mock gossip). Several files were merged (combining files while
    keeping every distinct it): mbps/l2_to_l1+l1_to_l2cross_chain_messages.parallel;
    mbps/proposed_anchor+non_validator_syncproposed_chain.parallel;
    prune/missed_l1_publish+orphan_block_pruneproposal_failure_recovery.parallel. l1_reorgs.parallel
    was split along its describes into l1-reorgs/blocks.parallel + messages.parallel.
  • A few it titles changed with their files: the two block_building "builds blocks without any errors"
    tests became "builds simple/high-tps blocks..."; "manually rolls back""...to an unfinalized block";
    and the pipelining test was renamed blob_promotion ("promotion-disabled node fetches blobs while peers
    skip them, and the checkpoint proves"
    ), trimmed to its unique blob-promotion + proving assertions (the
    redundant MBPS/pipelining-offset re-checks are still asserted by recovery/pipeline_prune).

Helpers introduced

  • SingleNodeTestContext — base context (environment, prover lifecycle, epoch/proof/reorg waiters); owns
    the shared timing profiles REORG_TIMING_BASE, FAST_REORG_TIMING, MULTI_VALIDATOR_REORG_TIMING,
    MULTI_VALIDATOR_BLOCK_PRODUCTION_TIMING, and WIDE_SLOT_TIMING (the 72s wide-slot cadence for
    prover-backed multi-block-per-slot tests).
  • MultiNodeTestContext — extends the above with the validator set, committee/gossip helpers, and the
    validator-registration API (validatorAt/addressAt/privateKeyAt/createValidatorNodeAt,
    getSlashingContracts(), and a slasherEnabled setup preset).
  • fixtures/wait_helpers.ts — intent-revealing waiters: waitForBlockNumber, waitForProvenBlock,
    waitForNodeCheckpoint, waitForNodeProvenCheckpoint, waitForTxs.
  • Co-located setup.ts per multi-file subfolder (proving, partial-proofs, l1-reorgs, recovery
    under single-node; block-production, slashing under multi-node). The block-production setup factors a
    shared buildValidatorCluster spine into two presets: setupSimpleBlockProduction (lean, prover-less
    liveness/throughput canary) and setupBlockProductionWithProver (prover-backed, with a wallet + contract +
    fail-event tracking wired up for content/proving assertions).

A follow-up pass pulled the most-repeated hand-rolled steps up into higher-level helpers so test bodies
read as a sequence of named steps (the long tail of smaller helpers is omitted here):

  • warpToBuildWindowForSlot / waitForBuildWindowForSlot (on the context) — the pipelining "build
    window" warp/wait (one L1 block before an L2 slot), replacing the getTimestampForSlot(slot) - L1_BLOCK_TIME + warp arithmetic copied across the consensus/recovery/HA tests.
  • proveAndSendTxs (test-wallet/utils.ts) — pre-prove a batch of interactions and send them
    NO_WAIT, collapsing the timesAsync(proveInteraction) + Promise.all(send) pair duplicated in ~13 tests.
  • startSequencers / watchNodeSequencerEvents (on the context) — start and watch a committee of node
    sequencers without the nodes.map(getSequencer) boilerplate.
  • waitForAllNodesToReachCheckpoint / waitForOffenseOnNodes (on MultiNodeTestContext) — the
    multi-node fan-out polls for a checkpointed tip and for slash-offense convergence.
  • assertMultipleBlocksPerSlot consolidated onto one context method that owns both wait modes
    (some-checkpoint-has-N-blocks and checkpointed-tip-reaches-block-X), replacing three divergent copies.

Semantic changes (setups that differ)

  • Slashing/equivocation tests run on the in-memory mock gossip bus instead of real libp2p (they moved out
    of e2e_p2p) — deterministic and faster.
  • prune_when_cannot_build migrated from MultiNodeTestContext to SingleNodeTestContext — it always
    stood up a single solo sequencer (zero extra validators), so it now uses only the base-class environment.
  • Hand-rolled retryUntil checkpoint / block-number polls replaced by the intent-revealing wait-helpers
    above. (retryUntil remains the primitive for tx-status and other custom predicates.)
  • Incidental per-test timing values converged onto the shared timing profiles owned by
    SingleNodeTestContext.

Test fixes introduced

  • Wait-helper falsy-0 fix (bbfbe69): waitForNodeCheckpoint / waitForBlockNumber now wrap their
    matched value so a correct answer of 0 is no longer read as falsy by retryUntil (which otherwise polls to
    timeout). This was the actual cause behind the l1-reorgs proof-reorg cases that were temporarily skipped
    during development (A-1266); they now run.
  • proving/optimistic.parallel deflake: the top-tree-proving gate now captures the live prover session
    directly instead of polling for an awaiting-checkpoints job — the session flips to awaiting-root before
    the hook fires, so the old predicate never matched and the gate never engaged, letting proving race the
    prune. Test-only.

API changes

Test-infrastructure only (no product/public API). EpochsTestContextMultiNodeTestContext (extends the new
SingleNodeTestContext); the standalone ValidatorRegistrationHarness is gone (folded into
MultiNodeTestContext); e2e_epochs/ removed and the epochs_ prefix dropped; all importers, the CI
test-discovery globs in bootstrap.sh, and the .test_patterns.yml path entries were updated to match.

Part of A-1176 (consolidation roadmap) / A-1064.

@spalladino spalladino added the ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure label Jun 20, 2026
@spalladino spalladino changed the title feat(e2e): pilot multi-node category (MultiNodeTestContext + helpers) refactor(e2e): consolidate the multi-node test category Jun 20, 2026
@spalladino
spalladino force-pushed the spl/e2e-consolidation branch from 6fbb112 to c2cd99f Compare June 23, 2026 20:36
@spalladino spalladino changed the title refactor(e2e): consolidate the multi-node test category refactor(e2e): consolidate the multi-node and single-node test categories Jun 23, 2026
@spalladino
spalladino force-pushed the spl/e2e-consolidation branch from e47bed6 to 4afe5c0 Compare June 24, 2026 12:57
@spalladino
spalladino force-pushed the spl/e2e-consolidation branch from e5d33c2 to df1c814 Compare June 25, 2026 17:32
PhilWindle pushed a commit that referenced this pull request Jun 26, 2026
## What

Adds per-test timing capture to the e2e suite (A-1178, Phase 1),
distinguishing **function-level** time inside our `setup()`/`teardown()`
from **hook-level** time across jest's before/after hooks, and uploads
one JSONL file per test to S3 with a `ci.sh` command to pull a run's
timings back.

This is the PR-independent subset that does not depend on the e2e
consolidation reorg (#24201). The reorg-gated pieces (`wait:*`/compute
body split, multi-node/prover-node start attribution) are deliberately
deferred — those files don't exist on this base.

Fixes A-1180

Fixes A-1181

## How

- **`yarn-project/end-to-end/src/shared/timing_env.mjs`** (new) — a
`TimingEnvironment` subclass of foundation's `CustomEnvironment`. Calls
`await super.handleTestEvent()` first (preserves the base env's
unhandled-rejection patching), then times jest-circus events
(`hook_start`/`hook_success`/`hook_failure` by hook type, `test_fn_*`
for the body, `test_start`/`test_done` for total). Installs a collector
on `this.global` and flushes JSONL on env `teardown`. Entirely gated on
`E2E_TIMING_FILE`; a pure delegate otherwise.
- **`package.json`** — `jest.testEnvironment` →
`./shared/timing_env.mjs`.
- **`fixtures/setup.ts`** — wraps the exported `setup()` and the inner
`teardown` closure (both `ctx.teardown()` and the exported
`teardown(context)` funnel through it) in try/finally, pushing their
durations to `globalThis.__e2eTimings`, tagged with the running test (or
`null` during `beforeAll`/`afterAll`).
- **`ci3/exec_test`** — exports `E2E_TIMING_FILE` and, after the test,
uploads the gzipped file via `cache_s3_transfer_to`.
- **`ci3/run_test_cmd`** — passes the attempt's log id to `exec_test` as
`E2E_LOG_ID` (tracks `rotate_log` on retries).
- **`ci.sh`** — new `e2e-timings <ci_log_id> <folder>` command: `aws s3
cp --recursive` the job's prefix and gunzip each file to
`<log_id>.jsonl`.
- `tsconfig.json` / `eslint.config.js` — exclude/ignore the env `.mjs`
(it imports foundation's env across packages and jest loads it from
source), matching how foundation already ignores its own
`src/jest/*.mjs`.

## JSONL schema (one line per test)

```jsonc
{ "suite":"e2e_fees", "name":"pays fee via private payment", "status":"passed",
  "commit":"abc1234", "branch":"merge-train/spartan-v5", "runId":"177...",
  "startedAt":"2026-06-24T12:00:02.000Z",
  "setupFnMs":42000, "beforeHooksMs":47000, "bodyMs":13000,
  "teardownFnMs":2500, "afterHooksMs":3000, "totalMs":63000 }
```

`beforeAll`/`afterAll` (once per describe) are emitted on a single
suite-scoped line with `name: null`.

## Examples from this PR's CI run

```json5
  // type:"test" — per-test, with beforeEach setup on the test line
  { "suite":"epochs_invalidate_block.parallel", "type":"test",
    "name":"e2e_epochs/epochs_invalidate_block proposer invalidates previous checkpoint…",
    "status":"passed", "setupFnMs":14330, "beforeHooksMs":19357, "bodyMs":179667,
    "teardownFnMs":732, "afterHooksMs":765, "totalMs":199797,
    "startedAt":"2026-06-25T12:58:34.643Z",
    "commit":"1925654a900…", "branch":"spl/a-1178-track-running-times", "runId":"28171484400" }
```

```json5
  // type:"suite" — the former name:null line (beforeAll/afterAll for the whole file)
  { "suite":"e2e_amm", "type":"suite", "name":null, "status":"passed",
    "setupFnMs":32591, "beforeHooksMs":61200, "teardownFnMs":102, "afterHooksMs":103, "totalMs":61303,
    "commit":"1925654a900…", "branch":"spl/a-1178-track-running-times", "runId":"28171484400" }
```

## Storage & retrieval

- S3 object:
`s3://aztec-ci-artifacts/logs/e2e-timings/<CI_LOG_ID>/<LOG_ID>.log.gz`
(gzipped JSONL).
- `CI_LOG_ID` = the job's top-level log id (the decimal id in a job's
`ci.aztec-labs.com/<id>` URL) — per-job, so it's collision-free under
grind.
- `LOG_ID` = the test's individual log id, the same id as the test's log
at `ci.aztec-labs.com/<log_id>`, so a test log maps straight to its
timing file.
- Download: `./ci.sh e2e-timings <ci_log_id> <folder>` →
`<folder>/<LOG_ID>.jsonl` per test (the `suite`/`name` fields inside
identify the test).

## Notes

- References A-1178 / A-1179 / A-1180 / A-1181 (Phase-1 subset; no
auto-closing keywords).
- No `ci-no-squash` label — single commit on merge.
spalladino and others added 20 commits June 26, 2026 09:08
…ared wait-helpers

First code commit of the e2e suite consolidation. Establishes the `multi-node`
category (N validators on the in-memory mock-gossip bus) by promoting the epochs
base class and extracting the highest-value shared wait-helpers.

Base class:
- Move end-to-end/src/e2e_epochs/epochs_test.ts ->
  end-to-end/src/multi-node/multi_node_test_context.ts and rename
  EpochsTestContext -> MultiNodeTestContext (EpochsTestOpts -> MultiNodeTestOpts).
  All 24 non-migrated epochs tests updated to import from the new location/name.

Migrated pilot tests into end-to-end/src/multi-node/:
- epochs_simple_block_building.test.ts (control)
- epochs_missed_l1_publish.test.ts (helper-extraction target)
- epochs_mbps.parallel.test.ts (stress + prover)

Helpers introduced/extended:
- fixtures/wait_helpers.ts: waitForBlockNumber / waitForProvenBlock, waitForTxs.
- ChainMonitor.waitUntilCheckpointProven (event-driven, mirrors waitUntilCheckpoint).
- MultiNodeTestContext.waitForAllNodes (+ waitForAllNodesToReachProvenCheckpoint /
  waitForAllNodesToReachBlockAtSlot): multi-node fan-out convergence.
- MultiNodeTestContext.findSlotsWithProposers: extracts the EpochNotStable
  slot-search/warp loop.
- MultiNodeTestContext.waitForSequencerEvent: one-shot sequencer-event waiter.

The migration changes only setup wiring and async-waiting style; no test assertions
were changed. Adds end-to-end/src/multi-node/README.md.
…s_ prefix

Relocates all 24 e2e_epochs/*.test.ts files into the multi-node/ category folder
established by the pilot commit, and drops the redundant epochs_ prefix from both
the filenames and the describe titles (now multi-node/<name>). Also retroactively
renames the 3 pilot files already in multi-node/ for consistency. e2e_epochs/ is
now empty and removed.

Rewires CI test discovery in end-to-end/bootstrap.sh: the epochs globs are replaced
with src/multi-node/!(long_proving_time).test.ts (plus the long_proving_time special
case), and the NAME derivation now strips the src/ prefix so it produces multi-node/
names for the new folder while leaving e2e_<dir>/<file> names unchanged. Re-points the
four .test_patterns.yml entries that referenced e2e_epochs paths / epochs_ test names to
the new multi-node/ paths.

No test assertions change — only file location, titles, and import/CI wiring.
Extracts a ValidatorRegistrationHarness (multi-node/validator_registration_harness.ts)
that composes MultiNodeTestContext: it registers a validator set on the in-memory
mock-gossip bus via initialValidators (genesis staking + validator-set-lag advance,
the mock-gossip replacement for P2PNetworkTest's MultiAdder/GSE flow), exposes the
per-validator keys/addresses, spawns validator nodes on the mock bus, and resolves the
slasher/slashing-proposer L1 contracts.

Converts the duplicate_attestation_slash and duplicate_proposal_slash tests from
P2PNetworkTest (which ran REAL libp2p despite mockGossipSubNetwork:true — the flag was
inert because setup_p2p_test.createNode never passed a p2pServiceFactory) to genuine
mock gossip, and relocates them into multi-node/e2e_slashing/. The generic offense/
proposer helpers in e2e_p2p/shared.ts are reused unchanged via cross-folder import.
No test assertions change — only the gossip transport, registration wiring, and location.

Wires src/multi-node/e2e_slashing/*.test.ts into CI discovery (end-to-end/bootstrap.sh)
and re-points the duplicate_proposal_slash flaky entry in .test_patterns.yml to the new path.
…ests to intent-revealing helpers

Add waitForNodeCheckpoint / waitForNodeProvenCheckpoint to fixtures/wait_helpers.ts,
a single-node checkpoint-number convergence wait supporting eq/gte/gt/lte/lt comparisons
(reorg/prune tests wait for the number to drop, not just rise). Convert l1_reorgs,
optimistic_proving, manual_rollback, partial_proof, and proof_public_cross_chain to the
new and existing helpers (waitForBlockNumber/waitForProvenBlock/waitUntilProvenCheckpointNumber),
removing hand-rolled retryUntil checkpoint/block polls and local getCheckpointNumber helpers.
… setup presets

Carve the prod-sequencer single-node lifecycle out of MultiNodeTestContext into a
new SingleNodeTestContext parent (environment, node spawning, prover lifecycle, and
the epoch/checkpoint/proof-window/reorg waiters), leaving MultiNodeTestContext with
only the validator-node spawning and committee-convergence helpers. Set inboxLag: 2
as a base default (the intended value when pipelining). Add shared presets/helpers:
buildMockGossipValidators, MOCK_GOSSIP_MULTI_VALIDATOR_OPTS, FAST_REORG_TIMING,
defaultSlashingPenalties/withOnlyOffense. Rename multi-node/e2e_slashing -> slashing
(git mv + describe titles + .test_patterns.yml + bootstrap.sh discovery). All 29
tests still import MultiNodeTestContext and pass unchanged.
…SingleNodeTestContext

Relocate the 14 single-node-topology tests into multi-node/single-node/ and switch
them onto SingleNodeTestContext. Apply the FAST_REORG_TIMING preset to l1_reorgs and
the six optimistic_proving reorg blocks; drop now-redundant explicit inboxLag: 2.
Merge the proving trio (multiple + empty_blocks_proof + long_proving_time) into
single-node/proving.parallel.test.ts (per-it setup, keeping multiple's world-state-
prune assertion as its own it). Co-locate partial_proof (its unique startProof path
preserved) with partial_proof_multi_root, and manual_rollback with sync_after_reorg
(node_reorg_recovery.test.ts, flattening the redundant nested describe). Update
.test_patterns.yml + bootstrap.sh CI discovery for the new subfolder (proving.parallel
keeps the 15m timeout the long-proving scenario needs).
…bfolders

Create consensus/, prune/, ha/ under multi-node/ and relocate the multi-validator tests,
adopting buildMockGossipValidators + MOCK_GOSSIP_MULTI_VALIDATOR_OPTS to drop the
copy-pasted validator-builder block and the shared mock-gossip setup cluster. Merge
simple_block_building + high_tps_block_building into consensus/block_building.parallel
(per-it setup). Keep missed_l1_publish + orphan_block_prune as two prune/ files,
reconciling orphan_block_prune's hand-rolled slot loop onto findSlotsWithProposers.
Extract setupHaPairs for the two ha/ tests' shared pair wiring (ha_sync keeps its
initial sequencer). Move equivocation + invalidate_block.parallel into slashing/.
Add CI discovery globs for the new subfolders.
Merge duplicate_attestation_slash + duplicate_proposal_slash into
slashing/equivocation_slash.parallel.test.ts (shared harness opts, per-it setup),
preserving the proposal test's all-node offense poll + proposer-for-slot check verbatim.
Apply withOnlyOffense('slashDuplicateProposalPenalty') to equivocation (replacing its
~9-line manual penalty zero-out) and MOCK_GOSSIP_MULTI_VALIDATOR_OPTS +
buildMockGossipValidators to equivocation and invalidate_block (penalties left explicit
where the test relies on config defaults). Update .test_patterns.yml flake entry to the
merged file. The remaining ~10 e2e_p2p slashing/sentinel conversions are deferred: they
run real libp2p (P2PNetworkTest/createNodes) and the heaviest ones stub libp2p internals
the mock bus cannot reproduce — genuine work, not a folder move.
…lder map

Document the SingleNodeTestContext -> MultiNodeTestContext split, the shared presets/helpers
(FAST_REORG_TIMING, buildMockGossipValidators, MOCK_GOSSIP_MULTI_VALIDATOR_OPTS,
defaultSlashingPenalties/withOnlyOffense, setupHaPairs), the single-node/consensus/prune/ha/
slashing subfolder map, the deferred top-level single-node category move, and that the MBPS
dissolution is pending review. CI discovery for all subfolders was added in the prior commits;
this confirms the mbps files stay discovered.
Dissolves the three MBPS test files in the multi-node category, relocating
every unique assertion verbatim:

- consensus/mbps.parallel.test.ts: the proposed-anchor monotonicity, L2->L1,
  L1->L2, non-validator re-exec/cold-sync, and deploy+call sub-slot ordering
  its from mbps.parallel; the proposer-pipelining offset + blob-promotion it
  from mbps.pipeline.parallel; and both redistribution its (budget
  redistribution kept verbatim, multiplier asymmetry) from mbps_redistribution.
- prune/pipeline_prune.parallel.test.ts: the prune-on-skip-publish-under-
  pipelining it from mbps.pipeline.parallel.

mbps.parallel it 1 (checkpointed-anchored MBPS block-count + proven) is dropped
as redundant: the block-count and proven-checkpoint properties are already
covered by consensus/block_building/high_tps.

assertProposerPipelining is lifted onto SingleNodeTestContext (with a shared
BlockProposedEvent type) so both relocated pipelining its reuse it. test_patterns
flake entries for the prune and redistribution its move to their new paths.

This is a behavior-preserving relocation, revertible as a unit.
…isk convergences

Introduce three named timing-only profiles in single_node_test_context.ts (re-exported
from multi_node_test_context.ts) to collapse the byte-identical timing clusters copied
across the multi-node category, plus a shared REORG_TIMING_BASE factored out of the reorg
profiles:

- REORG_TIMING_BASE = { aztecSlotDuration: 36, blockDurationMs: 8000, aztecEpochDuration: 4 }
- FAST_REORG_TIMING = base + { ethereumSlotDuration: 4, anvilSlotsInAnEpoch: 32 } (single-node reorg)
- MV_REORG_TIMING = base + { ethereumSlotDuration: 6, attestationPropagationTime: 0.5 } (MV prune/HA/equivocation)
- MV_CONSENSUS_TIMING = { ethereumSlotDuration: 12, aztecSlotDurationInL1Slots: 3, blockDurationMs: 6000 }
- MBPS_TIMING = { ethereumSlotDuration: 12, aztecSlotDuration: 72, blockDurationMs: 5500, aztecEpochDuration: 4, perBlockAllocationMultiplier: 8, aztecTargetCommitteeSize: 3 }

Profiles are spread BEFORE per-test overrides everywhere so per-test knobs win.
MV_CONSENSUS_TIMING keeps aztecSlotDurationInL1Slots:3 (not an explicit 36) to preserve
eth-coupling. MBPS_TIMING's JSDoc carries the A-914 rationale.

Reorg-profile unification attempt (FAST_REORG eth 4 -> 6 to share one L1 cadence with
MV_REORG): FELL BACK. Under eth=6 the single-node l1_reorgs suite times out on its
proof-removal and proof-restore reorg assertions (TimeoutError: checkpoint proven eq 0 /
checkpointed lte 1) because the longer L1 slot shifts the proof-submission window; 5/7
cases passed but those two failed. FAST_REORG stays at ethereumSlotDuration:4 (the shared
REORG_TIMING_BASE is kept either way). MV_REORG stays at eth=6.

Low-risk naming convergences (timing already equals the profile; byte-identical relocations):
- consensus/block_building simple + high_tps -> MV_CONSENSUS_TIMING (high_tps is 36s per Codex
  correction; keeps fakeProcessingDelayPerTxMs + attestationPropagationTime:1)
- consensus/first_slot -> MV_CONSENSUS_TIMING (keeps epoch 32, committee, propagation, polling)
- consensus/proof_at_boundary -> MV_CONSENSUS_TIMING
- prune/orphan_block_prune + prune/missed_l1_publish -> MV_REORG_TIMING
- consensus/mbps setupMbps + setupPipeline -> MBPS_TIMING (redistribution KEPT on its own 4s/36s/6s timing)
- prune/pipeline_prune -> MBPS_TIMING

Verified-risk convergence in this commit:
- slashing/equivocation -> MV_REORG_TIMING (timing byte-identical; run because of slashing
  offense-detection timing). HELD: test passed (DUPLICATE_PROPOSAL offense detected, chain healed).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…iles

Verify-risk value convergences for the multi-node category. Every change alters a value
(not pure naming); each affected test was run individually and sequentially. All HELD —
no fallbacks.

- ha/ha_sync: ethereumSlotDuration 4 -> 6 (adopt MV_REORG_TIMING). Keeps numberOfAccounts:1,
  min/maxTxs, pxeOpts.syncChainTip:'proposed', and the absence of skipInitialSequencer.
  HELD: 1/1 passed (23.7s).
- ha/ha_checkpoint_handoff: adopt MV_REORG_TIMING + aztecEpochDuration 8 -> 4. The
  same-HA-pair consecutive-slot finder did not starve at epoch=4 (found slots 17/18 without
  excess EpochNotStable warping). HELD: 1/1 passed (95s).
- single-node/missed_l1_slot: converge the 48s slot to 36 via ethereumSlotDuration:6 +
  aztecSlotDurationInL1Slots:6 (6*6=36), preserving the load-bearing 6-L1-slots-per-L2-slot
  invariant. Keeps perBlockAllocationMultiplier:8, useHardcodedAccount, mockGossipSubNetwork,
  blockDurationMs:8000. The bug-fix INITIALIZING_CHECKPOINT assertion and the
  account-fitting math hold at the 36s slot. HELD: 1/1 passed (77s).
- slashing/invalidate_block: aztecSlotDuration 32 -> 36 (eth8/block6000 unchanged); keeps the
  6-validator topology, anvil ports, slashing-round config, and committee-invalidation delay.
  The multi-block invalidation timetable holds. HELD: 9/9 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…I job

No test files live directly under src/multi-node/ (all are in subfolders),
so the src/multi-node/*.test.ts glob matched nothing. nullglob is not set in
ci3/source_options, so the unmatched pattern leaked as a literal into the test
command list, producing a job named multi-node/* that ran docker with
--name 'multi-node_*' and failed with an invalid-container-name error (exit 125).
The per-subfolder globs already cover every real multi-node test.
Both describes in block_building.parallel.test.ts had an
it('builds blocks without any errors'). CI's .parallel discovery
(bootstrap.sh extract_test_names) keys one isolated job per it title, so the
duplicate produced two byte-identical jobs sharing the same docker container
name; the second collided ("container is marked for removal and cannot be
started") and the test never ran. Rename to 'builds simple/high-tps blocks
without any errors' so each job has a unique NAME and its testNamePattern
selects exactly one describe.
The per-it CI job NAME for a .parallel.test.ts is derived from the test
title, and becomes the docker container name via docker_isolate. The old
sanitizer only replaced spaces, so a title with parentheses (e.g. the mbps
multiplier-asymmetry it: "... (no fair-share re-execution)") produced an
invalid container name and 'docker run --name' rejected it with exit 125
before the test could run, failing the whole fast CI run via fail-fast.

Collapse every character outside docker's allowed set [a-zA-Z0-9_.-] to an
underscore in both the simple and compat parallel-test command builders.
The proof-removal and proof-restore cases in l1_reorgs.parallel assert a transient
post-prune node tip (proven == 0 / checkpointed <= 1) in the single-node self-building
topology. After the proof is reorged out and the node prunes, its own sequencer rebuilds
the pending chain immediately and a DELETE_FORK "Fork not found" world-state inconsistency
disturbs the tip read, so under CI's L1 cadence the asserted value is never observed within
the wait window. Both the original run and the flake retry time out, which makes the flaky
mechanism hard-fail (it only tolerates fail-then-pass). The cases pass locally at the faster
L1=4 cadence.

This is pre-existing timing fragility, not introduced by the multi-node move: the file was
previously skip:true (skip lifted in #23642) and FAST_REORG_TIMING resolves byte-identically
to the prior eth4/slot36/block8000/epoch4/anvil32 config. Scoped per-it so the other five
cases in the file remain covered.
…ce fix landed

The two proof-reorg cases ("prunes L2 blocks if a proof is removed due to an L1 reorg"
and "restores L2 blocks if a proof is added due to an L1 reorg") were skipped because
after an L1 reorg drove a single-node prune/rebuild, the node could settle on a stable
but non-canonical tip and never reach the awaited block, timing the waiter out.

8f03778 (fix(world-state): verify archive root in sync_block to reject divergent state)
targets exactly this failure class: it makes sync_block reject and roll back a divergent
archive root instead of silently committing an orphan block. The thrown error is caught
by the block stream's poll loop and retried, so the node can recover to the canonical
chain once the archiver re-syncs the reorged state rather than wedging on a wrong tip.

Un-skipping to let CI judge; revert this single commit if the cases still fail.
…omment

- Remove the "MBPS/pipelining dissolution — PENDING REVIEW" section from multi-node/README.md
  (the files were dissolved into consensus/mbps/ in prior PRs and sign-off is recorded).
- Remove the stale top-level mbps row from the subfolder map.
- Rewrite the single-node row and note that single-node is now a sibling category (done in §1).
- Update the base-class hierarchy to reflect single_node_test_context.ts moving to ../single-node/.
- Fix the stale "we will later promote" comment in SingleNodeTestContext's JSDoc.
- Fix the stale "last block" comment in .test_patterns.yml redistribution entry to "last two blocks".
- git mv src/multi-node/single-node → src/single-node (all test files).
- git mv src/multi-node/single_node_test_context.ts → src/single-node/single_node_test_context.ts.
- Fix the two import sites in multi_node_test_context.ts:
  ./single_node_test_context.js → ../single-node/single_node_test_context.js.
- Fix imports in all moved test files: ../../fixtures/ → ../fixtures/,
  ../single_node_test_context.js → ./single_node_test_context.js.
- Update bootstrap.sh discovery glob from src/multi-node/single-node/*.test.ts
  to src/single-node/*.test.ts; re-point the 15m timeout case to single-node/proving/long_proving_time.
- Re-point the three l1_reorgs entries in .test_patterns.yml (keep them skipped; add A-1266).
- Add catch-all src/single-node/.*\.test\.ts owner entry in .test_patterns.yml.
- Add src/single-node/README.md.
- Update multi-node/README.md base-class hierarchy + subfolder map.
- Explode consensus/block_building.parallel.test.ts (2 describes) into:
    consensus/block_building/simple.test.ts
    consensus/block_building/high_tps.test.ts
  with shared setup in consensus/block_building/setup.ts.
- Explode consensus/mbps.parallel.test.ts (7 describes) into:
    consensus/mbps/proposed_anchor.test.ts
    consensus/mbps/l2_to_l1.test.ts
    consensus/mbps/l1_to_l2.test.ts
    consensus/mbps/non_validator_sync.test.ts
    consensus/mbps/deploy_and_call.test.ts
    consensus/mbps/pipelining.test.ts
    consensus/mbps/redistribution.parallel.test.ts  (2 its — keeps .parallel)
  with shared helpers in consensus/mbps/setup.ts.
- Update bootstrap.sh discovery to add consensus/block_building/*.test.ts
  and consensus/mbps/*.test.ts globs.
- Re-point .test_patterns.yml redistribution entry to new path.
Splits proving.parallel, node_reorg_recovery, partial_proof_multi_root,
and equivocation_slash.parallel into per-describe subfolders:

- single-node/proving/{multiple,empty_blocks,long_proving_time}.test.ts
- single-node/reorg-recovery/{manual_rollback,sync_after_reorg}.test.ts
- single-node/partial-proofs/{multi_root,single_root}.test.ts
- multi-node/slashing/equivocation/{duplicate_proposal,duplicate_attestation}.test.ts

Each subfolder has a setup.ts for jest.setTimeout and shared re-exports.
Updates bootstrap.sh globs and .test_patterns.yml flake path accordingly.
…te explosions

- single-node/*.ts: correct ../../shared/ and ../../test-wallet/ to ../shared/ and ../test-wallet/
- mbps/pipelining.test.ts: remove unused AztecNodeService, EndToEndContext, RegisteredValidator,
  TrackedSequencerEvent imports
- mbps/redistribution.parallel.test.ts: remove unused AztecNodeService and MBPS_TIMING imports
- equivocation/duplicate_*.test.ts: remove unused jest import (no jest.restoreAllMocks needed)
- proving/multiple.test.ts: remove unused EndToEndContext import
…bps pieces

The mbps explosion dropped the MBPS_TIMING import in pipelining.test.ts
(re-exported by mbps/setup.ts) and degraded the archiver cast in
redistribution.parallel.test.ts from 'as Archiver' to 'as any', which
forced four downstream lambdas to acquire ': any' annotations. Restore
the Archiver type import and the original untyped lambdas so the
redistribution body is byte-identical to the pre-explosion source.
…categories

Move epochs_prune_when_cannot_build into multi-node/prune/, update its
describe title and base class from EpochsTestContext to MultiNodeTestContext.
The new it('proposer invalidates a non-broadcast checkpoint whose blob is
withheld') was already carried into invalidate_block.parallel.test.ts by the
rebase. Import consolidations for missed_l1_publish and orphan_block_prune
were already applied by earlier commits in this branch.
… instead of awaiting-checkpoints jobs

The 'checkpoint reorg during proving' test gated top-tree proving by
looking for a job in 'awaiting-checkpoints', but EpochSession.beforeProve
flips the state to 'awaiting-root' *before* awaiting the beforeTopTreeProve
hook. By the time the hook ran, no session was in 'awaiting-checkpoints',
so the gate's early-return fired and proving was never actually blocked.
The reorg then raced the live prove: in the losing ordering the session
touched pruned block data, went terminal 'failed', and SessionManager
dropped it (terminal sessions are not recreated), so the test's wait for a
recreated smaller session timed out.

Identify the gating session directly: scan the session manager for the live
full session in 'awaiting-root' with >= 2 checkpoints (the one calling the
hook), capture it, and only reorg once it has parked on the gate. This keeps
the original session non-terminal across the prune so the production
recreate-with-survivors path runs as intended. Test-only; no production change.
waitForNodeCheckpoint and waitForBlockNumber returned the matched numeric
value straight into retryUntil, whose contract is "truthy result = done". When
the correct answer is 0 (e.g. proven checkpoint 'eq' 0 after a proof is reorged
out, or checkpointed 'lte' 1 when the node prunes to 0), retryUntil reads 0 as
falsy and keeps polling until it times out.

This is the actual cause of the two deterministic CI failures in
single-node/l1_reorgs.parallel.test.ts ("prunes L2 blocks if a proof is
removed" times out on 'proven eq 0'; "restores L2 blocks if a proof is added"
times out on 'checkpointed lte 1'). The node prunes correctly in both cases;
only the test waiter was broken. The DELETE_FORK "Fork not found" log is a
benign artifact of the prune path, not the cause.

Wrap the matched value in an object so retryUntil always sees a truthy result,
then unwrap. retryUntil itself is left unchanged: many callers deliberately
return a falsy value (0, false) to mean "keep retrying".

Removes the two A-1266 skip entries now that the cases pass.
…e current structure

Describe each category by node topology and the primary behavior under test,
with a per-subfolder line covering what each test exercises and which base
context it uses. Fold the slashing sub-folder description into the multi-node
README and remove its standalone file.
…e tests

Pull the recurring hand-rolled steps in the single-node and multi-node e2e
suites up into intent-revealing helpers on the test contexts and shared
fixtures, so test bodies read as a sequence of high-level steps.

Test contexts:
- buildWindowTimestampForSlot / warpToBuildWindowForSlot / waitForBuildWindowForSlot:
  the pipelining "build window" warp/wait (one L1 block before an L2 slot),
  replacing the getTimestampForSlot(slot) - L1_BLOCK_TIME + warp math copied
  across the consensus/recovery/HA tests.
- getSequencers / startSequencers, and watchNodeSequencerEvents: start and watch
  a committee of node sequencers without the .map(getSequencer) boilerplate.
- waitForAllNodesToReachCheckpoint and waitForOffenseOnNodes: multi-node fan-out
  polls for a checkpointed tip and for slash-offense convergence.
- waitForAllProversToSubmit and waitForSequencerState (idle-safe).
- assertMultipleBlocksPerSlot now owns both wait modes (some-checkpoint-has-N-blocks
  and checkpointed-tip-reaches-block-X), collapsing the three divergent copies.

Shared fixtures:
- test-wallet/utils.ts: proveTxs / proveAndSendTxs (pre-prove a batch and send
  NO_WAIT) and startMempoolFeeder.
- wait_helpers.ts: waitForBlocksAtSlots, waitForL2ToL1Witness; waitForNodeCheckpoint
  now takes a comparator lambda instead of a string-keyed comparator table.
- ChainMonitor.waitForCheckpoint(predicate).

Adopts these across the suites (dropping the matching REFACTOR markers) and lifts
the shared block-production beforeEach into setupBlockProduction.
…ValidatorCluster

- setupBlockProduction → setupSimpleBlockProduction (BlockProductionFixture → SimpleBlockProductionFixture)
- setupMbps → setupBlockProductionWithProver (MbpsFixture → BlockProductionWithProverFixture)
- MBPS_TIMING → WIDE_SLOT_TIMING
- MULTI_VALIDATOR_CONSENSUS_TIMING → MULTI_VALIDATOR_BLOCK_PRODUCTION_TIMING
- setupPipeline → setupBlobPromotion in blob_promotion.test.ts
- Extract shared buildValidatorCluster helper in setup.ts
- Update stale prose: "consensus timing" → "block-production timing", "consensus / prune" → "block-production / recovery", README MBPS/consensus → wide-slot / block-production
…NodeTestContext

The upstream refactor (#24283) replaced AztecNodeService.createAndSync with
the new createAztecNodeService factory in epochs_test.ts (the source that
SingleNodeTestContext was extracted from in this PR). Port the same rename
to the relocated file.
@spalladino
spalladino force-pushed the spl/e2e-consolidation branch from df1c814 to 8733749 Compare June 26, 2026 12:13
@spalladino
spalladino enabled auto-merge (squash) June 26, 2026 12:21
@spalladino
spalladino merged commit c1c3b0b into merge-train/spartan-v5 Jun 26, 2026
12 checks passed
@spalladino
spalladino deleted the spl/e2e-consolidation branch June 26, 2026 12:47
PhilWindle pushed a commit that referenced this pull request Jun 29, 2026
…egories (#24310)

Continues the e2e test consolidation building on the now-merged #24201.
This PR **completes the
`single-node` and `multi-node` categories** (no single-node- or
multi-node-topology test remains outside
them), settles `e2e_p2p/` down to transport-only, and adds a small
`infra/` category.

## Context

Leftovers from #24201's consolidation. `e2e_p2p/` had become a
catch-all: offense-detection (slashing),
sentinel, and governance tests sat alongside the genuine transport
tests, and `e2e_slashing/` duplicated the
slashing topology. Most of those aren't about real networking — they
exercise proposals/attestations/offense
detection that the in-memory `MockGossipSubNetwork` reproduces
faithfully. Separately, a long tail of
production-sequencer, prover, fee, cross-chain, and validator tests
still lived as loose `e2e_*` files
outside the categories the previous PR established.

A note on `mockGossipSubNetwork`: the flag is inert on the
`P2PNetworkTest` path — `setup_p2p_test.ts`'s
`createNode` never wires the mock `p2pServiceFactory`, so tests that set
it were in fact on real libp2p
despite their comments. `MultiNodeTestContext` wires the mock bus
genuinely, so each conversion moves the
test onto a real in-memory bus.

## Approach

- Move the slashing + sentinel tests into `multi-node/slashing/` and the
governance tests into
`multi-node/governance/`, converting each from `P2PNetworkTest` to
`MultiNodeTestContext` (mock gossip).
- Rename `e2e_p2p/` → `p2p/` once only transport tests remain, adding
explicit CI globs.
- Fold the single-sequencer production and prover tests into
`single-node/`, over named setup factories on a
none/fake/real prover axis, and make `FullProverTest` a
`SingleNodeTestContext` sub-base.
- Convert the remaining domain harnesses — `FeesTest`,
`CrossChainMessagingTest`, and the multi-validator
node test — onto the category bases, and move the public-testnet smoke
into a new `infra/` category. After
  this, `single-node` and `multi-node` are complete.

## Hierarchy before → after

-
`e2e_p2p/{broadcasted_invalid_block_proposal_slash,data_withholding_slash,slash_veto_demo,inactivity_slash,inactivity_slash_with_consecutive_epochs,validators_sentinel,multiple_validators_sentinel,sentinel_status_slash}`
and
`e2e_slashing/{attested_invalid_proposal,broadcasted_invalid_checkpoint_proposal_slash}`
→ `multi-node/slashing/` (sentinel folds in here — it exists to drive
slashing)
- `e2e_p2p/{add_rollup,upgrade_governance_proposer}` →
`multi-node/governance/`
- `e2e_multi_validator/e2e_multi_validator_node` →
`multi-node/block-production/multi_validator_node.parallel` (one node
hosting multiple validator keys; converted to `MultiNodeTestContext`)
- `e2e_p2p/` → `p2p/` (transport tests only; `reex.test.ts` deleted)
- `e2e_slashing/` removed (now empty)
- `e2e_block_building`, `e2e_debug_trace`, `e2e_multiple_blobs` →
`single-node/block-building/`
- `e2e_synching` → `single-node/sync/`
- `e2e_sequencer/*`, plus `e2e_multi_eoa`,
`e2e_publisher_funding_multi`, `e2e_sequencer_config` →
`single-node/sequencer/`
- `e2e_prover/{full,client}` → `single-node/prover/`
- `e2e_fees/*` (incl. `e2e_fee_asset_price_oracle`) →
`single-node/fees/`
- `e2e_cross_chain_messaging/*` → `single-node/cross-chain/`
- `e2e_bot` → `single-node/bot/`
- `e2e_public_testnet/*` → `infra/` (new category)
- `e2e_simple` → folded into `single-node/misc/node_block_api`
(block-data-API check; the redundant deploy-and-prove `it` dropped)
- `e2e_l1_with_wall_time` deleted (its multi-tx-block assertion is
covered by `single-node/block-building/block_building`)

## Helpers introduced

- **`single-node/setup.ts`** — two factories over
`SingleNodeTestContext.setup`: `setupWithProver` (fake
in-process prover — the single-node default) and `setupBlockProducer`
(no prover; raises
`aztecProofSubmissionEpochs` to `1024` so unproven blocks aren't pruned,
PXE on `syncChainTip: 'proposed'`).
- **`FullProverTest`, `FeesTest`, `CrossChainMessagingTest` now `extend
SingleNodeTestContext`** via a
`protected hydrateFromContext(context)` split out of `setup`, so each
reuses the base's

rollup/epoch-cache/chain-monitor/proven-chain-waiter/node-tracking/teardown
machinery while keeping its own
domain setup (real-BB env / FPC + gas state / `CrossChainTestHarness`).
- The slashing shared helpers (`advanceToEpochBeforeProposer`,
`awaitCommitteeExists`,
`awaitOffenseDetected`, `awaitCommitteeKicked`,
`findUpcomingProposerSlot`, `awaitProposalExecution`) move
from `e2e_p2p/shared.ts` into `multi-node/slashing/setup.ts`, plus
`submitTxsThroughNode`.
- `multi-node/slashing/inactivity_setup.ts` (`InactivityTest`) and
`multi-node/governance/setup.ts`
  (`GOVERNANCE_TIMING`).

## Semantic changes

- Slashing/sentinel/governance/multi-validator tests move from the
(effectively real-libp2p)
`P2PNetworkTest` harness to the genuine in-memory mock-gossip bus of
`MultiNodeTestContext`. Offense and
assertion semantics are preserved verbatim; node creation maps
`createNode(s)`/`createProverNode` onto
`createValidatorNodeAt`/`createNonValidatorNode`/the context's fake
prover; real-libp2p concerns
(`waitForP2PMeshConnectivity`, bootstrap nodes, `removeInitialNode`)
drop out.
- `add_rollup` converts to `MultiNodeTestContext` too: its real-libp2p
restart dance collapses to "stop old
nodes/prover, warp, spawn new nodes/prover on the new rollup version"
re-attaching to the shared in-memory
bus; the new rollup is deployed with a genesis matching
`context.genesis` so the fake prover can prove it.
- The prod-seq tests run with **no prover** (`setupBlockProducer`);
since the env default proof-submission
window is `1`, the factory raises it to `1024` so blocks aren't pruned.
Tests that depend on crossing the
window (the `block_building` reorg cases, `synching`'s prune) pin
`aztecProofSubmissionEpochs` explicitly.
- `FeesTest` / `CrossChainMessagingTest` keep their bespoke
prover-backed `setup()` opts byte-for-byte and
preserve `catchUpProvenChain`/`advanceToEpochProven` exactly; only the
shared environment machinery moves
to the base. `FullProverTest` likewise keeps its real-BB env (L1 env
epoch default, not the base's `6`).

## Test fixes

- Several merged/converted files gain the `.parallel` suffix where they
have >1 top-level `it`:
`attested_invalid_proposal`,
`broadcasted_invalid_checkpoint_proposal_slash`, `validators_sentinel`,
`multi_validator_node`, and the fees/cross-chain files (`failures`,
`gas_estimation`, `private_payments`;
  all five cross-chain files).
- The stale `bootstrap.sh` per-test timeout case for
`e2e_cross_chain_messaging/l1_to_l2` (which no longer
matched the `.parallel` test name) is repointed to
`single-node/cross-chain/l1_to_l2.parallel`.

## API changes

Test-infra only.

## Open decisions (not blocking; for follow-up)

- **`single-node/prover/client.test.ts` runs in no CI job.**
Pre-existing — the old `e2e_prover/` dir was
never globbed (only `full` had dedicated real/fake jobs); the move
preserved that. Decide whether to start
running it (e.g. add `src/single-node/prover/!(full).test.ts` to the CI
globs). Tracked as A-1301.

Part of A-1176
spalladino added a commit that referenced this pull request Jul 1, 2026
## Motivation

A-1178 (#24281) measures each e2e test's wall-clock as four coarse
buckets (`setupFnMs`, `beforeHooksMs`, `bodyMs`, `afterHooksMs`). That
tells us a test is slow but not *where* it loses time. Most e2e tests
spend their wall-clock in the same handful of operations — standing up
the environment, waiting for a tx to be mined/checkpointed/proven,
waiting for a committee to form or an offense to be detected,
client-side proving of seed txs, and warp scans hunting for a proposer
slot.

This decomposes that time into **named spans** so a single aggregate
query over a full CI run answers *"across the suite, how much total
wall-clock goes into proving / spinning up nodes / waiting for
checkpoints?"* — a ranked list of where to invest in speedups. It
extends A-1178 and builds on the consolidation (#24201/#24310) and
helpers (#24404) work that turned each repeated operation into one
shared definition, so wrapping it once instruments every test with no
per-test edits.

This is **instrumentation / data-gathering only** — no test behavior or
timing changes.

## Approach

- A generic `testSpan(name, fn)` / `testSpanSync(name, fn)` wrapper
(`fixtures/timing.ts`) records `{ owner, name, start, end }` into the
shared collector installed by the timing environment. When
`TEST_TIMING_FILE` is unset there is no collector, so `testSpan()` calls
`fn()` directly — exactly zero-cost, with no clock reads.
- At flush, the timing environment groups spans by owner then tag and
computes four numbers per `(owner, tag)`:
- `count` — multiplicity is itself a signal (waited for a checkpoint 14×
points at a loop to batch).
  - `totalMs` — naive sum (correct for serial repeats).
- `busyMs` — duration of the **union** of the spans' intervals;
concurrency-correct, so a `Promise.all` of 12 concurrent 3s spans reads
~3s instead of ~36s. The `busyMs ≪ totalMs` signature flags work that is
run serially where it could be parallel.
- `maxMs` — longest single occurrence, to catch one pathological wait
hiding in a cheap average.
- The aggregates attach as an additive `spans` map on each `type:"test"`
/ `type:"suite"` JSONL line. `setupFnMs` / `teardownFnMs` are kept for
back-compat, now derived from the `setup:env:<mode>` / `teardown:env`
tags.
- Tags follow a stable `category:label` taxonomy (`setup:`, `wait:`,
`tx:`, `warp:`, `wallet:`, `deploy:`, `other:`), tagged **by concept,
not by function** — e.g. every checkpoint waiter maps to
`wait:checkpoint` — so a label is a forever aggregation key regardless
of which helper a test happened to call.
- **One clock.** All spans use `performance.now()`. I verified Node's
`perf_hooks` clock is process-wide: a `performance.now()` taken inside a
fresh `vm` context (the jest sandbox realm) shares the same monotonic
origin as the host realm (delta ~0.002ms; a fresh
`perf_hooks.performance` inside a separate context anchors to the same
`timeOrigin`). Interval-merging for `busyMs` across the sandbox and host
realms is therefore valid, so **`busyMs` is included** (no fallback to
`totalMs`-only needed).
- **Background loop.** `startMempoolFeeder` runs interleaved with
arbitrary tests; without care its prove/send spans would smear onto
whichever test was current when each round fired. Its production is run
inside an `AsyncLocalStorage`-scoped owner override
(`other:mempool-feeder`), which pins every span in its async call tree
to a fixed owner — isolated from concurrent test-body spans. That owner
matches no test/suite record, so the feeder's spans are cleanly excluded
from the per-test view.

## Changes by phase

- **Phase 0 — collector + `testSpan()` + flush aggregation.** New
`fixtures/timing.ts` (`testSpan`/`testSpanSync`/`withTestSpanOwner`).
`timing_env.mjs` collector generalized from `fnSpans` to a `spans`
array; `finalizeAndFlush` groups by owner/tag, merges intervals for
`busyMs`, attaches the `spans` map, and derives the back-compat fields.
- **Phase 1 — shared waits + `setup:node`.** Wrapped the waiters in
`fixtures/wait_helpers.ts` (`wait:proposed` / `wait:proven`,
`wait:checkpoint` / `wait:proven-checkpoint`, `wait:tx-mined`,
`wait:l2-to-l1-witness`, `wait:pending-tx`, `wait:sequencer-state`) and
the wait methods on `SingleNodeTestContext` / `MultiNodeTestContext`
(`wait:epoch`, `wait:slot`, `wait:proof-window`, `wait:node-sync`,
`wait:proof-submitted`, `wait:offense`, the multi-node
`wait:checkpoint`/`wait:proven-checkpoint`/`wait:block` convergence
waiters). Wrapped `createNode`/`createProverNode` with `setup:node`.
- **Phase 2 — decompose `setup:env`.** Cracked the opaque setup in
`fixtures/setup.ts` into `setup:env:anvil`, `:l1-deploy`,
`:sequencer-start`, `:prover-node`, `:pxe`, and `wallet:create`. The
top-level `setup:env` is tagged with the prover mode (`setup:env:none` /
`:fake` / `:real`) so the three factories are comparable.
- **Phase 3 — tx leaf wraps.** `proveInteraction` → `tx:prove`,
`ProvenTx.send` → `tx:send`; everything else
(`proveTxs`/`proveAndSendTxs`/the submit helpers) aggregates through
those leaves. `startMempoolFeeder` handled as above.
- **Phase 4 — slashing/governance waiters.** Wrapped
`awaitCommitteeExists` (`wait:committee`), `awaitOffenseDetected`
(`wait:offense`), `awaitCommitteeKicked` (`wait:committee-kicked`),
`awaitProposalExecution` (`wait:slash-execution`), and the
`findUpcomingProposerSlot` / `advanceToEpochBeforeProposer` /
`findSlotsWithProposers` scans (`warp:find-proposer`).
- **Phase 5 — reporting.** The JSONL schema now carries the per-line
`spans` map, and `TEST_TIMING_SPANS=1` emits one `type:"span"` line per
occurrence (owner, name, ms) for deep dives. The repo now also ships a
`track-e2e-times` skill (`yarn-project/.claude/skills/track-e2e-times/`)
covering how to **collect and aggregate** the span timings locally: run
the suite with `TEST_TIMING_FILE` set, find the per-worker JSONL, and
print per-test sums plus the ranked span leaderboard (via the bundled
`row.sh`). Only the step of publishing aggregate numbers to a running
tracking log remains a separate out-of-band workflow. The leaderboard
rollup is a small additive jq over the new `spans` maps:

  ```
jq -rs '[ .[] | select(.type=="test" or .type=="suite") | (.spans // {})
| to_entries[] ]
    | group_by(.key)
    | map({ tag: .[0].key, count: (map(.value.count) | add),
busyMs: (map(.value.busyMs) | add), maxMs: (map(.value.maxMs) | max) })
| sort_by(-.busyMs) | .[] | "\(.busyMs)\t\(.count)\t\(.maxMs)\t\(.tag)"'
  ```

The existing `row.sh` is unaffected — it filters `type=="test"` and
reads the unchanged `setupFnMs`/`bodyMs`/etc., ignoring the new `spans`
key and `type:"span"` lines.

## Notes

- `waitForEpoch`/`waitForSlot` (added to `@aztec/ethereum`'s
`rollup_cheat_codes.ts` by #24404) are deliberately **not** instrumented
in-package: that package should not depend on the e2e timing collector,
and the `wait:epoch`/`wait:slot` concepts they cover are already
captured at the e2e-context wrapper layer (`waitUntilEpochStarts`, the
slot waiters). They have no e2e callers yet; a future call site picks
them up via its context wrapper.
- Spans do not partition `bodyMs` — a parent span includes its children,
so `sum(spans) ≠ bodyMs`. Tagging is at the leaf wait/setup/tx level
where additivity holds; the few intentional nests (e.g.
`wait:committee-kicked` contains `wait:slash-execution`) carry distinct
tags.

Verified: `yarn build`, `yarn lint end-to-end`, `yarn format --check
end-to-end` all pass. The flush aggregation (busyMs interval merge,
back-compat derivation, feeder-owner exclusion, opt-in raw lines) is
covered by unit tests in `end-to-end/src/shared/timing_env.test.ts`,
which exercise the extracted `aggregateSpans` / `foldSpansInto` pure
functions directly.

Fixes A-1179



## Example timing output

Each e2e test run in CI emits one JSONL record per test and per suite,
carrying a `spans` map keyed by `category:label` tag, where each value
is `{count, totalMs, busyMs, maxMs}` (`busyMs` is the de-overlapped
union duration, so concurrent occurrences of a span are counted once).
An illustrative `type:"test"` line from CI run 28469687883 (commit
de3635e):

```json
{
  "suite": "optimistic.parallel",
  "type": "test",
  "name": "single-node/proving/optimistic happy path proves multiple epochs via checkpoint-driven flow",
  "status": "passed",
  "setupFnMs": 4707,
  "beforeHooksMs": 4717,
  "bodyMs": 327324,
  "teardownFnMs": 615,
  "afterHooksMs": 624,
  "totalMs": 332668,
  "startedAt": "2026-06-30T19:20:58.279Z",
  "spans": {
    "setup:env:anvil": {
      "count": 1,
      "totalMs": 82,
      "busyMs": 82,
      "maxMs": 82
    },
    "setup:env:l1-deploy": {
      "count": 1,
      "totalMs": 470,
      "busyMs": 470,
      "maxMs": 470
    },
    "setup:env:sequencer-start": {
      "count": 1,
      "totalMs": 3141,
      "busyMs": 3141,
      "maxMs": 3141
    },
    "setup:env:prover-node": {
      "count": 1,
      "totalMs": 139,
      "busyMs": 139,
      "maxMs": 139
    },
    "setup:env:pxe": {
      "count": 1,
      "totalMs": 391,
      "busyMs": 391,
      "maxMs": 391
    },
    "wallet:create": {
      "count": 1,
      "totalMs": 79,
      "busyMs": 79,
      "maxMs": 79
    },
    "setup:env:fake": {
      "count": 1,
      "totalMs": 4707,
      "busyMs": 4707,
      "maxMs": 4707
    },
    "wait:epoch": {
      "count": 5,
      "totalMs": 180210,
      "busyMs": 180210,
      "maxMs": 36067
    },
    "tx:prove": {
      "count": 4,
      "totalMs": 2123,
      "busyMs": 2123,
      "maxMs": 621
    },
    "tx:send": {
      "count": 4,
      "totalMs": 144305,
      "busyMs": 144305,
      "maxMs": 36124
    },
    "wait:proven-checkpoint": {
      "count": 4,
      "totalMs": 0,
      "busyMs": 0,
      "maxMs": 0
    },
    "wait:node-sync": {
      "count": 4,
      "totalMs": 401,
      "busyMs": 401,
      "maxMs": 101
    },
    "teardown:env": {
      "count": 1,
      "totalMs": 615,
      "busyMs": 615,
      "maxMs": 615
    }
  },
  "commit": "de3635e74eb89071b52939d1583d072ba2c3852c",
  "branch": "spl/a1179-track-common-spans",
  "runId": "28469687883"
}
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants