Skip to content

test(e2e): consolidate more tests into multi-node and single-node categories - #24310

Merged
PhilWindle merged 33 commits into
merge-train/spartan-v5from
spl/e2e-consolidation-2
Jun 29, 2026
Merged

test(e2e): consolidate more tests into multi-node and single-node categories#24310
PhilWindle merged 33 commits into
merge-train/spartan-v5from
spl/e2e-consolidation-2

Conversation

@spalladino

@spalladino spalladino commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Continues the e2e test consolidation building on the now-merged #24201. This PR completes the
single-node and multi-node categories
(no single-node- or multi-node-topology test remains outside
them), settles e2e_p2p/ down to transport-only, and adds a small infra/ category.

Context

Leftovers from #24201's consolidation. e2e_p2p/ had become a catch-all: offense-detection (slashing),
sentinel, and governance tests sat alongside the genuine transport tests, and e2e_slashing/ duplicated the
slashing topology. Most of those aren't about real networking — they exercise proposals/attestations/offense
detection that the in-memory MockGossipSubNetwork reproduces faithfully. Separately, a long tail of
production-sequencer, prover, fee, cross-chain, and validator tests still lived as loose e2e_* files
outside the categories the previous PR established.

A note on mockGossipSubNetwork: the flag is inert on the P2PNetworkTest path — setup_p2p_test.ts's
createNode never wires the mock p2pServiceFactory, so tests that set it were in fact on real libp2p
despite their comments. MultiNodeTestContext wires the mock bus genuinely, so each conversion moves the
test onto a real in-memory bus.

Approach

  • Move the slashing + sentinel tests into multi-node/slashing/ and the governance tests into
    multi-node/governance/, converting each from P2PNetworkTest to MultiNodeTestContext (mock gossip).
  • Rename e2e_p2p/p2p/ once only transport tests remain, adding explicit CI globs.
  • Fold the single-sequencer production and prover tests into single-node/, over named setup factories on a
    none/fake/real prover axis, and make FullProverTest a SingleNodeTestContext sub-base.
  • Convert the remaining domain harnesses — FeesTest, CrossChainMessagingTest, and the multi-validator
    node test — onto the category bases, and move the public-testnet smoke into a new infra/ category. After
    this, single-node and multi-node are complete.

Hierarchy before → after

  • e2e_p2p/{broadcasted_invalid_block_proposal_slash,data_withholding_slash,slash_veto_demo,inactivity_slash,inactivity_slash_with_consecutive_epochs,validators_sentinel,multiple_validators_sentinel,sentinel_status_slash} and e2e_slashing/{attested_invalid_proposal,broadcasted_invalid_checkpoint_proposal_slash}multi-node/slashing/ (sentinel folds in here — it exists to drive slashing)
  • e2e_p2p/{add_rollup,upgrade_governance_proposer}multi-node/governance/
  • e2e_multi_validator/e2e_multi_validator_nodemulti-node/block-production/multi_validator_node.parallel (one node hosting multiple validator keys; converted to MultiNodeTestContext)
  • e2e_p2p/p2p/ (transport tests only; reex.test.ts deleted)
  • e2e_slashing/ removed (now empty)
  • e2e_block_building, e2e_debug_trace, e2e_multiple_blobssingle-node/block-building/
  • e2e_synchingsingle-node/sync/
  • e2e_sequencer/*, plus e2e_multi_eoa, e2e_publisher_funding_multi, e2e_sequencer_configsingle-node/sequencer/
  • e2e_prover/{full,client}single-node/prover/
  • e2e_fees/* (incl. e2e_fee_asset_price_oracle) → single-node/fees/
  • e2e_cross_chain_messaging/*single-node/cross-chain/
  • e2e_botsingle-node/bot/
  • e2e_public_testnet/*infra/ (new category)
  • e2e_simple → folded into single-node/misc/node_block_api (block-data-API check; the redundant deploy-and-prove it dropped)
  • e2e_l1_with_wall_time deleted (its multi-tx-block assertion is covered by single-node/block-building/block_building)

Helpers introduced

  • single-node/setup.ts — two factories over SingleNodeTestContext.setup: setupWithProver (fake
    in-process prover — the single-node default) and setupBlockProducer (no prover; raises
    aztecProofSubmissionEpochs to 1024 so unproven blocks aren't pruned, PXE on syncChainTip: 'proposed').
  • FullProverTest, FeesTest, CrossChainMessagingTest now extend SingleNodeTestContext via a
    protected hydrateFromContext(context) split out of setup, so each reuses the base's
    rollup/epoch-cache/chain-monitor/proven-chain-waiter/node-tracking/teardown machinery while keeping its own
    domain setup (real-BB env / FPC + gas state / CrossChainTestHarness).
  • The slashing shared helpers (advanceToEpochBeforeProposer, awaitCommitteeExists,
    awaitOffenseDetected, awaitCommitteeKicked, findUpcomingProposerSlot, awaitProposalExecution) move
    from e2e_p2p/shared.ts into multi-node/slashing/setup.ts, plus submitTxsThroughNode.
  • multi-node/slashing/inactivity_setup.ts (InactivityTest) and multi-node/governance/setup.ts
    (GOVERNANCE_TIMING).

Semantic changes

  • Slashing/sentinel/governance/multi-validator tests move from the (effectively real-libp2p)
    P2PNetworkTest harness to the genuine in-memory mock-gossip bus of MultiNodeTestContext. Offense and
    assertion semantics are preserved verbatim; node creation maps createNode(s)/createProverNode onto
    createValidatorNodeAt/createNonValidatorNode/the context's fake prover; real-libp2p concerns
    (waitForP2PMeshConnectivity, bootstrap nodes, removeInitialNode) drop out.
  • add_rollup converts to MultiNodeTestContext too: its real-libp2p restart dance collapses to "stop old
    nodes/prover, warp, spawn new nodes/prover on the new rollup version" re-attaching to the shared in-memory
    bus; the new rollup is deployed with a genesis matching context.genesis so the fake prover can prove it.
  • The prod-seq tests run with no prover (setupBlockProducer); since the env default proof-submission
    window is 1, the factory raises it to 1024 so blocks aren't pruned. Tests that depend on crossing the
    window (the block_building reorg cases, synching's prune) pin aztecProofSubmissionEpochs explicitly.
  • FeesTest / CrossChainMessagingTest keep their bespoke prover-backed setup() opts byte-for-byte and
    preserve catchUpProvenChain/advanceToEpochProven exactly; only the shared environment machinery moves
    to the base. FullProverTest likewise keeps its real-BB env (L1 env epoch default, not the base's 6).

Test fixes

  • Several merged/converted files gain the .parallel suffix where they have >1 top-level it:
    attested_invalid_proposal, broadcasted_invalid_checkpoint_proposal_slash, validators_sentinel,
    multi_validator_node, and the fees/cross-chain files (failures, gas_estimation, private_payments;
    all five cross-chain files).
  • The stale bootstrap.sh per-test timeout case for e2e_cross_chain_messaging/l1_to_l2 (which no longer
    matched the .parallel test name) is repointed to single-node/cross-chain/l1_to_l2.parallel.

API changes

Test-infra only.

Open decisions (not blocking; for follow-up)

  • single-node/prover/client.test.ts runs in no CI job. Pre-existing — the old e2e_prover/ dir was
    never globbed (only full had dedicated real/fake jobs); the move preserved that. Decide whether to start
    running it (e.g. add src/single-node/prover/!(full).test.ts to the CI globs). Tracked as A-1301.

Part of A-1176

@spalladino spalladino changed the title test(e2e): consolidation phase 2 — slashing/sentinel/governance to multi-node, p2p rename, single-node prod-seq test(e2e): consolidate slashing/sentinel/governance to multi-node, p2p rename, single-node prod-seq Jun 25, 2026
@spalladino spalladino changed the title test(e2e): consolidate slashing/sentinel/governance to multi-node, p2p rename, single-node prod-seq test(e2e): consolidate slashing/sentinel/governance into multi-node, prod-seq + prover into single-node, rename p2p Jun 25, 2026
@spalladino spalladino added S-do-not-merge Status: Do not merge this PR ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure labels Jun 25, 2026
@spalladino
spalladino force-pushed the spl/e2e-consolidation branch from df1c814 to 8733749 Compare June 26, 2026 12:13
Base automatically changed from spl/e2e-consolidation to merge-train/spartan-v5 June 26, 2026 12:47
@spalladino spalladino changed the title test(e2e): consolidate slashing/sentinel/governance into multi-node, prod-seq + prover into single-node, rename p2p test(e2e): consolidate more tests into multi-node and single-node categories Jun 26, 2026
@spalladino
spalladino force-pushed the spl/e2e-consolidation-2 branch from 9fe3255 to a218ac8 Compare June 26, 2026 13:11
@spalladino spalladino removed the S-do-not-merge Status: Do not merge this PR label Jun 26, 2026
Relocate the four offense-detection slashing tests that already ran on the
in-memory mock-gossip bus from e2e_p2p/ and e2e_slashing/ into
multi-node/slashing/, swapping P2PNetworkTest for MultiNodeTestContext. Move the
slashing-specific shared helpers (advanceToEpochBeforeProposer,
awaitCommitteeExists, awaitOffenseDetected, awaitCommitteeKicked,
findUpcomingProposerSlot, awaitProposalExecution) from e2e_p2p/shared.ts into
multi-node/slashing/setup.ts and repoint the duplicate_* imports.

Offense and assertion semantics are kept verbatim. Validator keys align because
buildMockGossipValidators derives from getPrivateKeyFromIndex(i + 3), matching
the ATTESTER_PRIVATE_KEYS_START_INDEX convention.
…MultiNodeTestContext

Relocate data_withholding_slash and the two inactivity-slash tests from e2e_p2p/
into multi-node/slashing/, converting them from the real-libp2p P2PNetworkTest
harness to the mock-gossip MultiNodeTestContext. Replace the standalone
P2PInactivityTest base with an InactivityTest fixture over the context's
fake-prover topology, and add a submitTxsThroughNode helper that repoints the
context wallet at a node (the mock-gossip equivalent of submitTransactions).

Offense and assertion semantics are preserved verbatim.
…ontext

Relocate slash_veto_demo and the three sentinel tests (validators_sentinel,
multiple_validators_sentinel, sentinel_status_slash) from e2e_p2p/ into
multi-node/slashing/, converting them from P2PNetworkTest to the mock-gossip
MultiNodeTestContext. validators_sentinel gains the .parallel suffix (4 top-level
its). The late-joining fresh node becomes a createNonValidatorNode sentinel.

Trim e2e_p2p/shared.ts down to the p2p tx helpers now that the slashing helpers
live in multi-node/slashing/setup.ts. Offense/assertion semantics preserved.
Relocate the two L1-governance tests out of e2e_p2p/ into a new
multi-node/governance/ folder. upgrade_governance_proposer converts to the
mock-gossip MultiNodeTestContext (MOCK_GOSSIP_MULTI_VALIDATOR_OPTS +
GOVERNANCE_TIMING). add_rollup stays on the P2PNetworkTest harness for now (its
bootstrap-node restart and new-rollup node migration depend on real libp2p);
only its imports and describe name are repointed. Mock conversion of add_rollup
is deferred.

Add the src/multi-node/governance/*.test.ts CI globs in bootstrap.sh, carry
add_rollup's 20m timeout case to the new path, repoint the
attested_invalid_proposal .test_patterns.yml entry, drop the stale
valid_epoch_pruned_slash entry, and document the slashing/governance folders in
the multi-node README.
Now that the slashing/sentinel/governance tests have moved to multi-node/, e2e_p2p/
holds only real-libp2p transport tests, so rename the folder to p2p/ for category
consistency and delete reex.test.ts (pays setup cost for zero tests).

The src/e2e_!(prover)/*.test.ts auto-glob stops matching after the rename, so add
explicit src/p2p/*.test.ts globs (and repoint the reqresp glob) in both bootstrap.sh
test arrays, making the compat name derivation robust to the slash-bearing p2p path.
Repoint the blanket src/e2e_p2p/.* flake pattern to src/p2p/.*, the add_rollup harness
import, and the AUTOMINE preset doc comment. reqresp_no_handshake stays until its flake
is fixed.
The converted test relies on test.teardown() to stop nodes, so the outer nodes
variable (and the AztecNodeService import) are no longer read.
…llers

Introduce single-node/setup.ts with setupWithProver, a thin wrapper over
SingleNodeTestContext.setup carrying today's fake-prover defaults, and migrate
every single-node call site (proving, partial-proofs, l1-reorgs, recovery, misc)
from SingleNodeTestContext.setup(...) to setupWithProver(...). Pure rename with no
behavior change; the named factory establishes the single-node setup surface over
the prover axis (the no-prover setupBlockProducer and real-BB setupWithRealProver
factories, plus relocating the prod-seq and prover test files, follow separately).
MultiNodeTestContext.setup is left untouched.
Moves multi-node/governance/add_rollup off P2PNetworkTest (real libp2p) onto
the in-memory MockGossipSubNetwork, matching the sibling governance and
slashing conversions. On the mock bus there is no bootstrap node or peer
discovery, so the real-libp2p restart dance (stop/clear/re-add the bootstrap
node, let new peers rediscover) collapses to: stop the old nodes/prover, warp,
then spawn new nodes/prover on the new rollup version — they simply re-attach
to the shared bus.

The new rollup is deployed with a genesis archive root derived from the same
funded-account set and timestamp the context used, so its genesis matches
context.genesis. That lets the fake prover node (which always runs on
context.genesis) prove the new rollup's checkpoints. Nodes repoint at the new
rollup via rollupVersion alone; the aztec-node factory re-derives the L1
contract addresses from the registry, so the redundant l1Contracts override is
dropped. Every governance assertion (rollup-add, migration, post-migration
state, cross-chain bridging on both rollups) is preserved verbatim.
Move e2e_block_building, e2e_debug_trace, and e2e_multiple_blobs into
single-node/block-building/ and convert them from raw setup() onto
SingleNodeTestContext via the setupBlockProducer (prover-less) factory.
Preserve each describe's behavior: block_building keeps
aztecProofSubmissionEpochs=1 in its reorg describe so the prune/reinclude
still fires, and multiple_blobs swaps its hand-rolled IDLE poll for the
context's waitForSequencerState helper.
Move the four e2e_sequencer tests (escape_hatch_vote_only, gov_proposal,
reload_keystore, slasher_config) into single-node/sequencer/ and convert
them onto SingleNodeTestContext via setupBlockProducer. Each preserves its
explicit opts (committee size, slot/epoch durations, proof-submission
window, governance round/quorum) and its PXE syncChainTip semantics
('proposed', matching the original raw setup() default). gov_proposal keeps
its .parallel suffix; the now-empty e2e_sequencer/ directory is removed.
Move e2e_synching into single-node/sync/ and convert its raw setup() calls
onto SingleNodeTestContext via setupBlockProducer; its prune/reorg coverage
is L1-reorg- and manual-markAsProven-driven (no prover), and it pins
aztecProofSubmissionEpochs=1 to keep the prune-timeliness window the
fixture math was tuned against.

Repoint the bootstrap.sh test-list array: add the
single-node/{block-building,sequencer,sync}/*.test.ts globs, drop the
hardcoded e2e_block_building line and its exclusion from the generic
e2e_!(...) pattern, and carry its 25m timeout + AVM-input dump to
single-node/block-building/block_building via the case mechanism (the
loop's set_dump_avm already applies per test). Extend the
avm_check_circuit glob one level deeper for the 3-segment dump path.
Move e2e_prover/{full,client}.test.ts into single-node/prover/ and drop
the redundant describe-title prefixes (full_prover/client_prover ->
single-node/prover/{full,client}). Repoint the two special CI jobs for
the heavy real/fake full-prover run at the new path, preserving the
CI_FULL real (CPUS=16:MEM=96g:TIMEOUT=20m, e2e_prover_full_real) vs fake
(FAKE_PROOFS=1, e2e_prover_full_fake) split. The prover/ subfolder is
intentionally left out of the generic test glob, matching today: full
runs only via its dedicated jobs and client runs in no CI job.
Make FullProverTest extend SingleNodeTestContext so the real-prover suite
joins the single-node category over the none/fake/real prover axis. Extract
the context-hydration tail of SingleNodeTestContext.setup into a protected
hydrateFromContext(context) that reads slot/epoch durations from the resolved
context.config; FullProverTest reuses it (and super.teardown) for the rollup /
epoch-cache / chain-monitor / node-tracking / teardown machinery.

The environment build stays byte-for-byte today's: FullProverTest still calls
the raw setup(0, {...}) with its own opts (realVerifier L1, fundSponsoredFPC,
2 funded schnorr accounts, no top-level realProofs) rather than routing through
the base's default node config, since the base hardcodes aztecEpochDuration:6
whereas the real-BB job runs on the L1 env default. Everything prover-specific
is kept verbatim: the real BB prover-node config, the TokenSimulator harness,
the snapshot/account setup, the epoch warp + markAsProven, and the proven PXE.
The prover-node-before-BB-destroy teardown order is preserved.
The e2e_prover/ directory moved to single-node/prover/ in the Group B
consolidation, so the !(prover) extglob exclusion on src/e2e_*/*.test.ts
no longer excludes anything. Simplify to src/e2e_*/*.test.ts; the matched
set is identical (153 files before and after).
…ropose revert

The block-building conversion moved debug_trace onto setupBlockProducer /
SingleNodeTestContext.setup, which defaults aztecEpochDuration to 6. The
original raw setup() used the production default of 32. With epoch length 6 an
epoch boundary lands at slot 6; the proposer-corruption test intercepts every
propose and polls long enough to cross it, where proposer selection changes for
the new epoch and the propose silently reverts (allowFailure:true inside the
multicall -> empty logs -> propose_action_not_successful), so blocks stop being
checkpointed and the 60s block-number wait times out. Pin aztecEpochDuration:32
to restore the single-epoch run the test had before the move.
The prod-seq tests were written against the raw setup() helper's 32-slot
production epoch. After folding them onto setupBlockProducer the context's
own aztecEpochDuration default of 6 applied, landing an epoch boundary
mid-test where proposer selection changes and the boundary-slot propose
silently reverts (no checkpoint lands, the chain stops). debug_trace caught
it in CI; block_building, multiple_blobs, synching, gov_proposal,
reload_keystore and slasher_config relied on the same default and were
masked by fail-fast. Restore the 32-slot epoch in the factory; tests that
need a shorter epoch still override it.
…ion collision

The MultiNodeTestContext conversion deployed the new rollup with a genesisArchiveRoot
equal to the context's primary rollup (same funded set + timestamp). Since the on-chain
rollup version is uint32(keccak256(abi.encode(config, genesisState))), the new rollup's
version collided with the primary's, so executing the governance proposal reverted with
Governance__CallFailed -> Registry__RollupAlreadyRegistered.

Offset the new rollup's genesis timestamp by 1s so its archive root (and thus version)
differs, and repoint context.genesis at that genesis before spawning the migrated
nodes/prover so the fake prover can still prove the new rollup's checkpoints. This
restores the original test's semantics, where the new rollup ran on its own genesis.
…g tests

The conversion onto setupBlockProducer dropped the funded-account count
(setup(1, ...)) for the 'can simulate public txs while building a block',
'clears up all nullifiers if tx processing fails', and reorgs tests, while
they still destructure accounts: [ownerAddress] from the context. With
numberOfAccounts defaulting to 0, ownerAddress was undefined, producing
'Undefined argument admin/owner' and 'Account not found in wallet'. Pass
numberOfAccounts: 1 to restore the single funded account these tests rely on.
FullProverTest builds its own real-BB environment rather than routing
through the factory, so setupWithRealProver had no callers.
Move e2e_fees/ into single-node/fees/ (joining the fee-asset price-oracle
test) and reconcile FeesTest onto SingleNodeTestContext. FeesTest now extends
the base and builds its prover-backed environment with its own setup() opts,
then calls hydrateFromContext to reuse the base rollup / epoch-cache /
chain-monitor / node-tracking / teardown machinery, keeping only the fee/gas
domain setup (FPC, fee juice, banana token) on the harness. The block-based
catchUpProvenChain is preserved verbatim. Adds the .parallel suffix to the
multi-it files and repoints the .test_patterns.yml fees entries.
Move e2e_cross_chain_messaging/ into the new single-node/cross-chain/ folder
and reconcile CrossChainMessagingTest onto SingleNodeTestContext. The harness
now extends the base and builds its environment with its own setup() opts
(optional prover node, EpochTestSettler auto-proving otherwise), then calls
hydrateFromContext to reuse the base rollup / epoch-cache / chain-monitor /
node-tracking / teardown machinery, keeping the CrossChainTestHarness domain
object and the L1 inbox/outbox handles on the harness. The shared
cross_chain_test_harness.ts stays in src/shared/ (it is also used by
bench/client_flows and shared/uniswap_l1_l2); the moved files' import paths
are repointed accordingly. Adds the single-node/cross-chain/ CI glob to both
bootstrap arrays, repoints the per-test timeout case and the
.test_patterns.yml entries, and applies the .parallel suffix to the multi-it
files.
@spalladino
spalladino force-pushed the spl/e2e-consolidation-2 branch from f5a11ea to 94e41a4 Compare June 26, 2026 20:55
The clean-lite step run on every yarn-project bootstrap build deletes
git-ignored files (build artifacts). It already spared node_modules and
.yarn; also spare the top-level tmp/ scratch directory so local scratch
files survive a build.
These three suites were converted to .parallel.test.ts in the e2e
consolidation, which makes the CI runner extract each it() and run it in its
own isolated container via run_test.sh "<test name>". That breaks them:

- token_bridge_private: the second test relies on the L1 tokens minted by the
  first test, so in isolation the portal deposit reverts with
  ERC20InsufficientBalance.
- failures (fees): "includes transaction that error in teardown" snapshots
  maxFeesPerGas at setup, but in isolation the chain reaches a higher
  feePerL2Gas by the time the tx executes, so the FPC setup phase reverts and
  the tx is dropped instead of mined-reverted.
- token_bridge_failure_cases: two of its test names start with "Can't", and
  the runner's extract_test_names regex truncates at the apostrophe to "Can",
  producing an ambiguous -t filter that never runs the intended test.

These suites share state via beforeAll and were only ever validated as
sequential whole files (their bodies are byte-identical to the base branch).
Rename them back to .test.ts so the runner executes the whole file in order.
Suites with per-test beforeEach setup (token_bridge_public, etc.) remain
parallel-safe and are left as .parallel.test.ts.
@PhilWindle
PhilWindle merged commit f95692b into merge-train/spartan-v5 Jun 29, 2026
12 checks passed
@PhilWindle
PhilWindle deleted the spl/e2e-consolidation-2 branch June 29, 2026 07:20
spalladino added a commit that referenced this pull request Jul 1, 2026
## Motivation

A-1178 (#24281) measures each e2e test's wall-clock as four coarse
buckets (`setupFnMs`, `beforeHooksMs`, `bodyMs`, `afterHooksMs`). That
tells us a test is slow but not *where* it loses time. Most e2e tests
spend their wall-clock in the same handful of operations — standing up
the environment, waiting for a tx to be mined/checkpointed/proven,
waiting for a committee to form or an offense to be detected,
client-side proving of seed txs, and warp scans hunting for a proposer
slot.

This decomposes that time into **named spans** so a single aggregate
query over a full CI run answers *"across the suite, how much total
wall-clock goes into proving / spinning up nodes / waiting for
checkpoints?"* — a ranked list of where to invest in speedups. It
extends A-1178 and builds on the consolidation (#24201/#24310) and
helpers (#24404) work that turned each repeated operation into one
shared definition, so wrapping it once instruments every test with no
per-test edits.

This is **instrumentation / data-gathering only** — no test behavior or
timing changes.

## Approach

- A generic `testSpan(name, fn)` / `testSpanSync(name, fn)` wrapper
(`fixtures/timing.ts`) records `{ owner, name, start, end }` into the
shared collector installed by the timing environment. When
`TEST_TIMING_FILE` is unset there is no collector, so `testSpan()` calls
`fn()` directly — exactly zero-cost, with no clock reads.
- At flush, the timing environment groups spans by owner then tag and
computes four numbers per `(owner, tag)`:
- `count` — multiplicity is itself a signal (waited for a checkpoint 14×
points at a loop to batch).
  - `totalMs` — naive sum (correct for serial repeats).
- `busyMs` — duration of the **union** of the spans' intervals;
concurrency-correct, so a `Promise.all` of 12 concurrent 3s spans reads
~3s instead of ~36s. The `busyMs ≪ totalMs` signature flags work that is
run serially where it could be parallel.
- `maxMs` — longest single occurrence, to catch one pathological wait
hiding in a cheap average.
- The aggregates attach as an additive `spans` map on each `type:"test"`
/ `type:"suite"` JSONL line. `setupFnMs` / `teardownFnMs` are kept for
back-compat, now derived from the `setup:env:<mode>` / `teardown:env`
tags.
- Tags follow a stable `category:label` taxonomy (`setup:`, `wait:`,
`tx:`, `warp:`, `wallet:`, `deploy:`, `other:`), tagged **by concept,
not by function** — e.g. every checkpoint waiter maps to
`wait:checkpoint` — so a label is a forever aggregation key regardless
of which helper a test happened to call.
- **One clock.** All spans use `performance.now()`. I verified Node's
`perf_hooks` clock is process-wide: a `performance.now()` taken inside a
fresh `vm` context (the jest sandbox realm) shares the same monotonic
origin as the host realm (delta ~0.002ms; a fresh
`perf_hooks.performance` inside a separate context anchors to the same
`timeOrigin`). Interval-merging for `busyMs` across the sandbox and host
realms is therefore valid, so **`busyMs` is included** (no fallback to
`totalMs`-only needed).
- **Background loop.** `startMempoolFeeder` runs interleaved with
arbitrary tests; without care its prove/send spans would smear onto
whichever test was current when each round fired. Its production is run
inside an `AsyncLocalStorage`-scoped owner override
(`other:mempool-feeder`), which pins every span in its async call tree
to a fixed owner — isolated from concurrent test-body spans. That owner
matches no test/suite record, so the feeder's spans are cleanly excluded
from the per-test view.

## Changes by phase

- **Phase 0 — collector + `testSpan()` + flush aggregation.** New
`fixtures/timing.ts` (`testSpan`/`testSpanSync`/`withTestSpanOwner`).
`timing_env.mjs` collector generalized from `fnSpans` to a `spans`
array; `finalizeAndFlush` groups by owner/tag, merges intervals for
`busyMs`, attaches the `spans` map, and derives the back-compat fields.
- **Phase 1 — shared waits + `setup:node`.** Wrapped the waiters in
`fixtures/wait_helpers.ts` (`wait:proposed` / `wait:proven`,
`wait:checkpoint` / `wait:proven-checkpoint`, `wait:tx-mined`,
`wait:l2-to-l1-witness`, `wait:pending-tx`, `wait:sequencer-state`) and
the wait methods on `SingleNodeTestContext` / `MultiNodeTestContext`
(`wait:epoch`, `wait:slot`, `wait:proof-window`, `wait:node-sync`,
`wait:proof-submitted`, `wait:offense`, the multi-node
`wait:checkpoint`/`wait:proven-checkpoint`/`wait:block` convergence
waiters). Wrapped `createNode`/`createProverNode` with `setup:node`.
- **Phase 2 — decompose `setup:env`.** Cracked the opaque setup in
`fixtures/setup.ts` into `setup:env:anvil`, `:l1-deploy`,
`:sequencer-start`, `:prover-node`, `:pxe`, and `wallet:create`. The
top-level `setup:env` is tagged with the prover mode (`setup:env:none` /
`:fake` / `:real`) so the three factories are comparable.
- **Phase 3 — tx leaf wraps.** `proveInteraction` → `tx:prove`,
`ProvenTx.send` → `tx:send`; everything else
(`proveTxs`/`proveAndSendTxs`/the submit helpers) aggregates through
those leaves. `startMempoolFeeder` handled as above.
- **Phase 4 — slashing/governance waiters.** Wrapped
`awaitCommitteeExists` (`wait:committee`), `awaitOffenseDetected`
(`wait:offense`), `awaitCommitteeKicked` (`wait:committee-kicked`),
`awaitProposalExecution` (`wait:slash-execution`), and the
`findUpcomingProposerSlot` / `advanceToEpochBeforeProposer` /
`findSlotsWithProposers` scans (`warp:find-proposer`).
- **Phase 5 — reporting.** The JSONL schema now carries the per-line
`spans` map, and `TEST_TIMING_SPANS=1` emits one `type:"span"` line per
occurrence (owner, name, ms) for deep dives. The repo now also ships a
`track-e2e-times` skill (`yarn-project/.claude/skills/track-e2e-times/`)
covering how to **collect and aggregate** the span timings locally: run
the suite with `TEST_TIMING_FILE` set, find the per-worker JSONL, and
print per-test sums plus the ranked span leaderboard (via the bundled
`row.sh`). Only the step of publishing aggregate numbers to a running
tracking log remains a separate out-of-band workflow. The leaderboard
rollup is a small additive jq over the new `spans` maps:

  ```
jq -rs '[ .[] | select(.type=="test" or .type=="suite") | (.spans // {})
| to_entries[] ]
    | group_by(.key)
    | map({ tag: .[0].key, count: (map(.value.count) | add),
busyMs: (map(.value.busyMs) | add), maxMs: (map(.value.maxMs) | max) })
| sort_by(-.busyMs) | .[] | "\(.busyMs)\t\(.count)\t\(.maxMs)\t\(.tag)"'
  ```

The existing `row.sh` is unaffected — it filters `type=="test"` and
reads the unchanged `setupFnMs`/`bodyMs`/etc., ignoring the new `spans`
key and `type:"span"` lines.

## Notes

- `waitForEpoch`/`waitForSlot` (added to `@aztec/ethereum`'s
`rollup_cheat_codes.ts` by #24404) are deliberately **not** instrumented
in-package: that package should not depend on the e2e timing collector,
and the `wait:epoch`/`wait:slot` concepts they cover are already
captured at the e2e-context wrapper layer (`waitUntilEpochStarts`, the
slot waiters). They have no e2e callers yet; a future call site picks
them up via its context wrapper.
- Spans do not partition `bodyMs` — a parent span includes its children,
so `sum(spans) ≠ bodyMs`. Tagging is at the leaf wait/setup/tx level
where additivity holds; the few intentional nests (e.g.
`wait:committee-kicked` contains `wait:slash-execution`) carry distinct
tags.

Verified: `yarn build`, `yarn lint end-to-end`, `yarn format --check
end-to-end` all pass. The flush aggregation (busyMs interval merge,
back-compat derivation, feeder-owner exclusion, opt-in raw lines) is
covered by unit tests in `end-to-end/src/shared/timing_env.test.ts`,
which exercise the extracted `aggregateSpans` / `foldSpansInto` pure
functions directly.

Fixes A-1179



## Example timing output

Each e2e test run in CI emits one JSONL record per test and per suite,
carrying a `spans` map keyed by `category:label` tag, where each value
is `{count, totalMs, busyMs, maxMs}` (`busyMs` is the de-overlapped
union duration, so concurrent occurrences of a span are counted once).
An illustrative `type:"test"` line from CI run 28469687883 (commit
de3635e):

```json
{
  "suite": "optimistic.parallel",
  "type": "test",
  "name": "single-node/proving/optimistic happy path proves multiple epochs via checkpoint-driven flow",
  "status": "passed",
  "setupFnMs": 4707,
  "beforeHooksMs": 4717,
  "bodyMs": 327324,
  "teardownFnMs": 615,
  "afterHooksMs": 624,
  "totalMs": 332668,
  "startedAt": "2026-06-30T19:20:58.279Z",
  "spans": {
    "setup:env:anvil": {
      "count": 1,
      "totalMs": 82,
      "busyMs": 82,
      "maxMs": 82
    },
    "setup:env:l1-deploy": {
      "count": 1,
      "totalMs": 470,
      "busyMs": 470,
      "maxMs": 470
    },
    "setup:env:sequencer-start": {
      "count": 1,
      "totalMs": 3141,
      "busyMs": 3141,
      "maxMs": 3141
    },
    "setup:env:prover-node": {
      "count": 1,
      "totalMs": 139,
      "busyMs": 139,
      "maxMs": 139
    },
    "setup:env:pxe": {
      "count": 1,
      "totalMs": 391,
      "busyMs": 391,
      "maxMs": 391
    },
    "wallet:create": {
      "count": 1,
      "totalMs": 79,
      "busyMs": 79,
      "maxMs": 79
    },
    "setup:env:fake": {
      "count": 1,
      "totalMs": 4707,
      "busyMs": 4707,
      "maxMs": 4707
    },
    "wait:epoch": {
      "count": 5,
      "totalMs": 180210,
      "busyMs": 180210,
      "maxMs": 36067
    },
    "tx:prove": {
      "count": 4,
      "totalMs": 2123,
      "busyMs": 2123,
      "maxMs": 621
    },
    "tx:send": {
      "count": 4,
      "totalMs": 144305,
      "busyMs": 144305,
      "maxMs": 36124
    },
    "wait:proven-checkpoint": {
      "count": 4,
      "totalMs": 0,
      "busyMs": 0,
      "maxMs": 0
    },
    "wait:node-sync": {
      "count": 4,
      "totalMs": 401,
      "busyMs": 401,
      "maxMs": 101
    },
    "teardown:env": {
      "count": 1,
      "totalMs": 615,
      "busyMs": 615,
      "maxMs": 615
    }
  },
  "commit": "de3635e74eb89071b52939d1583d072ba2c3852c",
  "branch": "spl/a1179-track-common-spans",
  "runId": "28469687883"
}
```
vezenovm added a commit that referenced this pull request Jul 1, 2026
…stant

The reverted comment (copied verbatim from before #24429) still said the
value must be larger than MAX_PRIVATE_LOGS_PER_TX while setting it to 20,
and cited e2e_l1_with_wall_time, which no longer exists (deleted in #24310).
Replace with an accurate description of the real bound and mark the value
explicitly as an unsafe test-only setting.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants