test(e2e): instrument and diagnose bot suite setup cost by spalladino · Pull Request #24534 · AztecProtocol/aztec-packages

spalladino · 2026-07-06T13:46:34Z

What / why

Round-4 e2e speedup, PR 3 of the effort. The single-node/bot suite's file-level beforeAll costs
~178s on CI with 97% untagged, and the residual proven-chain waits in the private_payments/failures
fees suites (~32s/shard) were also invisible. This PR instruments both so the cost is attributable, and
diagnoses the bot residual.

Spans are pure passthrough when TEST_TIMING_FILE is unset (see fixtures/timing.ts), so this is zero
behavior change for normal runs and CI.

Commit

test(e2e): instrument bot suite setup and fees proven-chain residuals
- Bot suite (single-node/bot/bot.test.ts): setup:wallet around EmbeddedWallet.create,
  wallet:create around createSchnorrInitializerlessAccount, and setup:bot around the four nested
  describe beforeAll hooks (Bot.create / AmmBot.create / CrossChainBot.create / BotStore).
- Fees harness (single-node/fees): wait:proven-checkpoint around catchUpProvenChain's poll loop,
  and warp:proven-checkpoint-epoch around advanceToNextEpoch via a new instrumented
  FeesTest.advanceToNextEpoch() wrapper; waitForEpochProven and the direct call sites in
  failures/private_payments now route through it.
- Reuses existing tags where the concept matches (wallet:create is already what
  createFundedInitializerlessAccounts fires; warp:proven-checkpoint-epoch is already what
  advanceToNextEpoch fires in block-production/setup.ts) rather than minting near-duplicates.

Diagnosis

The timing environment (shared/timing_env.mjs) folds every beforeAll in a file — the file-level
one plus each nested describe's — into a single suite-scoped beforeHooksMs, and every span fired
during any beforeAll lands on that same suite line. The shortlist read the whole number as the
file-level hook and, seeing only setup:env:* tagged, attributed the rest to the one visible untagged
file-level call (createSchnorrInitializerlessAccount). It is actually the nested hooks.

Local read (TEST_TIMING_SPANS=1, PIPELINING_SETUP_OPTS = 12s L2 slots, production sequencer, fake
prover), suite beforeHooksMs = 153,733 ms:

setup:env:* (file-level setup()): ~4.6s — already tagged
setup:wallet (EmbeddedWallet.create, ephemeral): 137 ms — the PXE has autoSync:false, no
genesis walk
wallet:create (createSchnorrInitializerlessAccount): 69 ms — initializerless, no deploy tx, as
designed
setup:bot (4 nested hooks): 149,582 ms = 97.3% of the file's beforeAll
- transaction-bot Bot.create: 32.2s (token deploy class+instance, then private+public mint)
- bridge-resume new BotStore(...): 0.9 ms
- amm-bot AmmBot.create: 73.9s (deploys 4 contracts + set_minter + mint + add_liquidity)
- cross-chain-bot CrossChainBot.create: 43.5s (TestContract deploy + seed 2 L1→L2 msgs + wait ready)

Root cause: structural, not a bug. The bot suite runs the production Sequencer to exercise
CHECKPOINTED/PROPOSED follow modes and L1 fee-juice bridging, so each Bot.create deploys real
contracts and mints serially at slot cadence (bot/src/factory.ts). The two calls the shortlist
suspected are ~0.1s each.

No fix is shipped here because there is no safe one-liner: speeding this up means batching/parallelizing
the bot/src/factory.ts deploy/mint paths (production code, PR-6-shaped), genesis-seeding (PR 1/2), or
cutting the CI slot time (PR 5) — none belong in the instrumentation PR. Tracked as a round-5 item; the
new setup:bot span makes that win measurable. Full write-up in tmp/SPEEDUP_ROUND4_FINDINGS.md.

Expected effect

No wall-clock change (passthrough spans). The deliverable is visibility: the previously-invisible
setup:bot / setup:wallet / wallet:create decomposition of the bot beforeAll, and the
wait:proven-checkpoint / warp:proven-checkpoint-epoch residual in the fees suites.

Measured numbers from a full green CI run will be appended below once available.

Measured impact

Both runs are full CI runs. PR run: ci/x-fast on head 1f7898cd2b (CI 1783345614459429, 1,799 test
lines). Baseline: ci/x-full-no-test-cache on the exact merge-base a8cfa93df5 (CI 1783341726388943,
1,863 test lines). No wall-clock change claimed (spans are passthrough; suite-level noise floor is
~11s median / ~52s p90) — the deliverable is attribution, measured as span busyMs.

Bot suite (suite line, beforeAll = 181.5s on the PR run vs 182.1s baseline):

Baseline: 7.4s tagged (setup:env:* only) → 174.8s / 96% invisible.
PR run: setup:bot 174,203 ms (4 occurrences, max 86,961 ms = the amm-bot hook),
setup:wallet 250 ms, wallet:create 144 ms; residual now −1.5s ≈ 0 →
100% of the beforeAll is attributed.
This confirms the diagnosis: EmbeddedWallet.create + createSchnorrInitializerlessAccount are
~0.4s combined; the "invisible 178s" is the four nested describes' bot-factory setup
(deploys + mints at production 12s-slot cadence), folded into the file's suite line by the
hook accounting.

Fees suites (private_payments.parallel × 8 shards + failures):

Baseline: avg 32.3s/shard untagged residual (zero wait:proven-checkpoint /
warp:proven-checkpoint-epoch lines on these suites).
PR run: wait:proven-checkpoint fires on all 9 suite hooks, 293,888 ms total
(avg 32.7s/shard — matches the residual exactly), plus one in-test occurrence (32.1s);
warp:proven-checkpoint-epoch 19 occurrences, 361 ms total (the warp RPC itself is
near-free — the cost is entirely the proven-chain poll that follows). Per-shard residual is now
≈ −1s, i.e. fully attributed.

Net: ~469s/run of previously invisible setup/wait cost is now tagged and attributable
(174.2s setup:bot + 0.4s wallet spans + 294.3s fees proven-chain waits), with no behavior or
wall-clock change.

Fixes A-1407

Adds timing spans so the untagged residual of the bot suite's file-level beforeAll (~178s on CI, 97% invisible) can be decomposed and the ~32s/shard proven-chain residual in the private_payments/failures fees suites becomes visible. Bot suite (single-node/bot/bot.test.ts): - setup:wallet around EmbeddedWallet.create (stands up the ephemeral PXE) - wallet:create around createSchnorrInitializerlessAccount (same concept as createFundedInitializerlessAccounts, already tagged wallet:create) - setup:bot around the nested describes' Bot.create/AmmBot.create/ CrossChainBot.create/BotStore creation, which the suite-line accounting folds into the same suite beforeHooksMs as the file hook Fees harness (single-node/fees): - wait:proven-checkpoint around catchUpProvenChain's poll loop - warp:proven-checkpoint-epoch around advanceToNextEpoch, via a new instrumented FeesTest.advanceToNextEpoch() wrapper that waitForEpochProven and the direct call sites in failures/private_payments now route through Reuses existing tags where the concept matches rather than inventing near-duplicates. Spans are passthrough when TEST_TIMING_FILE is unset, so this is zero behavior change.

Parallelizes the independent setup steps in the bot factory (`@aztec/bot`, `bot/src/factory.ts`). This is production bot code that runs against live networks, so the changes are strictly behavior-preserving: same contracts at the same (salt-derived) addresses, same amounts, same final state. Only the ordering/concurrency of provably-independent setup steps changes. The bot factory is the single largest cost in the `single-node/bot` e2e suite: each `*.create` runs the production sequencer at 12s L2 slots (`PIPELINING_SETUP_OPTS`) and deploys/mints one tx per slot, fully serial. `AmmBot.create` alone was ~74s (7 serial txs). ## Dependency graph per bot type Edges below are "must be mined before", verified against the Noir contract source (`token_contract`, `amm_contract`). ### Transaction bot -- `setup()` (unchanged) `setupAccount` -> register recipient -> `ensureFeeJuiceBalance` -> deploy token -> mint. This chain is fully data-dependent: funding gates the deploy, the deploy gates the mint. The recipient (`createSchnorrAccount`) is register-only (no tx), and the private+public mints are already batched into one tx. Nothing is independent, so `setup()` is left serial. ### AMM bot -- `setupAmm()` Steps: deploy token0 (D0), token1 (D1), LP token (DL), AMM (DA); `set_minter` granting the AMM rights over the LP token (SM); mint token0+token1 to the provider (M, one batched tx); `add_liquidity` (AL). - D0, D1, DL: independent -- distinct contracts, no cross-references. - DA: the AMM constructor only stores the three token addresses (no calls into the tokens). Those addresses are salt-derived, so DA needs only the (pre-derived) addresses, not the token deploys mined. - SM: `Token::set_minter` just writes `minters[amm] = true`; it does not call or validate the AMM contract. It needs the LP token deployed and the (derivable) AMM address only -- not the AMM deployed. The deployer is authorized because the Token constructor sets `minters[admin] = true`. - M: `mint_to_private`/`mint_to_public` require the caller be a minter; the deployer is admin=minter from the constructor, so the token0/token1 mints need only those tokens deployed (no `set_minter`). - AL: `add_liquidity` transfers token0/token1 from the provider (needs M for balances), mints LP tokens (needs SM so the AMM is an LP minter), and targets the AMM (needs DA). Restructured from 7 serial txs into 3 slot-phases: - Phase 1: `Promise.all([D0, D1, DL])` - Phase 2: `Promise.all([DA, SM, M])` - Phase 3: `AL` ### Cross-chain bot -- `setupCrossChain()` Steps: deploy TestContract (an L2 tx), seed N L1->L2 messages (L1 txs), wait for the first message ready. - The seeds reference only the L2 recipient (TestContract) address, which is salt-derived. L1->L2 messages are queued on L1 and do not require the L2 contract to exist yet; they are consumed later in `bot.run()`, after setup completes. So the L2 deploy and the L1 seeding are independent. - The L2 deploy pays via the L2 wallet/PXE account; the seeds use the bot's L1 client -- different accounts, so there is no L1 nonce interaction between them. Restructured to overlap the deploy with the seed loop: `Promise.all([deploy TestContract, seed loop])`, then the message-ready wait. ## What stayed serial, and why - The whole transaction-bot `setup()` (data-dependent chain, nothing independent). - `ensureFeeJuiceBalance` before every deploy (funding must precede spending). - `add_liquidity` after its mints + minter grant + AMM deploy. - The individual L1->L2 seeds inside the loop: they share the bot's single L1 account, so concurrent sends would race on the L1 nonce. The suite already documents this nonce hazard. Only the loop as a whole overlaps the (L2) TestContract deploy. ## Production safety / idempotent re-entry - Every deploy still goes through `registerOrDeployContract`, which checks `getContractMetadata(address).isContractPublished` per contract and only registers (no tx) if already deployed. If one parallel deploy fails and the others succeed, the next `*.create` recovers: the succeeded contracts register-only, the failed one redeploys. Addresses are salt-derived and identical across runs. - `set_minter` is idempotent (writes `true` again). The AMM path's mint and `add_liquidity` always run (no balance guard) -- unchanged from the previous `fundAmm`. - Same-sender concurrent `.send()`s are safe by construction: the PXE serializes simulate/prove through its `SerialQueue` and setup txs pay with random tx nonces. The bot runs the same `EmbeddedWallet` -> PXE path in the e2e suite and in production. - Deliberate, benign failure-path change: because Phase 2 runs DA/SM/M concurrently, a failure of one no longer prevents the others from being sent. Their effects (a deployed-but-unused AMM, a minter grant, an extra mint) are all idempotent-safe, and `add_liquidity` uses fixed amounts, so re-entry converges to the same final state. - `withNoMinTxsPerBlock` (the `flushSetupTransactions` save/zero/restore wrapper around each setup tx) was not safe to re-enter concurrently: interleaved save/zero/restore could read the already-zeroed value and "restore" `minTxsPerBlock` to 0 permanently. It now reference-counts entrants -- the first saves and zeroes, the last restores -- with unit tests covering the overlapping and failure paths (`bot/src/factory.test.ts`). Serial callers see the exact same RPC sequence as before. Final state is identical in every path: same contract addresses, same minter grant, same minted amounts, same liquidity. ## Local evidence Two full local runs of `single-node/bot/bot.test.ts` (production sequencer, 12s L2 slots), all 10 tests passing in both. Hook durations from factory log timestamps, against the round-4 local baseline measured on the same machine at the same cadence (findings from #24534): - `Bot.create` (transaction-bot hook, untouched): 31.8s vs 32.2s baseline -- unchanged, as intended. - `AmmBot.create`: 57.2s vs 73.9s baseline (-16.7s). The three token deploy txs go out within 300ms of each other; the AMM deploy, `set_minter` and the mint batch all go out within the following slot. - `CrossChainBot.create`: 24.0s vs 43.5s baseline (-19.5s). The TestContract deploy overlaps both L1 seeds; the first message is ready almost immediately after the deploy is mined. - Suite-level `beforeHooksMs`: 116.9s vs 153.7s baseline (-36.8s), matching the per-hook deltas. - Unit: `bot/src/factory.test.ts` (4 tests) covering the `withNoMinTxsPerBlock` reentrancy fix. ## Expected effect on CI - `AmmBot.create`: 7 serial txs -> 3 slot-phases (~87s on CI per #24534's spans -> ~55-60s). - `CrossChainBot.create`: deploy overlaps the seed pipeline (~43s -> ~25s). - Transaction bot: unchanged. - Aggregate: roughly -35 to -45s on the bot suite's beforeAll wall time. - Fix-phase proof metric (once #24534's `setup:bot` instrumentation is on this base): the amm-bot and cross-chain `setup:bot` occurrences drop per the numbers above. Measured CI numbers to be appended when a full run lands. ## Measured impact The bot suite (a single shard) drops 216.3s → 118.3s total (**−98.0s, −45%**); beforeHooks 215.9s → 118.0s. - No `setup:bot` spans exist on this base (that instrumentation lives in the unmerged #24534), so the metric is the suite-level hook fold, which covers all three nested bot factory setups (Bot, AmmBot, CrossChainBot). - Noise context from the same run pair: untouched light suites move −2 to −3s per shard (private_initialization −2.9s/shard over 20 shards, deploy_method −2.1s/shard over 14), and the largest counter-move is validators_sentinel +24s (multi-node variance). A −98s single-shard delta is far outside that envelope. - The CI delta exceeds the local −36.8s because the CI baseline pays more missed-slot penalties per serial tx (baseline beforeHooks 215.9s on CI vs 153.7s locally at the same 12s cadence); collapsing 7 serial txs into 3 slot-phases removes proportionally more where slots are more often missed. Baseline CI run 1783393028773172 (merge-base 30966a4, full). PR CI run 1783427250198215 (x-fast). Fixes A-1408

## What / why Round-4 e2e speedup, PR 3 of the effort. The `single-node/bot` suite's file-level `beforeAll` costs ~178s on CI with 97% untagged, and the residual proven-chain waits in the `private_payments`/`failures` fees suites (~32s/shard) were also invisible. This PR instruments both so the cost is attributable, and diagnoses the bot residual. Spans are pure passthrough when `TEST_TIMING_FILE` is unset (see `fixtures/timing.ts`), so this is zero behavior change for normal runs and CI. ## Commit - `test(e2e): instrument bot suite setup and fees proven-chain residuals` - Bot suite (`single-node/bot/bot.test.ts`): `setup:wallet` around `EmbeddedWallet.create`, `wallet:create` around `createSchnorrInitializerlessAccount`, and `setup:bot` around the four nested `describe` beforeAll hooks (`Bot.create` / `AmmBot.create` / `CrossChainBot.create` / `BotStore`). - Fees harness (`single-node/fees`): `wait:proven-checkpoint` around `catchUpProvenChain`'s poll loop, and `warp:proven-checkpoint-epoch` around `advanceToNextEpoch` via a new instrumented `FeesTest.advanceToNextEpoch()` wrapper; `waitForEpochProven` and the direct call sites in `failures`/`private_payments` now route through it. - Reuses existing tags where the concept matches (`wallet:create` is already what `createFundedInitializerlessAccounts` fires; `warp:proven-checkpoint-epoch` is already what `advanceToNextEpoch` fires in `block-production/setup.ts`) rather than minting near-duplicates. ## Diagnosis The timing environment (`shared/timing_env.mjs`) folds *every* `beforeAll` in a file — the file-level one plus each nested `describe`'s — into a single suite-scoped `beforeHooksMs`, and every span fired during any beforeAll lands on that same suite line. The shortlist read the whole number as the file-level hook and, seeing only `setup:env:*` tagged, attributed the rest to the one visible untagged file-level call (`createSchnorrInitializerlessAccount`). It is actually the nested hooks. Local read (`TEST_TIMING_SPANS=1`, `PIPELINING_SETUP_OPTS` = 12s L2 slots, production sequencer, fake prover), suite `beforeHooksMs = 153,733 ms`: - `setup:env:*` (file-level `setup()`): ~4.6s — already tagged - `setup:wallet` (`EmbeddedWallet.create`, ephemeral): **137 ms** — the PXE has `autoSync:false`, no genesis walk - `wallet:create` (`createSchnorrInitializerlessAccount`): **69 ms** — initializerless, no deploy tx, as designed - `setup:bot` (4 nested hooks): **149,582 ms = 97.3%** of the file's beforeAll - transaction-bot `Bot.create`: 32.2s (token deploy class+instance, then private+public mint) - bridge-resume `new BotStore(...)`: 0.9 ms - amm-bot `AmmBot.create`: 73.9s (deploys 4 contracts + set_minter + mint + add_liquidity) - cross-chain-bot `CrossChainBot.create`: 43.5s (TestContract deploy + seed 2 L1→L2 msgs + wait ready) **Root cause: structural, not a bug.** The bot suite runs the production Sequencer to exercise CHECKPOINTED/PROPOSED follow modes and L1 fee-juice bridging, so each `Bot.create` deploys real contracts and mints serially at slot cadence (`bot/src/factory.ts`). The two calls the shortlist suspected are ~0.1s each. No fix is shipped here because there is no safe one-liner: speeding this up means batching/parallelizing the `bot/src/factory.ts` deploy/mint paths (production code, PR-6-shaped), genesis-seeding (PR 1/2), or cutting the CI slot time (PR 5) — none belong in the instrumentation PR. Tracked as a round-5 item; the new `setup:bot` span makes that win measurable. Full write-up in `tmp/SPEEDUP_ROUND4_FINDINGS.md`. ## Expected effect No wall-clock change (passthrough spans). The deliverable is visibility: the previously-invisible `setup:bot` / `setup:wallet` / `wallet:create` decomposition of the bot beforeAll, and the `wait:proven-checkpoint` / `warp:proven-checkpoint-epoch` residual in the fees suites. Measured numbers from a full green CI run will be appended below once available. ## Measured impact Both runs are full CI runs. PR run: `ci/x-fast` on head `1f7898cd2b` (CI 1783345614459429, 1,799 test lines). Baseline: `ci/x-full-no-test-cache` on the exact merge-base `a8cfa93df5` (CI 1783341726388943, 1,863 test lines). No wall-clock change claimed (spans are passthrough; suite-level noise floor is ~11s median / ~52s p90) — the deliverable is attribution, measured as span `busyMs`. Bot suite (`suite` line, beforeAll = 181.5s on the PR run vs 182.1s baseline): - Baseline: 7.4s tagged (`setup:env:*` only) → **174.8s / 96% invisible**. - PR run: `setup:bot` **174,203 ms** (4 occurrences, max 86,961 ms = the amm-bot hook), `setup:wallet` **250 ms**, `wallet:create` **144 ms**; residual now **−1.5s ≈ 0** → **100% of the beforeAll is attributed**. - This confirms the diagnosis: `EmbeddedWallet.create` + `createSchnorrInitializerlessAccount` are ~0.4s combined; the "invisible 178s" is the four nested describes' bot-factory setup (deploys + mints at production 12s-slot cadence), folded into the file's suite line by the hook accounting. Fees suites (`private_payments.parallel` × 8 shards + `failures`): - Baseline: avg **32.3s/shard untagged residual** (zero `wait:proven-checkpoint` / `warp:proven-checkpoint-epoch` lines on these suites). - PR run: `wait:proven-checkpoint` fires on all 9 suite hooks, **293,888 ms total** (avg 32.7s/shard — matches the residual exactly), plus one in-test occurrence (32.1s); `warp:proven-checkpoint-epoch` 19 occurrences, 361 ms total (the warp RPC itself is near-free — the cost is entirely the proven-chain poll that follows). Per-shard residual is now ≈ −1s, i.e. fully attributed. Net: **~469s/run of previously invisible setup/wait cost is now tagged and attributable** (174.2s `setup:bot` + 0.4s wallet spans + 294.3s fees proven-chain waits), with no behavior or wall-clock change. Fixes A-1407 (cherry picked from commit 1476179)

Parallelizes the independent setup steps in the bot factory (`@aztec/bot`, `bot/src/factory.ts`). This is production bot code that runs against live networks, so the changes are strictly behavior-preserving: same contracts at the same (salt-derived) addresses, same amounts, same final state. Only the ordering/concurrency of provably-independent setup steps changes. The bot factory is the single largest cost in the `single-node/bot` e2e suite: each `*.create` runs the production sequencer at 12s L2 slots (`PIPELINING_SETUP_OPTS`) and deploys/mints one tx per slot, fully serial. `AmmBot.create` alone was ~74s (7 serial txs). ## Dependency graph per bot type Edges below are "must be mined before", verified against the Noir contract source (`token_contract`, `amm_contract`). ### Transaction bot -- `setup()` (unchanged) `setupAccount` -> register recipient -> `ensureFeeJuiceBalance` -> deploy token -> mint. This chain is fully data-dependent: funding gates the deploy, the deploy gates the mint. The recipient (`createSchnorrAccount`) is register-only (no tx), and the private+public mints are already batched into one tx. Nothing is independent, so `setup()` is left serial. ### AMM bot -- `setupAmm()` Steps: deploy token0 (D0), token1 (D1), LP token (DL), AMM (DA); `set_minter` granting the AMM rights over the LP token (SM); mint token0+token1 to the provider (M, one batched tx); `add_liquidity` (AL). - D0, D1, DL: independent -- distinct contracts, no cross-references. - DA: the AMM constructor only stores the three token addresses (no calls into the tokens). Those addresses are salt-derived, so DA needs only the (pre-derived) addresses, not the token deploys mined. - SM: `Token::set_minter` just writes `minters[amm] = true`; it does not call or validate the AMM contract. It needs the LP token deployed and the (derivable) AMM address only -- not the AMM deployed. The deployer is authorized because the Token constructor sets `minters[admin] = true`. - M: `mint_to_private`/`mint_to_public` require the caller be a minter; the deployer is admin=minter from the constructor, so the token0/token1 mints need only those tokens deployed (no `set_minter`). - AL: `add_liquidity` transfers token0/token1 from the provider (needs M for balances), mints LP tokens (needs SM so the AMM is an LP minter), and targets the AMM (needs DA). Restructured from 7 serial txs into 3 slot-phases: - Phase 1: `Promise.all([D0, D1, DL])` - Phase 2: `Promise.all([DA, SM, M])` - Phase 3: `AL` ### Cross-chain bot -- `setupCrossChain()` Steps: deploy TestContract (an L2 tx), seed N L1->L2 messages (L1 txs), wait for the first message ready. - The seeds reference only the L2 recipient (TestContract) address, which is salt-derived. L1->L2 messages are queued on L1 and do not require the L2 contract to exist yet; they are consumed later in `bot.run()`, after setup completes. So the L2 deploy and the L1 seeding are independent. - The L2 deploy pays via the L2 wallet/PXE account; the seeds use the bot's L1 client -- different accounts, so there is no L1 nonce interaction between them. Restructured to overlap the deploy with the seed loop: `Promise.all([deploy TestContract, seed loop])`, then the message-ready wait. ## What stayed serial, and why - The whole transaction-bot `setup()` (data-dependent chain, nothing independent). - `ensureFeeJuiceBalance` before every deploy (funding must precede spending). - `add_liquidity` after its mints + minter grant + AMM deploy. - The individual L1->L2 seeds inside the loop: they share the bot's single L1 account, so concurrent sends would race on the L1 nonce. The suite already documents this nonce hazard. Only the loop as a whole overlaps the (L2) TestContract deploy. ## Production safety / idempotent re-entry - Every deploy still goes through `registerOrDeployContract`, which checks `getContractMetadata(address).isContractPublished` per contract and only registers (no tx) if already deployed. If one parallel deploy fails and the others succeed, the next `*.create` recovers: the succeeded contracts register-only, the failed one redeploys. Addresses are salt-derived and identical across runs. - `set_minter` is idempotent (writes `true` again). The AMM path's mint and `add_liquidity` always run (no balance guard) -- unchanged from the previous `fundAmm`. - Same-sender concurrent `.send()`s are safe by construction: the PXE serializes simulate/prove through its `SerialQueue` and setup txs pay with random tx nonces. The bot runs the same `EmbeddedWallet` -> PXE path in the e2e suite and in production. - Deliberate, benign failure-path change: because Phase 2 runs DA/SM/M concurrently, a failure of one no longer prevents the others from being sent. Their effects (a deployed-but-unused AMM, a minter grant, an extra mint) are all idempotent-safe, and `add_liquidity` uses fixed amounts, so re-entry converges to the same final state. - `withNoMinTxsPerBlock` (the `flushSetupTransactions` save/zero/restore wrapper around each setup tx) was not safe to re-enter concurrently: interleaved save/zero/restore could read the already-zeroed value and "restore" `minTxsPerBlock` to 0 permanently. It now reference-counts entrants -- the first saves and zeroes, the last restores -- with unit tests covering the overlapping and failure paths (`bot/src/factory.test.ts`). Serial callers see the exact same RPC sequence as before. Final state is identical in every path: same contract addresses, same minter grant, same minted amounts, same liquidity. ## Local evidence Two full local runs of `single-node/bot/bot.test.ts` (production sequencer, 12s L2 slots), all 10 tests passing in both. Hook durations from factory log timestamps, against the round-4 local baseline measured on the same machine at the same cadence (findings from #24534): - `Bot.create` (transaction-bot hook, untouched): 31.8s vs 32.2s baseline -- unchanged, as intended. - `AmmBot.create`: 57.2s vs 73.9s baseline (-16.7s). The three token deploy txs go out within 300ms of each other; the AMM deploy, `set_minter` and the mint batch all go out within the following slot. - `CrossChainBot.create`: 24.0s vs 43.5s baseline (-19.5s). The TestContract deploy overlaps both L1 seeds; the first message is ready almost immediately after the deploy is mined. - Suite-level `beforeHooksMs`: 116.9s vs 153.7s baseline (-36.8s), matching the per-hook deltas. - Unit: `bot/src/factory.test.ts` (4 tests) covering the `withNoMinTxsPerBlock` reentrancy fix. ## Expected effect on CI - `AmmBot.create`: 7 serial txs -> 3 slot-phases (~87s on CI per #24534's spans -> ~55-60s). - `CrossChainBot.create`: deploy overlaps the seed pipeline (~43s -> ~25s). - Transaction bot: unchanged. - Aggregate: roughly -35 to -45s on the bot suite's beforeAll wall time. - Fix-phase proof metric (once #24534's `setup:bot` instrumentation is on this base): the amm-bot and cross-chain `setup:bot` occurrences drop per the numbers above. Measured CI numbers to be appended when a full run lands. ## Measured impact The bot suite (a single shard) drops 216.3s → 118.3s total (**−98.0s, −45%**); beforeHooks 215.9s → 118.0s. - No `setup:bot` spans exist on this base (that instrumentation lives in the unmerged #24534), so the metric is the suite-level hook fold, which covers all three nested bot factory setups (Bot, AmmBot, CrossChainBot). - Noise context from the same run pair: untouched light suites move −2 to −3s per shard (private_initialization −2.9s/shard over 20 shards, deploy_method −2.1s/shard over 14), and the largest counter-move is validators_sentinel +24s (multi-node variance). A −98s single-shard delta is far outside that envelope. - The CI delta exceeds the local −36.8s because the CI baseline pays more missed-slot penalties per serial tx (baseline beforeHooks 215.9s on CI vs 153.7s locally at the same 12s cadence); collapsing 7 serial txs into 3 slot-phases removes proportionally more where slots are more often missed. Baseline CI run 1783393028773172 (merge-base 30966a4, full). PR CI run 1783427250198215 (x-fast). Fixes A-1408 (cherry picked from commit 5d2b6df)

## What / why Round-4 e2e speedup, PR 3 of the effort. The `single-node/bot` suite's file-level `beforeAll` costs ~178s on CI with 97% untagged, and the residual proven-chain waits in the `private_payments`/`failures` fees suites (~32s/shard) were also invisible. This PR instruments both so the cost is attributable, and diagnoses the bot residual. Spans are pure passthrough when `TEST_TIMING_FILE` is unset (see `fixtures/timing.ts`), so this is zero behavior change for normal runs and CI. ## Commit - `test(e2e): instrument bot suite setup and fees proven-chain residuals` - Bot suite (`single-node/bot/bot.test.ts`): `setup:wallet` around `EmbeddedWallet.create`, `wallet:create` around `createSchnorrInitializerlessAccount`, and `setup:bot` around the four nested `describe` beforeAll hooks (`Bot.create` / `AmmBot.create` / `CrossChainBot.create` / `BotStore`). - Fees harness (`single-node/fees`): `wait:proven-checkpoint` around `catchUpProvenChain`'s poll loop, and `warp:proven-checkpoint-epoch` around `advanceToNextEpoch` via a new instrumented `FeesTest.advanceToNextEpoch()` wrapper; `waitForEpochProven` and the direct call sites in `failures`/`private_payments` now route through it. - Reuses existing tags where the concept matches (`wallet:create` is already what `createFundedInitializerlessAccounts` fires; `warp:proven-checkpoint-epoch` is already what `advanceToNextEpoch` fires in `block-production/setup.ts`) rather than minting near-duplicates. ## Diagnosis The timing environment (`shared/timing_env.mjs`) folds *every* `beforeAll` in a file — the file-level one plus each nested `describe`'s — into a single suite-scoped `beforeHooksMs`, and every span fired during any beforeAll lands on that same suite line. The shortlist read the whole number as the file-level hook and, seeing only `setup:env:*` tagged, attributed the rest to the one visible untagged file-level call (`createSchnorrInitializerlessAccount`). It is actually the nested hooks. Local read (`TEST_TIMING_SPANS=1`, `PIPELINING_SETUP_OPTS` = 12s L2 slots, production sequencer, fake prover), suite `beforeHooksMs = 153,733 ms`: - `setup:env:*` (file-level `setup()`): ~4.6s — already tagged - `setup:wallet` (`EmbeddedWallet.create`, ephemeral): **137 ms** — the PXE has `autoSync:false`, no genesis walk - `wallet:create` (`createSchnorrInitializerlessAccount`): **69 ms** — initializerless, no deploy tx, as designed - `setup:bot` (4 nested hooks): **149,582 ms = 97.3%** of the file's beforeAll - transaction-bot `Bot.create`: 32.2s (token deploy class+instance, then private+public mint) - bridge-resume `new BotStore(...)`: 0.9 ms - amm-bot `AmmBot.create`: 73.9s (deploys 4 contracts + set_minter + mint + add_liquidity) - cross-chain-bot `CrossChainBot.create`: 43.5s (TestContract deploy + seed 2 L1→L2 msgs + wait ready) **Root cause: structural, not a bug.** The bot suite runs the production Sequencer to exercise CHECKPOINTED/PROPOSED follow modes and L1 fee-juice bridging, so each `Bot.create` deploys real contracts and mints serially at slot cadence (`bot/src/factory.ts`). The two calls the shortlist suspected are ~0.1s each. No fix is shipped here because there is no safe one-liner: speeding this up means batching/parallelizing the `bot/src/factory.ts` deploy/mint paths (production code, PR-6-shaped), genesis-seeding (PR 1/2), or cutting the CI slot time (PR 5) — none belong in the instrumentation PR. Tracked as a round-5 item; the new `setup:bot` span makes that win measurable. Full write-up in `tmp/SPEEDUP_ROUND4_FINDINGS.md`. ## Expected effect No wall-clock change (passthrough spans). The deliverable is visibility: the previously-invisible `setup:bot` / `setup:wallet` / `wallet:create` decomposition of the bot beforeAll, and the `wait:proven-checkpoint` / `warp:proven-checkpoint-epoch` residual in the fees suites. Measured numbers from a full green CI run will be appended below once available. ## Measured impact Both runs are full CI runs. PR run: `ci/x-fast` on head `1f7898cd2b` (CI 1783345614459429, 1,799 test lines). Baseline: `ci/x-full-no-test-cache` on the exact merge-base `a8cfa93df5` (CI 1783341726388943, 1,863 test lines). No wall-clock change claimed (spans are passthrough; suite-level noise floor is ~11s median / ~52s p90) — the deliverable is attribution, measured as span `busyMs`. Bot suite (`suite` line, beforeAll = 181.5s on the PR run vs 182.1s baseline): - Baseline: 7.4s tagged (`setup:env:*` only) → **174.8s / 96% invisible**. - PR run: `setup:bot` **174,203 ms** (4 occurrences, max 86,961 ms = the amm-bot hook), `setup:wallet` **250 ms**, `wallet:create` **144 ms**; residual now **−1.5s ≈ 0** → **100% of the beforeAll is attributed**. - This confirms the diagnosis: `EmbeddedWallet.create` + `createSchnorrInitializerlessAccount` are ~0.4s combined; the "invisible 178s" is the four nested describes' bot-factory setup (deploys + mints at production 12s-slot cadence), folded into the file's suite line by the hook accounting. Fees suites (`private_payments.parallel` × 8 shards + `failures`): - Baseline: avg **32.3s/shard untagged residual** (zero `wait:proven-checkpoint` / `warp:proven-checkpoint-epoch` lines on these suites). - PR run: `wait:proven-checkpoint` fires on all 9 suite hooks, **293,888 ms total** (avg 32.7s/shard — matches the residual exactly), plus one in-test occurrence (32.1s); `warp:proven-checkpoint-epoch` 19 occurrences, 361 ms total (the warp RPC itself is near-free — the cost is entirely the proven-chain poll that follows). Per-shard residual is now ≈ −1s, i.e. fully attributed. Net: **~469s/run of previously invisible setup/wait cost is now tagged and attributable** (174.2s `setup:bot` + 0.4s wallet spans + 294.3s fees proven-chain waits), with no behavior or wall-clock change. Fixes A-1407 (cherry picked from commit 1476179)

Parallelizes the independent setup steps in the bot factory (`@aztec/bot`, `bot/src/factory.ts`). This is production bot code that runs against live networks, so the changes are strictly behavior-preserving: same contracts at the same (salt-derived) addresses, same amounts, same final state. Only the ordering/concurrency of provably-independent setup steps changes. The bot factory is the single largest cost in the `single-node/bot` e2e suite: each `*.create` runs the production sequencer at 12s L2 slots (`PIPELINING_SETUP_OPTS`) and deploys/mints one tx per slot, fully serial. `AmmBot.create` alone was ~74s (7 serial txs). ## Dependency graph per bot type Edges below are "must be mined before", verified against the Noir contract source (`token_contract`, `amm_contract`). ### Transaction bot -- `setup()` (unchanged) `setupAccount` -> register recipient -> `ensureFeeJuiceBalance` -> deploy token -> mint. This chain is fully data-dependent: funding gates the deploy, the deploy gates the mint. The recipient (`createSchnorrAccount`) is register-only (no tx), and the private+public mints are already batched into one tx. Nothing is independent, so `setup()` is left serial. ### AMM bot -- `setupAmm()` Steps: deploy token0 (D0), token1 (D1), LP token (DL), AMM (DA); `set_minter` granting the AMM rights over the LP token (SM); mint token0+token1 to the provider (M, one batched tx); `add_liquidity` (AL). - D0, D1, DL: independent -- distinct contracts, no cross-references. - DA: the AMM constructor only stores the three token addresses (no calls into the tokens). Those addresses are salt-derived, so DA needs only the (pre-derived) addresses, not the token deploys mined. - SM: `Token::set_minter` just writes `minters[amm] = true`; it does not call or validate the AMM contract. It needs the LP token deployed and the (derivable) AMM address only -- not the AMM deployed. The deployer is authorized because the Token constructor sets `minters[admin] = true`. - M: `mint_to_private`/`mint_to_public` require the caller be a minter; the deployer is admin=minter from the constructor, so the token0/token1 mints need only those tokens deployed (no `set_minter`). - AL: `add_liquidity` transfers token0/token1 from the provider (needs M for balances), mints LP tokens (needs SM so the AMM is an LP minter), and targets the AMM (needs DA). Restructured from 7 serial txs into 3 slot-phases: - Phase 1: `Promise.all([D0, D1, DL])` - Phase 2: `Promise.all([DA, SM, M])` - Phase 3: `AL` ### Cross-chain bot -- `setupCrossChain()` Steps: deploy TestContract (an L2 tx), seed N L1->L2 messages (L1 txs), wait for the first message ready. - The seeds reference only the L2 recipient (TestContract) address, which is salt-derived. L1->L2 messages are queued on L1 and do not require the L2 contract to exist yet; they are consumed later in `bot.run()`, after setup completes. So the L2 deploy and the L1 seeding are independent. - The L2 deploy pays via the L2 wallet/PXE account; the seeds use the bot's L1 client -- different accounts, so there is no L1 nonce interaction between them. Restructured to overlap the deploy with the seed loop: `Promise.all([deploy TestContract, seed loop])`, then the message-ready wait. ## What stayed serial, and why - The whole transaction-bot `setup()` (data-dependent chain, nothing independent). - `ensureFeeJuiceBalance` before every deploy (funding must precede spending). - `add_liquidity` after its mints + minter grant + AMM deploy. - The individual L1->L2 seeds inside the loop: they share the bot's single L1 account, so concurrent sends would race on the L1 nonce. The suite already documents this nonce hazard. Only the loop as a whole overlaps the (L2) TestContract deploy. ## Production safety / idempotent re-entry - Every deploy still goes through `registerOrDeployContract`, which checks `getContractMetadata(address).isContractPublished` per contract and only registers (no tx) if already deployed. If one parallel deploy fails and the others succeed, the next `*.create` recovers: the succeeded contracts register-only, the failed one redeploys. Addresses are salt-derived and identical across runs. - `set_minter` is idempotent (writes `true` again). The AMM path's mint and `add_liquidity` always run (no balance guard) -- unchanged from the previous `fundAmm`. - Same-sender concurrent `.send()`s are safe by construction: the PXE serializes simulate/prove through its `SerialQueue` and setup txs pay with random tx nonces. The bot runs the same `EmbeddedWallet` -> PXE path in the e2e suite and in production. - Deliberate, benign failure-path change: because Phase 2 runs DA/SM/M concurrently, a failure of one no longer prevents the others from being sent. Their effects (a deployed-but-unused AMM, a minter grant, an extra mint) are all idempotent-safe, and `add_liquidity` uses fixed amounts, so re-entry converges to the same final state. - `withNoMinTxsPerBlock` (the `flushSetupTransactions` save/zero/restore wrapper around each setup tx) was not safe to re-enter concurrently: interleaved save/zero/restore could read the already-zeroed value and "restore" `minTxsPerBlock` to 0 permanently. It now reference-counts entrants -- the first saves and zeroes, the last restores -- with unit tests covering the overlapping and failure paths (`bot/src/factory.test.ts`). Serial callers see the exact same RPC sequence as before. Final state is identical in every path: same contract addresses, same minter grant, same minted amounts, same liquidity. ## Local evidence Two full local runs of `single-node/bot/bot.test.ts` (production sequencer, 12s L2 slots), all 10 tests passing in both. Hook durations from factory log timestamps, against the round-4 local baseline measured on the same machine at the same cadence (findings from #24534): - `Bot.create` (transaction-bot hook, untouched): 31.8s vs 32.2s baseline -- unchanged, as intended. - `AmmBot.create`: 57.2s vs 73.9s baseline (-16.7s). The three token deploy txs go out within 300ms of each other; the AMM deploy, `set_minter` and the mint batch all go out within the following slot. - `CrossChainBot.create`: 24.0s vs 43.5s baseline (-19.5s). The TestContract deploy overlaps both L1 seeds; the first message is ready almost immediately after the deploy is mined. - Suite-level `beforeHooksMs`: 116.9s vs 153.7s baseline (-36.8s), matching the per-hook deltas. - Unit: `bot/src/factory.test.ts` (4 tests) covering the `withNoMinTxsPerBlock` reentrancy fix. ## Expected effect on CI - `AmmBot.create`: 7 serial txs -> 3 slot-phases (~87s on CI per #24534's spans -> ~55-60s). - `CrossChainBot.create`: deploy overlaps the seed pipeline (~43s -> ~25s). - Transaction bot: unchanged. - Aggregate: roughly -35 to -45s on the bot suite's beforeAll wall time. - Fix-phase proof metric (once #24534's `setup:bot` instrumentation is on this base): the amm-bot and cross-chain `setup:bot` occurrences drop per the numbers above. Measured CI numbers to be appended when a full run lands. ## Measured impact The bot suite (a single shard) drops 216.3s → 118.3s total (**−98.0s, −45%**); beforeHooks 215.9s → 118.0s. - No `setup:bot` spans exist on this base (that instrumentation lives in the unmerged #24534), so the metric is the suite-level hook fold, which covers all three nested bot factory setups (Bot, AmmBot, CrossChainBot). - Noise context from the same run pair: untouched light suites move −2 to −3s per shard (private_initialization −2.9s/shard over 20 shards, deploy_method −2.1s/shard over 14), and the largest counter-move is validators_sentinel +24s (multi-node variance). A −98s single-shard delta is far outside that envelope. - The CI delta exceeds the local −36.8s because the CI baseline pays more missed-slot penalties per serial tx (baseline beforeHooks 215.9s on CI vs 153.7s locally at the same 12s cadence); collapsing 7 serial txs into 3 slot-phases removes proportionally more where slots are more often missed. Baseline CI run 1783393028773172 (merge-base 30966a4, full). PR CI run 1783427250198215 (x-fast). Fixes A-1408 (cherry picked from commit 5d2b6df)

…l#24534) ## What / why Round-4 e2e speedup, PR 3 of the effort. The `single-node/bot` suite's file-level `beforeAll` costs ~178s on CI with 97% untagged, and the residual proven-chain waits in the `private_payments`/`failures` fees suites (~32s/shard) were also invisible. This PR instruments both so the cost is attributable, and diagnoses the bot residual. Spans are pure passthrough when `TEST_TIMING_FILE` is unset (see `fixtures/timing.ts`), so this is zero behavior change for normal runs and CI. ## Commit - `test(e2e): instrument bot suite setup and fees proven-chain residuals` - Bot suite (`single-node/bot/bot.test.ts`): `setup:wallet` around `EmbeddedWallet.create`, `wallet:create` around `createSchnorrInitializerlessAccount`, and `setup:bot` around the four nested `describe` beforeAll hooks (`Bot.create` / `AmmBot.create` / `CrossChainBot.create` / `BotStore`). - Fees harness (`single-node/fees`): `wait:proven-checkpoint` around `catchUpProvenChain`'s poll loop, and `warp:proven-checkpoint-epoch` around `advanceToNextEpoch` via a new instrumented `FeesTest.advanceToNextEpoch()` wrapper; `waitForEpochProven` and the direct call sites in `failures`/`private_payments` now route through it. - Reuses existing tags where the concept matches (`wallet:create` is already what `createFundedInitializerlessAccounts` fires; `warp:proven-checkpoint-epoch` is already what `advanceToNextEpoch` fires in `block-production/setup.ts`) rather than minting near-duplicates. ## Diagnosis The timing environment (`shared/timing_env.mjs`) folds *every* `beforeAll` in a file — the file-level one plus each nested `describe`'s — into a single suite-scoped `beforeHooksMs`, and every span fired during any beforeAll lands on that same suite line. The shortlist read the whole number as the file-level hook and, seeing only `setup:env:*` tagged, attributed the rest to the one visible untagged file-level call (`createSchnorrInitializerlessAccount`). It is actually the nested hooks. Local read (`TEST_TIMING_SPANS=1`, `PIPELINING_SETUP_OPTS` = 12s L2 slots, production sequencer, fake prover), suite `beforeHooksMs = 153,733 ms`: - `setup:env:*` (file-level `setup()`): ~4.6s — already tagged - `setup:wallet` (`EmbeddedWallet.create`, ephemeral): **137 ms** — the PXE has `autoSync:false`, no genesis walk - `wallet:create` (`createSchnorrInitializerlessAccount`): **69 ms** — initializerless, no deploy tx, as designed - `setup:bot` (4 nested hooks): **149,582 ms = 97.3%** of the file's beforeAll - transaction-bot `Bot.create`: 32.2s (token deploy class+instance, then private+public mint) - bridge-resume `new BotStore(...)`: 0.9 ms - amm-bot `AmmBot.create`: 73.9s (deploys 4 contracts + set_minter + mint + add_liquidity) - cross-chain-bot `CrossChainBot.create`: 43.5s (TestContract deploy + seed 2 L1→L2 msgs + wait ready) **Root cause: structural, not a bug.** The bot suite runs the production Sequencer to exercise CHECKPOINTED/PROPOSED follow modes and L1 fee-juice bridging, so each `Bot.create` deploys real contracts and mints serially at slot cadence (`bot/src/factory.ts`). The two calls the shortlist suspected are ~0.1s each. No fix is shipped here because there is no safe one-liner: speeding this up means batching/parallelizing the `bot/src/factory.ts` deploy/mint paths (production code, PR-6-shaped), genesis-seeding (PR 1/2), or cutting the CI slot time (PR 5) — none belong in the instrumentation PR. Tracked as a round-5 item; the new `setup:bot` span makes that win measurable. Full write-up in `tmp/SPEEDUP_ROUND4_FINDINGS.md`. ## Expected effect No wall-clock change (passthrough spans). The deliverable is visibility: the previously-invisible `setup:bot` / `setup:wallet` / `wallet:create` decomposition of the bot beforeAll, and the `wait:proven-checkpoint` / `warp:proven-checkpoint-epoch` residual in the fees suites. Measured numbers from a full green CI run will be appended below once available. ## Measured impact Both runs are full CI runs. PR run: `ci/x-fast` on head `1f7898cd2b` (CI 1783345614459429, 1,799 test lines). Baseline: `ci/x-full-no-test-cache` on the exact merge-base `a8cfa93df5` (CI 1783341726388943, 1,863 test lines). No wall-clock change claimed (spans are passthrough; suite-level noise floor is ~11s median / ~52s p90) — the deliverable is attribution, measured as span `busyMs`. Bot suite (`suite` line, beforeAll = 181.5s on the PR run vs 182.1s baseline): - Baseline: 7.4s tagged (`setup:env:*` only) → **174.8s / 96% invisible**. - PR run: `setup:bot` **174,203 ms** (4 occurrences, max 86,961 ms = the amm-bot hook), `setup:wallet` **250 ms**, `wallet:create` **144 ms**; residual now **−1.5s ≈ 0** → **100% of the beforeAll is attributed**. - This confirms the diagnosis: `EmbeddedWallet.create` + `createSchnorrInitializerlessAccount` are ~0.4s combined; the "invisible 178s" is the four nested describes' bot-factory setup (deploys + mints at production 12s-slot cadence), folded into the file's suite line by the hook accounting. Fees suites (`private_payments.parallel` × 8 shards + `failures`): - Baseline: avg **32.3s/shard untagged residual** (zero `wait:proven-checkpoint` / `warp:proven-checkpoint-epoch` lines on these suites). - PR run: `wait:proven-checkpoint` fires on all 9 suite hooks, **293,888 ms total** (avg 32.7s/shard — matches the residual exactly), plus one in-test occurrence (32.1s); `warp:proven-checkpoint-epoch` 19 occurrences, 361 ms total (the warp RPC itself is near-free — the cost is entirely the proven-chain poll that follows). Per-shard residual is now ≈ −1s, i.e. fully attributed. Net: **~469s/run of previously invisible setup/wait cost is now tagged and attributable** (174.2s `setup:bot` + 0.4s wallet spans + 294.3s fees proven-chain waits), with no behavior or wall-clock change. Fixes A-1407 (cherry picked from commit 1476179)

…ocol#24581) Parallelizes the independent setup steps in the bot factory (`@aztec/bot`, `bot/src/factory.ts`). This is production bot code that runs against live networks, so the changes are strictly behavior-preserving: same contracts at the same (salt-derived) addresses, same amounts, same final state. Only the ordering/concurrency of provably-independent setup steps changes. The bot factory is the single largest cost in the `single-node/bot` e2e suite: each `*.create` runs the production sequencer at 12s L2 slots (`PIPELINING_SETUP_OPTS`) and deploys/mints one tx per slot, fully serial. `AmmBot.create` alone was ~74s (7 serial txs). ## Dependency graph per bot type Edges below are "must be mined before", verified against the Noir contract source (`token_contract`, `amm_contract`). ### Transaction bot -- `setup()` (unchanged) `setupAccount` -> register recipient -> `ensureFeeJuiceBalance` -> deploy token -> mint. This chain is fully data-dependent: funding gates the deploy, the deploy gates the mint. The recipient (`createSchnorrAccount`) is register-only (no tx), and the private+public mints are already batched into one tx. Nothing is independent, so `setup()` is left serial. ### AMM bot -- `setupAmm()` Steps: deploy token0 (D0), token1 (D1), LP token (DL), AMM (DA); `set_minter` granting the AMM rights over the LP token (SM); mint token0+token1 to the provider (M, one batched tx); `add_liquidity` (AL). - D0, D1, DL: independent -- distinct contracts, no cross-references. - DA: the AMM constructor only stores the three token addresses (no calls into the tokens). Those addresses are salt-derived, so DA needs only the (pre-derived) addresses, not the token deploys mined. - SM: `Token::set_minter` just writes `minters[amm] = true`; it does not call or validate the AMM contract. It needs the LP token deployed and the (derivable) AMM address only -- not the AMM deployed. The deployer is authorized because the Token constructor sets `minters[admin] = true`. - M: `mint_to_private`/`mint_to_public` require the caller be a minter; the deployer is admin=minter from the constructor, so the token0/token1 mints need only those tokens deployed (no `set_minter`). - AL: `add_liquidity` transfers token0/token1 from the provider (needs M for balances), mints LP tokens (needs SM so the AMM is an LP minter), and targets the AMM (needs DA). Restructured from 7 serial txs into 3 slot-phases: - Phase 1: `Promise.all([D0, D1, DL])` - Phase 2: `Promise.all([DA, SM, M])` - Phase 3: `AL` ### Cross-chain bot -- `setupCrossChain()` Steps: deploy TestContract (an L2 tx), seed N L1->L2 messages (L1 txs), wait for the first message ready. - The seeds reference only the L2 recipient (TestContract) address, which is salt-derived. L1->L2 messages are queued on L1 and do not require the L2 contract to exist yet; they are consumed later in `bot.run()`, after setup completes. So the L2 deploy and the L1 seeding are independent. - The L2 deploy pays via the L2 wallet/PXE account; the seeds use the bot's L1 client -- different accounts, so there is no L1 nonce interaction between them. Restructured to overlap the deploy with the seed loop: `Promise.all([deploy TestContract, seed loop])`, then the message-ready wait. ## What stayed serial, and why - The whole transaction-bot `setup()` (data-dependent chain, nothing independent). - `ensureFeeJuiceBalance` before every deploy (funding must precede spending). - `add_liquidity` after its mints + minter grant + AMM deploy. - The individual L1->L2 seeds inside the loop: they share the bot's single L1 account, so concurrent sends would race on the L1 nonce. The suite already documents this nonce hazard. Only the loop as a whole overlaps the (L2) TestContract deploy. ## Production safety / idempotent re-entry - Every deploy still goes through `registerOrDeployContract`, which checks `getContractMetadata(address).isContractPublished` per contract and only registers (no tx) if already deployed. If one parallel deploy fails and the others succeed, the next `*.create` recovers: the succeeded contracts register-only, the failed one redeploys. Addresses are salt-derived and identical across runs. - `set_minter` is idempotent (writes `true` again). The AMM path's mint and `add_liquidity` always run (no balance guard) -- unchanged from the previous `fundAmm`. - Same-sender concurrent `.send()`s are safe by construction: the PXE serializes simulate/prove through its `SerialQueue` and setup txs pay with random tx nonces. The bot runs the same `EmbeddedWallet` -> PXE path in the e2e suite and in production. - Deliberate, benign failure-path change: because Phase 2 runs DA/SM/M concurrently, a failure of one no longer prevents the others from being sent. Their effects (a deployed-but-unused AMM, a minter grant, an extra mint) are all idempotent-safe, and `add_liquidity` uses fixed amounts, so re-entry converges to the same final state. - `withNoMinTxsPerBlock` (the `flushSetupTransactions` save/zero/restore wrapper around each setup tx) was not safe to re-enter concurrently: interleaved save/zero/restore could read the already-zeroed value and "restore" `minTxsPerBlock` to 0 permanently. It now reference-counts entrants -- the first saves and zeroes, the last restores -- with unit tests covering the overlapping and failure paths (`bot/src/factory.test.ts`). Serial callers see the exact same RPC sequence as before. Final state is identical in every path: same contract addresses, same minter grant, same minted amounts, same liquidity. ## Local evidence Two full local runs of `single-node/bot/bot.test.ts` (production sequencer, 12s L2 slots), all 10 tests passing in both. Hook durations from factory log timestamps, against the round-4 local baseline measured on the same machine at the same cadence (findings from AztecProtocol#24534): - `Bot.create` (transaction-bot hook, untouched): 31.8s vs 32.2s baseline -- unchanged, as intended. - `AmmBot.create`: 57.2s vs 73.9s baseline (-16.7s). The three token deploy txs go out within 300ms of each other; the AMM deploy, `set_minter` and the mint batch all go out within the following slot. - `CrossChainBot.create`: 24.0s vs 43.5s baseline (-19.5s). The TestContract deploy overlaps both L1 seeds; the first message is ready almost immediately after the deploy is mined. - Suite-level `beforeHooksMs`: 116.9s vs 153.7s baseline (-36.8s), matching the per-hook deltas. - Unit: `bot/src/factory.test.ts` (4 tests) covering the `withNoMinTxsPerBlock` reentrancy fix. ## Expected effect on CI - `AmmBot.create`: 7 serial txs -> 3 slot-phases (~87s on CI per AztecProtocol#24534's spans -> ~55-60s). - `CrossChainBot.create`: deploy overlaps the seed pipeline (~43s -> ~25s). - Transaction bot: unchanged. - Aggregate: roughly -35 to -45s on the bot suite's beforeAll wall time. - Fix-phase proof metric (once AztecProtocol#24534's `setup:bot` instrumentation is on this base): the amm-bot and cross-chain `setup:bot` occurrences drop per the numbers above. Measured CI numbers to be appended when a full run lands. ## Measured impact The bot suite (a single shard) drops 216.3s → 118.3s total (**−98.0s, −45%**); beforeHooks 215.9s → 118.0s. - No `setup:bot` spans exist on this base (that instrumentation lives in the unmerged AztecProtocol#24534), so the metric is the suite-level hook fold, which covers all three nested bot factory setups (Bot, AmmBot, CrossChainBot). - Noise context from the same run pair: untouched light suites move −2 to −3s per shard (private_initialization −2.9s/shard over 20 shards, deploy_method −2.1s/shard over 14), and the largest counter-move is validators_sentinel +24s (multi-node variance). A −98s single-shard delta is far outside that envelope. - The CI delta exceeds the local −36.8s because the CI baseline pays more missed-slot penalties per serial tx (baseline beforeHooks 215.9s on CI vs 153.7s locally at the same 12s cadence); collapsing 7 serial txs into 3 slot-phases removes proportionally more where slots are more often missed. Baseline CI run 1783393028773172 (merge-base 30966a4, full). PR CI run 1783427250198215 (x-fast). Fixes A-1408 (cherry picked from commit 5d2b6df)

…rotocol#24930) Forward-ports the **end-to-end / testing** slice of the v5-next → next backlog (work merged to `v5-next` after the ~2026-07-08 cut that reshaped `next`). ## Applied (clean cherry-picks, chronological) - AztecProtocol#24518 docs(e2e): update READMEs for the consolidated suite layout - AztecProtocol#24534 test(e2e): instrument and diagnose bot suite setup cost - AztecProtocol#24566 perf(e2e): warp dead waits in multi-node recovery and proving tests - AztecProtocol#24570 perf(e2e): shrink e2e slot times - AztecProtocol#24564 perf(e2e): seed BananaFPC fee juice at genesis instead of bridging - AztecProtocol#24568 perf(e2e): seed standard contracts at genesis - AztecProtocol#24597 chore: add writing-e2e-tests skill - AztecProtocol#24590 feat(e2e): interactive handshake e2e - AztecProtocol#24671 test: fix proof_boundary startup race ## ⚠️ Needs owner conflict-resolution (conflict against reshaped `next`; not included here) Cherry-pick onto this branch and resolve: - [ ] AztecProtocol#24569 `git cherry-pick -x 1d280af` — perf(e2e): overlap and batch setup txs in e2e harnesses - [ ] AztecProtocol#24479 `git cherry-pick -x da9ac1c` — feat: assert non revertible phase when setting fee payer Part of the manual v5-next backlog sweep. Draft until owners resolve the conflicts above and CI is green.

spalladino added the ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure label Jul 6, 2026

spalladino force-pushed the spl/e2e-bot-setup-spans branch from 1f7898c to 46074c2 Compare July 6, 2026 21:50

spalladino mentioned this pull request Jul 7, 2026

perf(bot): parallelize independent bot factory setup steps #24581

Merged

spalladino added wip Work in progress and removed wip Work in progress labels Jul 7, 2026

PhilWindle approved these changes Jul 8, 2026

View reviewed changes

PhilWindle merged commit 1476179 into merge-train/spartan-v5 Jul 8, 2026
32 checks passed

PhilWindle deleted the spl/e2e-bot-setup-spans branch July 8, 2026 09:59

AztecBot mentioned this pull request Jul 8, 2026

feat: merge-train/spartan-v5 #24606

Merged

AztecBot mentioned this pull request Jul 21, 2026

chore: port v5-next to next #24825

Open

PhilWindle mentioned this pull request Jul 21, 2026

chore: forward-port clean v5-next backlog to next #24849

Closed

PhilWindle mentioned this pull request Jul 23, 2026

chore(port): forward-port v5-next e2e/testing backlog to next #24930

Merged

2 tasks

Navigation Menu

Uh oh!

test(e2e): instrument and diagnose bot suite setup cost - #24534

test(e2e): instrument and diagnose bot suite setup cost#24534
PhilWindle merged 1 commit into
merge-train/spartan-v5from
spl/e2e-bot-setup-spans

spalladino commented Jul 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

spalladino commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What / why

Commit

Diagnosis

Expected effect

Measured impact

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

spalladino commented Jul 6, 2026 •

edited

Loading