fix(ethereum): broadcast L1 deploy txs one at a time on anvil to dodge automine race - #24378
Merged
spalladino merged 3 commits intoJun 29, 2026
Merged
Conversation
spalladino
force-pushed
the
spl/forge-broadcast-batch-size-1
branch
from
June 29, 2026 18:27
7e0256b to
d9ed663
Compare
The deploy wrapper script was matched by no rebuild pattern, so changes to it left both the l1-contracts and yarn-project content hashes unchanged and CI served cached (skipped) test runs. yarn-project's hash folds in l1-contracts/.rebuild_patterns, so registering the file there covers both.
Broadcasting one tx at a time dodges anvil's automine batch race, but it serializes the deploy to one tx per block under interval mining. The e2e suites that deploy L1 contracts with automineL1Setup:false run anvil at a 4s block interval, so --batch-size 1 stalled each deploy tx for a full block and blew past the broadcast timeout. Gate --batch-size 1 on anvil_getAutomine so interval-mining anvil (and real chains) keep the larger batch size, where no automine race exists.
spalladino
force-pushed
the
spl/forge-broadcast-batch-size-1
branch
from
June 29, 2026 19:41
c293af9 to
bab9650
Compare
spalladino
enabled auto-merge (squash)
June 29, 2026 20:09
ludamad
approved these changes
Jun 29, 2026
This was referenced Jun 29, 2026
spalladino
pushed a commit
that referenced
this pull request
Jun 29, 2026
…ut as the known flake (#24385) ## What Extends the existing `.test_patterns.yml` flake entry for `src/composed/e2e_token_bridge_tutorial_test.test.ts` so it also matches the viem `WaitForTransactionReceiptTimeoutError` confirmation-timeout failure mode, not just the jest per-test timeout. ## Why — the `merge-train/spartan-v5` CI red on this branch The train CI run failed on this one test: ``` FAIL src/composed/e2e_token_bridge_tutorial_test.test.ts ✕ Deploys tokens & bridges to L1 & L2, mints & publicly bridges tokens (189531 ms) WaitForTransactionReceiptTimeoutError: Timed out while waiting for transaction with hash "0x01f3fb09…" to be confirmed. Version: viem@2.38.2 ``` (test log: http://ci.aztec-labs.com/1643e817d7242e0e) This test is **already known-flaky** — there is an entry for it owned by @spalladino — but its `error_regex` only covered `Exceeded timeout of N ms for a test` (the jest per-test budget). In this run the test failed *faster* (~189s of its 900s budget) because viem's own per-tx confirmation timeout fired first, producing a different error string. The flake matcher (`grep -E` over the test log in `ci3/get_test_entry`) therefore did not classify it, so the failure went red and blocked the train. The L1↔L2 bridging flow sends a series of L1 txs against an automining anvil that the local-network sequencer and cheat codes also drive concurrently. Under load a tx can sit unmined past viem's confirmation window — the same automine-race class, surfacing here in the test's own L1 txs. Both observed symptoms (jest timeout, viem confirmation timeout) are the same flake. ## Not caused by #24378 The train HEAD is #24378 (`broadcast L1 deploy txs one at a time on anvil`), but that change only touches `l1-contracts/scripts/forge_broadcast.js` (the L1 *contract deploy* path). In the failing run that deploy **succeeded** (`[forge_broadcast] Broadcast succeeded` at 20:52:19); the failure is ~3.5 min later in the test body, which never goes through `forge_broadcast.js`. In every version (pre-#24316, #24316, #24378) anvil ends in automine after deploy, so #24378 does not change the test-body mining environment. #24378 was simply the train HEAD when this pre-existing flake surfaced in a mode the pattern didn't catch. ## Verification A full `./bootstrap.sh ci` run is neither meaningful nor feasible for a flake-classification config change (and the test is non-deterministic by nature). The meaningful check is that `ci3/get_test_entry` now classifies the observed failure: - `yq` parses the updated entry cleanly. - `get_test_entry "<compose token_bridge_tutorial cmd>" <log-with-WaitForTransactionReceiptTimeoutError>` now returns the entry (→ flake, non-blocking). - A negative control (an unrelated error line) still returns nothing, so the broadened regex stays scoped to this test's timeout flake. --- *Created by [claudebox](https://claudebox.work/v2/sessions/c01cbe3dd422bf00) · group: `slackbot`*
This was referenced Jun 30, 2026
PhilWindle
pushed a commit
that referenced
this pull request
Jun 30, 2026
Speeds up the slowest individual e2e tests. Shared clock-warp helpers (`warpToEpochStart`, `warpNearSubmissionWindowEnd`) were also lifted into `single_node_test_context.ts`. Further production and shared-fixture speedup opportunities found along the way are catalogued for a follow-up PR. ## Measured speedup From full instrumented CI runs (per-test timing JSONL), comparing this PR's green run (`40b792f7`, CI `1782821707760521`) against the nearest healthy baseline on the same post-consolidation merge-train era (`bab9650f` / #24378). Timings are **sums across parallel CI processes, not wall-clock**. - **Across the 24 changed suites: −38.8% total time** (11,002s → 6,735s, ≈71 min of summed process time saved per run), **−38.7% body**. Every changed suite got faster. - **Whole e2e suite (~630 tests): −20.1% overall** (8h 44m → 6h 59m), **−21.1% body** — the changed suites account for ~85% of the total body reduction; the rest is run-to-run noise across unchanged tests. Biggest wins (total time, base → PR): | Suite | Base | PR | Δ | | -- | -- | -- | -- | | `proof_boundary.parallel` | 2535s | 1333s | −1202s (−47%) | | `multiple_validators_sentinel.parallel` | 617s | 123s | −495s (−80%)* | | `optimistic.parallel` | 2249s | 1775s | −474s (−21%) | | `world_state_pruning` | 350s | 113s | −236s (−68%) | | `publisher_funding_multi` | 253s | 17s | −236s (−93%) | | `escape_hatch_vote_only` | 260s | 89s | −170s (−66%) | | `sync_after_reorg` | 339s | 182s | −156s (−46%) | | `sentinel_status_slash.parallel` | 306s | 165s | −140s (−46%) | | `upload_failed_proof` | 187s | 67s | −120s (−64%) | | `long_proving_time` | 390s | 270s | −119s (−31%) | …and 14 more suites, all faster (full per-suite table in the Linear "Times tracking" doc). *`multiple_validators_sentinel` merged two overlapping attestation tests into one (no coverage lost). ## Strategies used - **Clock warping** — fast-forward cheat-clock waits (proof-submission windows, epoch/proven-checkpoint tails, dead slots) instead of sleeping in real time. - **Slot-cadence cut** — shrink L1/L2 slot durations (down to the 4s/12s floor) for single-node prover tests that march through empty checkpoints in real wall-clock. - **Shared setup** — collapse per-`it` `beforeEach` setups into a single `beforeAll` where the tests are independent. - **Drive the chain directly** — replace real-time tx-confirmation waits with `NO_WAIT` send + `advanceToSlot`, and trim redundant tx counts. A handful of speedups were reverted or tuned after CI showed they raced the production sequencer (warp loops) or broke fresh-per-test state; those are catalogued for the follow-up PR, along with a production prover-shutdown defect found in passing. ## Tests changed **Single-node** - proving: `multi_proof`, `optimistic`, `proof_fails`, `world_state_pruning`, `cross_chain_public_message`, `long_proving_time`, `upload_failed_proof` - partial-proofs: `multi_root`, `single_root` - recovery: `manual_rollback`, `prune_when_cannot_build`, `sync_after_reorg` - l1-reorgs: `blocks` - sequencer: `escape_hatch_vote_only`, `publisher_funding_multi` **Multi-node** - block-production: `blob_promotion`, `proof_boundary` - governance: `add_rollup` - slashing: `multiple_validators_sentinel`, `sentinel_status_slash` **p2p** - `gossip_network`, `fee_asset_price_oracle_gossip`, `late_prover_tx_collection` **automine** - `ordering` Speedups to `empty_blocks`, `block_building`, `account_init`, `messages`, `l1_to_l2`, and `l2_to_l1` were reverted to baseline after CI flagged instability, and are left for the follow-up PR.
PhilWindle
pushed a commit
that referenced
this pull request
Jun 30, 2026
## Context The first checkpoint publish in e2e setup pays two independent ~2.2s KZG trusted-setup inits serially (~4.4s total): - **Ours** — the `@crate-crypto/node-eth-kzg` singleton (`getKzg()`), a synchronous, event-loop-blocking precomp build that inits lazily on the first blob commitment computation. - **Anvil's own** — anvil loads its own KZG trusted setup the first time it validates a blob sidecar in `eth_sendRawTransaction`. Both land on the first publish, one after the other. ## Approach Add `warmBlobKzg` to `@aztec/ethereum/test` and call it during `setupInner`, just before the L1 contract deploy. It warms our kzg and anvil's in parallel with each other: - Warming anvil needs no real kzg — a pure-JS fake sidecar (48-byte garbage commitment + proof) makes anvil load its trusted setup to validate, then reject the tx. No block is mined and no nonce is burned. - The recipe is deterministic: all RPC round trips happen up front (`prepareTransactionRequest` + `signTransaction`), so only a single socket write needs to flush (one `setImmediate` yield) before the loop-blocking `getKzg()` call. That gives full overlap (~2.2s measured) without a racy fixed delay — anvil warms in its own process while we build our precomp tables. - The warm-up reuses the injected publisher `l1Client`. The fake-sidecar tx is rejected at admission, so it never enters the pool, mines no block, and burns no nonce — sending it from the publisher account is harmless. - Best-effort: any failure is debug-logged and swallowed, leaving lazy init as the fallback. Gated to locally-started anvil runs (`anvil && isAnvilTestChain`). A behavior-based test (`blob_kzg_warmup.test.ts`) asserts the warm-up resolves without throwing and that a subsequent real blob tx is accepted and mined by anvil. ## Measured impact From the per-process timing JSONL of this branch's full e2e CI run vs. the nearest healthy full run on the same base (#24378). Both are cache-busted full runs; figures are **sums across 316 parallel processes** (~633 real tests), not wall-clock. | Metric | #24378 (base) | This PR | Δ | | -- | --: | --: | --: | | Overall | 31,439,600 ms | 31,297,153 ms | −0.5% | | Before-hooks (setup) | 8,448,072 ms | 8,158,701 ms | −3.4% | | — of which `setup()` | 2,171,459 ms | 2,581,000 ms | +18.9% | | Body | 22,550,764 ms | 22,630,788 ms | +0.4% | The only metric that rises is time inside `setup()` (+18.9%), and that is a **relocation of work, not new cost**: the warm-up moves the ~2.2s `getKzg()` build out of lazy node-init / first-publish and into `setup()`. Per suite, `setup()` time rises while other before-hook time (node/sequencer init) falls by a comparable-or-larger amount — globally `setup()` +410k ms vs. other before-hooks −699k ms — so total before-hooks per process drops and overall is flat-to-slightly-better. The per-process wall-clock win from overlapping anvil's init with ours is real but not separately visible in these summed metrics. Fixes A-1297
spalladino
pushed a commit
that referenced
this pull request
Jul 2, 2026
…encer nonce race (#24386) ## Root cause `e2e_token_bridge_tutorial_test` flakes with `WaitForTransactionReceiptTimeoutError: Timed out while waiting for transaction … to be confirmed` ([example](http://ci.aztec-labs.com/1643e817d7242e0e)). It is **not** caused by #24378 — that PR only touches `forge_broadcast.js` (the L1 *contract deploy*), which succeeds in the failing run; the timeout is ~3.5 min later on one of the test's own L1 txs. The test runs against `aztec start --local-network`, whose `AutomineSequencer` continuously publishes checkpoint txs to L1. In `local-network.ts` the sequencer-publisher and validator keys are derived from `DefaultMnemonic` (`'test test … junk'`) at **address index 0** — `0xf39Fd6…` (visible as the publisher in the log). The test's L1 client was created with the **same** mnemonic and the **same** default index 0: ```ts const l1Client = createExtendedL1Client(ETHEREUM_HOSTS.split(','), MNEMONIC); // → account index 0 ``` So the test and the node's sequencer send L1 txs from one account, sharing a single nonce sequence. Against an automining anvil they race: a test tx can land with a nonce the sequencer just consumed (dropped / "already known"), or with a future-nonce gap that automine won't mine until the gap fills. Either way the tx gets a hash but never confirms, and viem's per-tx confirmation timeout fires. This is intermittent because it only bites when a test tx and a sequencer publish overlap. ## Fix Derive the test's L1 client from a **different** address index so its nonce space is independent of the sequencer's: ```ts const l1Client = createExtendedL1Client(ETHEREUM_HOSTS.split(','), mnemonicToAccount(MNEMONIC, { addressIndex: 1 })); ``` Index 1 (`0x70997970…`) is funded by anvil's default mnemonic and is unused by the local-network node (publisher/validator = index 0; index 2 is the prover and 3+ are attesters by the existing `setup.ts` / `setup_p2p_test.ts` convention, none of which run in local-network). The test is fully self-contained on its own L1 account — it deploys and owns the `TestERC20`, `FeeAssetHandler` and `TokenPortal`, and bridges to/from `l1Client.account.address` — so nothing requires it to be account 0. All of the test's L1 txs (including the `L1TokenManager` / `L1TokenPortalManager`, built from `l1Client`) now go through index 1. ## Verification This is an `compose` e2e test that needs docker + a full network; the container here is a source-only checkout (no `node_modules`, no docker), so I could not run the test or `./bootstrap.sh ci` locally — flagging that explicitly per the repo's red/green policy. The fix is verified by analysis: the timed-out tx is one of the test's own L1 txs, the shared-account nonce race is the established cause of this exact viem symptom, and account index 0 vs 1 are the well-known distinct anvil accounts (idx 0 = `0xf39Fd6…`, the publisher in the log; idx 1 = `0x70997970…`). ## Relationship to #24385 #24385 broadened this test's `.test_patterns.yml` flake pattern to keep CI from going red on the residual flake. This PR removes the underlying cause. The flake entry can stay as a safety net, or be narrowed back to the jest-timeout-only form once this has soaked — happy to follow up either way. --- *Created by [claudebox](https://claudebox.work/v2/sessions/c01cbe3dd422bf00) · group: `slackbot`*
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
anvil's automine races a batched
forge scriptbroadcast: it mines a block on the first ready tx and leaves txs that arrived just after the trigger sitting in the pool, so forge waits forever for their receipts. The previous workaround kept automine on and ran a JS txpool watchdog thatevm_mined any stragglers — extra moving parts on every deploy.This PR is a fix for an issue introduced in #24316.
Approach
Sidestep the race instead of policing it: on anvil, broadcast one tx at a time (
--batch-size 1). With only a single tx in flight there is nothing for the auto-miner to strand, and automine still mines it instantly so deploys stay fast. Real chains keep forge's default batch size.anvil_getAutomineprobe, and all mining-mode manipulation; keepweb3_clientVersionanvil detection only to choose the batch size.FORGE_BROADCAST_TIMEOUT_MS, default 120s anvil / 600s otherwise) that SIGTERMs then SIGKILLs forge and exits non-zero, so a hung broadcast fails fast instead of waiting out the jest hook budget.JSON DEPLOY RESULT:.Validated locally: 20/20 fresh-anvil deploys clean at ~350ms (on par with the watchdog path, far faster than interval mining), and
@aztec/ethereumrollup_cheat_codes.test.tspasses.