fix(ethereum): mine empty L1 blocks without touching the mempool - #24414
Merged
PhilWindle merged 2 commits intoJul 1, 2026
Conversation
The L1Publisher integration timeout tests advance L1 blocks with ethCheatCodes.mineEmptyBlock() while a propose tx is pending (automine off). mineEmptyBlock drops the pending txs, mines, then re-adds them, and the mined block can land at exactly txTimeoutAt with the re-added propose included. The publisher's L1 tx monitor checks "mined" before "timed out", so a propose mined at the timeout boundary is reported as a successful proposal instead of being cancelled, flaking the timeout assertions. Replace mineEmptyBlock with a helper that drops the pending txs and mines without re-adding them, so the in-flight propose stays absent from the chain and the monitor cancels it deterministically once its timeout elapses.
mineEmptyBlock dropped all pending txs, mined a block, then re-added them. That drop->mine->re-add sequence is not atomic against a concurrent publisher whose L1 tx monitor re-broadcasts its in-flight tx: the tx can reappear in the pool during the window and be swept into the supposedly empty block. In the timeout publisher tests this mined a propose the test required to stay pending, flaking the cancel assertion. Mine empty blocks by temporarily lowering the block gas limit below any tx's intrinsic cost (so nothing fits) and then reorging those blocks into empty blocks at the restored gas limit. This never manipulates the mempool, so pending txs are preserved and the race is gone for all callers. Revert the test-local mineBlockWithoutPendingTxs workaround.
PhilWindle
approved these changes
Jul 1, 2026
rangozd
pushed a commit
to rangozd/aztec-packages
that referenced
this pull request
Aug 5, 2026
BEGIN_COMMIT_OVERRIDE refactor(e2e): relocate no-node straggler tests (AztecProtocol#24344) test(e2e): pin gas_estimation public-payment txs to one block to deflake fee comparison (AztecProtocol#24382) feat(archiver): event-trigger L2BlockStream sync from archiver updates (AztecProtocol#24317) test(e2e): speed up individual e2e tests (AztecProtocol#24345) chore(e2e): warm blob KZGs in parallel during setup (AztecProtocol#24383) chore: add perf as a valid PR title prefix (AztecProtocol#24412) test(e2e): adopt shared wait helpers (AztecProtocol#24404) test(e2e): run prover client.test.ts in CI (AztecProtocol#24399) fix(ethereum): mine empty L1 blocks without touching the mempool (AztecProtocol#24414) test(e2e): deflake empty block proving test (AztecProtocol#24411) test(e2e): allocate HA node p2p ports above the ephemeral range to deflake e2e_ha_full (AztecProtocol#24418) fix(sequencer): use evmMine in automine auto-settle to avoid dropping test L1 txs (AztecProtocol#24421) test(e2e): instrument common spans for wall-clock tracking (AztecProtocol#24407) test(e2e): remove redundant reqresp_no_handshake e2e test (AztecProtocol#24424) END_COMMIT_OVERRIDE
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
L1Publisher integration › timeouts › cancels block proposal when the L2 slot endsflakes in CI (failed run http://ci.aztec-labs.com/997f3ffdac47572f). The same commit passes on retry. The failure isexpect(sendRequestsResult).toBeNull()receiving a successful propose result (sentActions: ["propose"], successfulActions: ["propose"]) instead ofnull: the propose got mined when the test required it to stay pending until the L2 slot timeout cancelled it.Root cause
The three
timeoutstests advance L1 blocks withethCheatCodes.mineEmptyBlock()while a propose tx is in flight (anvil automine off). The oldmineEmptyBlockworked by: read the pool,anvil_dropAllTransactions,hardhat_mine, then re-add the raw txs viaeth_sendRawTransaction. That drop -> mine -> re-add sequence is not atomic against the publisher, which is concurrently monitoring its in-flight tx on a 1s loop. During the window between the drop and the mine, the tx reappears in the pool (the publisher re-broadcasts it, andmineEmptyBlockitself re-adds it on the prior iteration), so the supposedly-empty block is not empty —hardhat_minesweeps the pending propose into it. The propose is mined, the monitor reports MINED, andsendRequestsresolves with a successful proposal instead ofnull.Log evidence from the failed run:
Sent L1 transaction 0xd2d5… nonce 0 isBlobTx:true— one propose send, no speed-ups anywhere in the run.Mined 1 empty L1 block(call Initial test issue #1), then on call feat(json-rpc): initial package #2:Failed to re-add transaction: … Details: nonce too low. The re-add of the propose failed because its nonce was already consumed on-chain — i.e.mineEmptyBlock's own mine step had just included it.L1 transaction 0x08fa… with nonce 0 minedblockNumber 36 — the propose mined into the "empty" block.This was reproduced deterministically at the anvil level: with the old implementation, dropping all txs, re-broadcasting (as the publisher monitor does), then
hardhat_mineproduces a block containing the pending tx.The previous explanation in this PR — "the re-added propose lands in the block whose timestamp ==
txTimeoutAt, and the monitor checks mined before timed-out" — was incomplete. The boundary timestamp is incidental (a consequence of the 12s L1-block grid and where the loop happened to be); the real determinant is that the tx is present in the pool whenmineEmptyBlockmines. The monitor's mined-before-timeout ordering is a red herring here: once a tx is mined, it is mined regardless of check order.Fix
Reimplement
mineEmptyBlockso it never touches the mempool, eliminating the race for all callers rather than working around it in one test:hardhat_minecannot include any pending tx; the pending tx stays in the pool untouched.anvil_reorgthe just-mined blocks into empty blocks at the restored gas limit (same height and timestamps). The reorg is required because anvil applies a new gas limit only to future blocks — without it the just-mined blocks would keep the tiny limit and a latereth_callagainstlatest(whose gas is capped by the block gas limit) would revert with "intrinsic gas too high".This is strictly more robust than dropping the pending tx (the approach in #24401 / this PR's prior commit): it removes the drop/re-add window entirely instead of narrowing it, and it preserves pending txs (the existing
eth_cheat_codes.test.tscontract). Consequently the test-localmineBlockWithoutPendingTxsworkaround is removed and the three timeout tests callmineEmptyBlock()again — the change to the test file is a net revert.Verification
hardhat_minemines the pending tx into the "empty" block.eth_cheat_codes.test.ts › mineEmptyBlock › mines an empty block while preserving pending transactions: passes (block advances, block empty, pending tx preserved and mined later).timeoutsintegration tests pass; the cancel test passes 3/3 repeats.ethereum/src/l1_tx_utils/l1_tx_utils.test.ts(51 tests, 22mineEmptyBlockcalls incl. multi-block) passes — the L1TxUtils monitor suite that interacts most withmineEmptyBlock.Relation to #24401
#24401 independently landed the same drop-pending-txs-without-re-add workaround in these tests. This PR supersedes that approach by fixing the shared
mineEmptyBlockhelper, so no per-test workaround is needed.