Skip to content

fix(ethereum): mine empty L1 blocks without touching the mempool - #24414

Merged
PhilWindle merged 2 commits into
merge-train/spartan-v5from
spl/fix-l1-publisher-cancel-flake
Jul 1, 2026
Merged

fix(ethereum): mine empty L1 blocks without touching the mempool#24414
PhilWindle merged 2 commits into
merge-train/spartan-v5from
spl/fix-l1-publisher-cancel-flake

Conversation

@spalladino

@spalladino spalladino commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Problem

L1Publisher integration › timeouts › cancels block proposal when the L2 slot ends flakes in CI (failed run http://ci.aztec-labs.com/997f3ffdac47572f). The same commit passes on retry. The failure is expect(sendRequestsResult).toBeNull() receiving a successful propose result (sentActions: ["propose"], successfulActions: ["propose"]) instead of null: the propose got mined when the test required it to stay pending until the L2 slot timeout cancelled it.

Root cause

The three timeouts tests advance L1 blocks with ethCheatCodes.mineEmptyBlock() while a propose tx is in flight (anvil automine off). The old mineEmptyBlock worked by: read the pool, anvil_dropAllTransactions, hardhat_mine, then re-add the raw txs via eth_sendRawTransaction. That drop -> mine -> re-add sequence is not atomic against the publisher, which is concurrently monitoring its in-flight tx on a 1s loop. During the window between the drop and the mine, the tx reappears in the pool (the publisher re-broadcasts it, and mineEmptyBlock itself re-adds it on the prior iteration), so the supposedly-empty block is not empty — hardhat_mine sweeps the pending propose into it. The propose is mined, the monitor reports MINED, and sendRequests resolves with a successful proposal instead of null.

Log evidence from the failed run:

  • Sent L1 transaction 0xd2d5… nonce 0 isBlobTx:true — one propose send, no speed-ups anywhere in the run.
  • Mined 1 empty L1 block (call Initial test issue #1), then on call feat(json-rpc): initial package #2: Failed to re-add transaction: … Details: nonce too low. The re-add of the propose failed because its nonce was already consumed on-chain — i.e. mineEmptyBlock's own mine step had just included it.
  • L1 transaction 0x08fa… with nonce 0 mined blockNumber 36 — the propose mined into the "empty" block.

This was reproduced deterministically at the anvil level: with the old implementation, dropping all txs, re-broadcasting (as the publisher monitor does), then hardhat_mine produces a block containing the pending tx.

The previous explanation in this PR — "the re-added propose lands in the block whose timestamp == txTimeoutAt, and the monitor checks mined before timed-out" — was incomplete. The boundary timestamp is incidental (a consequence of the 12s L1-block grid and where the loop happened to be); the real determinant is that the tx is present in the pool when mineEmptyBlock mines. The monitor's mined-before-timeout ordering is a red herring here: once a tx is mined, it is mined regardless of check order.

Fix

Reimplement mineEmptyBlock so it never touches the mempool, eliminating the race for all callers rather than working around it in one test:

  • Temporarily lower the block gas limit below any transaction's 21000-gas intrinsic minimum, so hardhat_mine cannot include any pending tx; the pending tx stays in the pool untouched.
  • Restore the gas limit, then anvil_reorg the just-mined blocks into empty blocks at the restored gas limit (same height and timestamps). The reorg is required because anvil applies a new gas limit only to future blocks — without it the just-mined blocks would keep the tiny limit and a later eth_call against latest (whose gas is capped by the block gas limit) would revert with "intrinsic gas too high".

This is strictly more robust than dropping the pending tx (the approach in #24401 / this PR's prior commit): it removes the drop/re-add window entirely instead of narrowing it, and it preserves pending txs (the existing eth_cheat_codes.test.ts contract). Consequently the test-local mineBlockWithoutPendingTxs workaround is removed and the three timeout tests call mineEmptyBlock() again — the change to the test file is a net revert.

Verification

  • Deterministic RED: with the old implementation, drop-all -> re-broadcast -> hardhat_mine mines the pending tx into the "empty" block.
  • Unit eth_cheat_codes.test.ts › mineEmptyBlock › mines an empty block while preserving pending transactions: passes (block advances, block empty, pending tx preserved and mined later).
  • All three timeouts integration tests pass; the cancel test passes 3/3 repeats.
  • Full ethereum/src/l1_tx_utils/l1_tx_utils.test.ts (51 tests, 22 mineEmptyBlock calls incl. multi-block) passes — the L1TxUtils monitor suite that interacts most with mineEmptyBlock.

Relation to #24401

#24401 independently landed the same drop-pending-txs-without-re-add workaround in these tests. This PR supersedes that approach by fixing the shared mineEmptyBlock helper, so no per-test workaround is needed.

The L1Publisher integration timeout tests advance L1 blocks with
ethCheatCodes.mineEmptyBlock() while a propose tx is pending (automine off).
mineEmptyBlock drops the pending txs, mines, then re-adds them, and the mined
block can land at exactly txTimeoutAt with the re-added propose included. The
publisher's L1 tx monitor checks "mined" before "timed out", so a propose mined
at the timeout boundary is reported as a successful proposal instead of being
cancelled, flaking the timeout assertions.

Replace mineEmptyBlock with a helper that drops the pending txs and mines
without re-adding them, so the in-flight propose stays absent from the chain and
the monitor cancels it deterministically once its timeout elapses.
@spalladino spalladino added the S-do-not-merge Status: Do not merge this PR label Jun 30, 2026
mineEmptyBlock dropped all pending txs, mined a block, then re-added them.
That drop->mine->re-add sequence is not atomic against a concurrent
publisher whose L1 tx monitor re-broadcasts its in-flight tx: the tx can
reappear in the pool during the window and be swept into the supposedly
empty block. In the timeout publisher tests this mined a propose the test
required to stay pending, flaking the cancel assertion.

Mine empty blocks by temporarily lowering the block gas limit below any
tx's intrinsic cost (so nothing fits) and then reorging those blocks into
empty blocks at the restored gas limit. This never manipulates the
mempool, so pending txs are preserved and the race is gone for all
callers. Revert the test-local mineBlockWithoutPendingTxs workaround.
@spalladino spalladino changed the title test(sequencer): keep timeout publisher txs unmined fix(ethereum): mine empty L1 blocks without touching the mempool Jun 30, 2026
@spalladino spalladino removed the S-do-not-merge Status: Do not merge this PR label Jul 1, 2026
@PhilWindle
PhilWindle merged commit db6cf24 into merge-train/spartan-v5 Jul 1, 2026
16 checks passed
@PhilWindle
PhilWindle deleted the spl/fix-l1-publisher-cancel-flake branch July 1, 2026 11:21
rangozd pushed a commit to rangozd/aztec-packages that referenced this pull request Aug 5, 2026
BEGIN_COMMIT_OVERRIDE
refactor(e2e): relocate no-node straggler tests (AztecProtocol#24344)
test(e2e): pin gas_estimation public-payment txs to one block to deflake
fee comparison (AztecProtocol#24382)
feat(archiver): event-trigger L2BlockStream sync from archiver updates
(AztecProtocol#24317)
test(e2e): speed up individual e2e tests (AztecProtocol#24345)
chore(e2e): warm blob KZGs in parallel during setup (AztecProtocol#24383)
chore: add perf as a valid PR title prefix (AztecProtocol#24412)
test(e2e): adopt shared wait helpers (AztecProtocol#24404)
test(e2e): run prover client.test.ts in CI (AztecProtocol#24399)
fix(ethereum): mine empty L1 blocks without touching the mempool
(AztecProtocol#24414)
test(e2e): deflake empty block proving test (AztecProtocol#24411)
test(e2e): allocate HA node p2p ports above the ephemeral range to
deflake e2e_ha_full (AztecProtocol#24418)
fix(sequencer): use evmMine in automine auto-settle to avoid dropping
test L1 txs (AztecProtocol#24421)
test(e2e): instrument common spans for wall-clock tracking (AztecProtocol#24407)
test(e2e): remove redundant reqresp_no_handshake e2e test (AztecProtocol#24424)
END_COMMIT_OVERRIDE
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants