Skip to content

test: deflake e2e_offchain_payment reorg reprocessing race - #24309

Merged
spalladino merged 1 commit into
merge-train/spartan-v5from
spl/deflake-offchain-payment-reorg
Jun 25, 2026
Merged

test: deflake e2e_offchain_payment reorg reprocessing race#24309
spalladino merged 1 commit into
merge-train/spartan-v5from
spl/deflake-offchain-payment-reorg

Conversation

@spalladino

Copy link
Copy Markdown
Contributor

Problem

e2e_offchain_payment.test.ts › "reprocesses an offchain-delivered payment after an L1 reorg" flakes (CI run 7d8ad07c73ce84e2) with:

expect(received).toBe(expected) // Object.is equality
  Expected: 100n
  Received: 60n
  > 191 |     expect(aliceAfterRollback).toBe(mintAmount);

60n == 100 - 40: the reverted transfer's effect was still visible to the PXE when the post-reorg balance was read, instead of the rolled-back full mint of 100n.

Root cause

A race between the AutomineSequencer's automatic re-mine of the restored transfer tx and the test's post-reorg assertions:

  • forceReorgrevertToCheckpoint(6) prunes block 7 and restores the un-mined transfer to the pending pool (p2pClient.sync() in runRevert, automine_sequencer.ts:629), then returns. It does not itself re-mine.
  • The sequencer runs a ~50ms mempool poller (automine_sequencer.ts:175, default pollIntervalMs 50ms) whose buildIfPending autonomously re-mines block 7 from the restored pool.
  • The PXE has autoSync: true, so get_balance(...).simulate() re-syncs before every read. Whether Alice's read at line 191 returns 100n (PXE at pruned tip, block 6) or 60n (PXE re-synced to the re-mined block 7) depends purely on this timing.

The failed log shows Updated pxe last block to 7 landing between Bob's read (passes) and Alice's read (fails); the passed log keeps the PXE pinned at block 6 across both reads. Both runs show Building automine checkpoint {checkpointNumber:7, txCount:1}, confirming the poller drives the re-mine.

Fix (test only)

yarn-project/end-to-end/src/e2e_offchain_payment.test.ts:

  • forceReorg: automine.pause() before revertToCheckpoint. pause() gates only the mempool poller; explicit ops (revertToCheckpoint, buildEmptyBlock) still run. Pausing before the transfer is restored to the pool means the poller can never enqueue an auto re-mine, so the post-reorg assertions observe a stable pruned state.
  • Stay paused through the explicit forceEmptyBlock(). buildEmptyBlock (allowEmpty:true) permits an empty block but still drains the pending pool, so it deterministically re-mines the restored transfer with no poller racing it. The existing retryUntil still waits for the PXE re-sync, so the reprocessing behavior under test is preserved.
  • Add an afterEach that calls getAutomineSequencer()?.resume() (idempotent), so a pause is never left dangling.

This removes the nondeterministic auto re-mine that raced the assertions rather than relaxing the aliceAfterRollback == 100 assertion — which is behavioral (it verifies the reorg rolled Alice's note state back) and is kept. Not a skip, not a .test_patterns.yml entry.

Verification

  • yarn build: exit 0, no type errors.
  • Local e2e red/green not run: the race is nondeterministic (a single local pass would not prove the fix) and the test is infra-heavy. The fix is grounded in the log timeline (PXE re-sync to block 7 lands between Bob's and Alice's reads only in the failed run) and the source mechanism (poller-driven re-mine gated by paused; buildEmptyBlock still drains pending), and was cross-checked to confirm it does not break the later retryUntil.

The 'reprocesses an offchain-delivered payment after an L1 reorg' test asserts
Alice's balance is the rolled-back mint (100) after forceReorg, but reads 60
intermittently. revertToCheckpoint prunes the block and restores the un-mined
transfer to the pending pool; the AutomineSequencer's ~50ms mempool poller then
re-mines it autonomously, and because the PXE auto-syncs on every simulate, the
post-reorg balance read races whether the PXE has caught up to the re-mined
block.

Pause the AutomineSequencer before revertToCheckpoint so its poller cannot win
the re-mine race, stay paused through the explicit forceEmptyBlock re-mine
(which still drains the pending pool), and resume in afterEach. Closes the race
without relaxing the behavioral aliceAfterRollback == 100 assertion.
@spalladino
spalladino enabled auto-merge (squash) June 25, 2026 18:24
@AztecBot

Copy link
Copy Markdown
Collaborator

Flakey Tests

🤖 says: This CI run detected 2 tests that failed, but were tolerated due to a .test_patterns.yml entry.

\033FLAKED\033 (8;;http://ci.aztec-labs.com/5333ebc71b758d48�5333ebc71b758d488;;�): yarn-project/kv-store/scripts/run_test.sh src/bench/sqlite-opfs-encrypted/map_bench.test.ts (1s) (code: 0)
\033FLAKED\033 (8;;http://ci.aztec-labs.com/379edb201d919f0f�379edb201d919f0f8;;�): yarn-project/kv-store/scripts/run_test.sh src/sqlite-opfs/internal/ordered-binary-browser.test.ts (2s) (code: 0)

@spalladino
spalladino merged commit f36840b into merge-train/spartan-v5 Jun 25, 2026
12 checks passed
@spalladino
spalladino deleted the spl/deflake-offchain-payment-reorg branch June 25, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants