Skip to content

test(e2e): allocate HA node p2p ports above the ephemeral range to deflake e2e_ha_full - #24418

Merged
PhilWindle merged 1 commit into
merge-train/spartan-v5from
spl/fix-ha-full-p2p-port-flake
Jul 1, 2026
Merged

test(e2e): allocate HA node p2p ports above the ephemeral range to deflake e2e_ha_full#24418
PhilWindle merged 1 commit into
merge-train/spartan-v5from
spl/fix-ha-full-p2p-port-flake

Conversation

@spalladino

Copy link
Copy Markdown
Contributor

Summary

Deflakes e2e_ha_full.parallel.test.ts, which FLAKED in CI (run 6d29cd3fcf374a68) reporting should not be affected by process.env.TZ changes as the failing test.

Root cause

  • The reported test name is a red herring. The failure happened in beforeAll while creating the second HA node (HA-1): CodeError: Transport (@libp2p/tcp) could not listen on any available address, code ERR_NO_VALID_ADDRESSES. jest attributed the beforeAll throw to the first test in the suite (the timezone test, a pure Postgres test that never ran) and skipped the other 7.
  • The suite assigned fixed p2p ports (config.p2pPort ?? 40400) + i + 1, i.e. 40400 (bootstrap), 40401-40405 (HA nodes). These sit inside the Linux default ephemeral port range (32768-60999, per /proc/sys/net/ipv4/ip_local_port_range).
  • The OS draws ephemeral ports from that range for the in-process prover node (created by setup() with p2pPort: 0) and for outbound TCP connections. When an ephemeral socket already held one of 40401-40405 at bind time, that HA node's libp2p TCP listen() failed with EADDRINUSE. libp2p runs the listen attempt under Promise.allSettled and, finding no fulfilled result, throws the aggregate ERR_NO_VALID_ADDRESSES without surfacing the underlying EADDRINUSE — which is why no EADDRINUSE string appears in the logs.
  • Confirmed against the passing retry (061f6753f56b00fd): identical fixed ports, but there HA-0..HA-4 bound 40401-40405 cleanly. The only divergence is HA-1 failing to bind 40402 in the failed run.

Fix

  • Allocate every node's p2p port via get-port from 61000-65535, above the ephemeral range, so neither in-process ephemeral sockets nor concurrent CI jobs can be holding a node's port when libp2p binds it. Applied to the bootstrap node (via setup() opts) and to each HA node.
  • Set p2pBroadcastPort alongside p2pPort: discv5 defaults the broadcast port by mutating the config object in place, and that mutated value would otherwise leak from the bootstrap config into the HA node configs (which are built by spreading config), making them advertise the wrong port in their ENR.

Verification

  • TypeScript build of @aztec/end-to-end passes; prettier clean.
  • A local repro of the flake is not feasible: it is a probabilistic ephemeral-port collision that cannot be deterministically forced, and the test requires the full docker-compose HA stack (Postgres + Web3Signer). The fix is grounded in both the logs (HA-1 failing to bind fixed port 40402 with ERR_NO_VALID_ADDRESSES) and source (fixed ports inside the ephemeral range; p2pPort: 0 prover node; discv5 binding UDP on the same port) — moving the ports above the ephemeral range removes the collision source entirely.

…flake e2e_ha_full

The HA full suite assigned fixed p2p ports `40400 + i + 1` (40400-40405) to its
bootstrap and HA nodes. Those ports sit inside the Linux default ephemeral range
(32768-60999), which the OS draws from for the in-process prover node (it listens
on p2pPort 0) and for outbound connections. When an ephemeral socket already held
one of those ports at bind time, the corresponding node's libp2p TCP listener
failed to bind and threw `ERR_NO_VALID_ADDRESSES` (libp2p's wrapper around the
swallowed EADDRINUSE), aborting `beforeAll`. jest then attributed the failure to
the first test in the suite ("should not be affected by process.env.TZ changes")
and skipped the rest, making the flake masquerade as a timezone bug.

Allocate every node's port via get-port from 61000-65535, above the ephemeral
range, so neither in-process ephemeral sockets nor concurrent CI jobs can hold a
node's port. Set p2pBroadcastPort alongside p2pPort because discv5 mutates the
config in place to default the broadcast port, and that value would otherwise leak
from the bootstrap config into the HA nodes' configs (built by spreading it).
@PhilWindle
PhilWindle merged commit d3e4633 into merge-train/spartan-v5 Jul 1, 2026
12 checks passed
@PhilWindle
PhilWindle deleted the spl/fix-ha-full-p2p-port-flake branch July 1, 2026 11:22
rangozd pushed a commit to rangozd/aztec-packages that referenced this pull request Aug 5, 2026
BEGIN_COMMIT_OVERRIDE
refactor(e2e): relocate no-node straggler tests (AztecProtocol#24344)
test(e2e): pin gas_estimation public-payment txs to one block to deflake
fee comparison (AztecProtocol#24382)
feat(archiver): event-trigger L2BlockStream sync from archiver updates
(AztecProtocol#24317)
test(e2e): speed up individual e2e tests (AztecProtocol#24345)
chore(e2e): warm blob KZGs in parallel during setup (AztecProtocol#24383)
chore: add perf as a valid PR title prefix (AztecProtocol#24412)
test(e2e): adopt shared wait helpers (AztecProtocol#24404)
test(e2e): run prover client.test.ts in CI (AztecProtocol#24399)
fix(ethereum): mine empty L1 blocks without touching the mempool
(AztecProtocol#24414)
test(e2e): deflake empty block proving test (AztecProtocol#24411)
test(e2e): allocate HA node p2p ports above the ephemeral range to
deflake e2e_ha_full (AztecProtocol#24418)
fix(sequencer): use evmMine in automine auto-settle to avoid dropping
test L1 txs (AztecProtocol#24421)
test(e2e): instrument common spans for wall-clock tracking (AztecProtocol#24407)
test(e2e): remove redundant reqresp_no_handshake e2e test (AztecProtocol#24424)
END_COMMIT_OVERRIDE
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants