Skip to content

fix: sync proposed checkpoint on HA peers (A-1165) - #23940

Merged
PhilWindle merged 4 commits into
merge-train/spartan-v5from
spl/fix-ha-checkpoint-proposals
Jun 8, 2026
Merged

fix: sync proposed checkpoint on HA peers (A-1165)#23940
PhilWindle merged 4 commits into
merge-train/spartan-v5from
spl/fix-ha-checkpoint-proposals

Conversation

@spalladino

@spalladino spalladino commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Validators were skipping processing of CheckpointProposals coming from their own keys. In an HA setup, this means the checkpoint proposal produced by another node in their same setup was never synced, so the archiver ended up pruning blocks without corresponding checkpoint proposal.

Issue

In an HA deployment, peer nodes share validator signing keys for redundancy. When the active proposer broadcasts a checkpoint proposal, its HA peer receives the proposal over gossip and classifies it as "own" — it owns the signing key — so the all-nodes checkpoint handler returned early without recording the proposed-checkpoint metadata in its archiver.

That early return was written on the assumption that an own checkpoint proposal can only originate from this node's own sequencer, which already pushed the proposed checkpoint locally before broadcasting. That assumption breaks for HA peers: the peer never built the checkpoint, so it lacked the proposed-checkpoint metadata needed to build the next slot on top of the still-proposed checkpoint. When that peer had to take over for the following slot, it pruned the proposed block as an orphan, rebuilt the same checkpoint, and clashed instead of producing the next checkpoint.

Approach

For own checkpoint proposals, the all-nodes handler now inspects the proposed checkpoint already stored for that slot:

  • Matching archive — idempotent no-op. This is the true local proposer (or an already-hydrated peer); nothing to validate or write.
  • Conflicting archive — log a structured warning and do not overwrite. Duplicate/equivocation handling remains the responsibility of the existing detection paths.
  • Missing — the HA peer that received the proposal only via gossip. Wait for the last block to sync (using the same publish-deadline-bounded wait as the foreign-validation path) and hydrate the proposed-checkpoint metadata from local block data.

The block-sync wait (syncImmediate + deadline-bounded retry) is extracted into a shared helper so the own/HA hydration path has the same reliability as foreign validation, rather than a single no-wait fetch that could silently miss a not-yet-synced block.

Checkpoint attestation behavior is unchanged: a validator still ignores proposals from its own keys and does not produce duplicate HA attestations. Block proposal handling is also unchanged.

Changes

  • validator-client: Rework the isOwnProposal branch of the all-nodes checkpoint proposal handler to hydrate missing proposed-checkpoint metadata for HA peers while keeping the local-proposer fast path idempotent. Widen the handler's archiver dependency to include getProposedCheckpointData. Extract a shared waitForLastBlockData helper reused by the foreign-validation path; setProposedCheckpointFromValidation now takes the already-synced block.
  • validator-client (tests): Replace the unit test that asserted own proposals never touch the archiver. Add coverage for own-proposal matching/missing/conflicting cases, the block-sync wait (missing-then-appears), and the never-syncs degraded case.
  • end-to-end (tests): Add e2e_epochs/epochs_ha_checkpoint_handoff reproducing the handoff: it routes two consecutive slots to one HA pair, has the builder propose the first slot, and asserts the HA peer records the proposed checkpoint and produces the next slot's checkpoint. The primary assertion (peer records the proposed-checkpoint metadata) is timing-independent.

Fixes A-1165

@spalladino spalladino changed the title fix: hydrate proposed checkpoint on HA peers for own-key checkpoint proposals fix: hydrate proposed checkpoint on HA peers for own-key checkpoint proposals (A-1165) Jun 8, 2026
@AztecBot

AztecBot commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Flakey Tests

🤖 says: This CI run detected 1 tests that failed, but were tolerated due to a .test_patterns.yml entry.

\033FLAKED\033 (8;;http://ci.aztec-labs.com/58e402d886210b2b�58e402d886210b2b8;;�): yarn-project/end-to-end/scripts/run_test.sh ha src/composed/ha/e2e_ha_full.test.ts (232s) (code: 0)

@spalladino spalladino changed the title fix: hydrate proposed checkpoint on HA peers for own-key checkpoint proposals (A-1165) fix: sync proposed checkpoint on HA peers for own-key checkpoint proposals (A-1165) Jun 8, 2026
@spalladino spalladino changed the title fix: sync proposed checkpoint on HA peers for own-key checkpoint proposals (A-1165) fix: sync proposed checkpoint on HA peers (A-1165) Jun 8, 2026
…roposals

In an HA setup, peer nodes share validator signing keys. When the proposer
broadcasts a checkpoint proposal, its HA peer receives it over gossip and
classifies it as "own" (it owns the signing key), so the all-nodes checkpoint
handler returned early without recording the proposed-checkpoint metadata. The
peer never built the checkpoint, so it lacked the metadata needed to build the
next slot on top of the proposed checkpoint: it pruned the proposed block as an
orphan, rebuilt the same checkpoint, and failed to take over.

For own proposals, the all-nodes handler now inspects the stored proposed
checkpoint for the slot: if it matches the proposal's archive it is an
idempotent no-op (true local proposer), if it conflicts it warns without
overwriting, and if it is missing it waits for the last block to sync (same
deadline as the foreign-validation path) and hydrates the metadata from local
block data. The block-sync wait is extracted into a shared helper reused by the
foreign-validation path.

Adds an e2e reproduction (epochs_ha_checkpoint_handoff) and unit coverage for
the own-proposal matching, missing, conflicting, and block-sync-wait cases.
For own checkpoint proposals, only fast-path the true local proposer — detected
as a matching proposed checkpoint already stored for the slot — and fall through
to the normal validate-and-persist path for everything else, including the HA
peer that received the gossiped own-key proposal but never built the checkpoint.
This drops the bespoke wait/hydrate branch and the conflicting-archive
special-case, removes the waitForLastBlockData helper, and reverts
validateCheckpointProposal to its original inline block-sync wait.

Strengthen the e2e: select whichever HA pair owns two consecutive proposal slots
(not just the first pair), and on the secondary path assert that both the S1 and
S2 checkpoints post to L1, with S2's covered block carrying the peer's coinbase
to prove the peer produced it.
@spalladino
spalladino force-pushed the spl/fix-ha-checkpoint-proposals branch from 66744d7 to 951b98e Compare June 8, 2026 20:15
@PhilWindle
PhilWindle merged commit f91b7a7 into merge-train/spartan-v5 Jun 8, 2026
12 checks passed
@PhilWindle
PhilWindle deleted the spl/fix-ha-checkpoint-proposals branch June 8, 2026 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants