fix: sync proposed checkpoint on HA peers (A-1165) - #23940
Merged
PhilWindle merged 4 commits intoJun 8, 2026
Conversation
Collaborator
Flakey Tests🤖 says: This CI run detected 1 tests that failed, but were tolerated due to a .test_patterns.yml entry. |
…roposals In an HA setup, peer nodes share validator signing keys. When the proposer broadcasts a checkpoint proposal, its HA peer receives it over gossip and classifies it as "own" (it owns the signing key), so the all-nodes checkpoint handler returned early without recording the proposed-checkpoint metadata. The peer never built the checkpoint, so it lacked the metadata needed to build the next slot on top of the proposed checkpoint: it pruned the proposed block as an orphan, rebuilt the same checkpoint, and failed to take over. For own proposals, the all-nodes handler now inspects the stored proposed checkpoint for the slot: if it matches the proposal's archive it is an idempotent no-op (true local proposer), if it conflicts it warns without overwriting, and if it is missing it waits for the last block to sync (same deadline as the foreign-validation path) and hydrates the metadata from local block data. The block-sync wait is extracted into a shared helper reused by the foreign-validation path. Adds an e2e reproduction (epochs_ha_checkpoint_handoff) and unit coverage for the own-proposal matching, missing, conflicting, and block-sync-wait cases.
For own checkpoint proposals, only fast-path the true local proposer — detected as a matching proposed checkpoint already stored for the slot — and fall through to the normal validate-and-persist path for everything else, including the HA peer that received the gossiped own-key proposal but never built the checkpoint. This drops the bespoke wait/hydrate branch and the conflicting-archive special-case, removes the waitForLastBlockData helper, and reverts validateCheckpointProposal to its original inline block-sync wait. Strengthen the e2e: select whichever HA pair owns two consecutive proposal slots (not just the first pair), and on the secondary path assert that both the S1 and S2 checkpoints post to L1, with S2's covered block carrying the peer's coinbase to prove the peer produced it.
spalladino
force-pushed
the
spl/fix-ha-checkpoint-proposals
branch
from
June 8, 2026 20:15
66744d7 to
951b98e
Compare
PhilWindle
approved these changes
Jun 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Validators were skipping processing of
CheckpointProposals coming from their own keys. In an HA setup, this means the checkpoint proposal produced by another node in their same setup was never synced, so the archiver ended up pruning blocks without corresponding checkpoint proposal.Issue
In an HA deployment, peer nodes share validator signing keys for redundancy. When the active proposer broadcasts a checkpoint proposal, its HA peer receives the proposal over gossip and classifies it as "own" — it owns the signing key — so the all-nodes checkpoint handler returned early without recording the proposed-checkpoint metadata in its archiver.
That early return was written on the assumption that an own checkpoint proposal can only originate from this node's own sequencer, which already pushed the proposed checkpoint locally before broadcasting. That assumption breaks for HA peers: the peer never built the checkpoint, so it lacked the proposed-checkpoint metadata needed to build the next slot on top of the still-proposed checkpoint. When that peer had to take over for the following slot, it pruned the proposed block as an orphan, rebuilt the same checkpoint, and clashed instead of producing the next checkpoint.
Approach
For own checkpoint proposals, the all-nodes handler now inspects the proposed checkpoint already stored for that slot:
The block-sync wait (
syncImmediate+ deadline-bounded retry) is extracted into a shared helper so the own/HA hydration path has the same reliability as foreign validation, rather than a single no-wait fetch that could silently miss a not-yet-synced block.Checkpoint attestation behavior is unchanged: a validator still ignores proposals from its own keys and does not produce duplicate HA attestations. Block proposal handling is also unchanged.
Changes
isOwnProposalbranch of the all-nodes checkpoint proposal handler to hydrate missing proposed-checkpoint metadata for HA peers while keeping the local-proposer fast path idempotent. Widen the handler's archiver dependency to includegetProposedCheckpointData. Extract a sharedwaitForLastBlockDatahelper reused by the foreign-validation path;setProposedCheckpointFromValidationnow takes the already-synced block.e2e_epochs/epochs_ha_checkpoint_handoffreproducing the handoff: it routes two consecutive slots to one HA pair, has the builder propose the first slot, and asserts the HA peer records the proposed checkpoint and produces the next slot's checkpoint. The primary assertion (peer records the proposed-checkpoint metadata) is timing-independent.Fixes A-1165