Skip to content

refactor(blockstream): add tips-only mode to avoid pulling blocks when not needed - #24034

Merged
PhilWindle merged 9 commits into
spl/block-stream-pass-atomicityfrom
spl/tips-only-block-stream
Jun 16, 2026
Merged

refactor(blockstream): add tips-only mode to avoid pulling blocks when not needed#24034
PhilWindle merged 9 commits into
spl/block-stream-pass-atomicityfrom
spl/tips-only-block-stream

Conversation

@spalladino

@spalladino spalladino commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Motivation

Two of the four L2BlockStream consumers download every block payload only to discard it — the stream
needed the blocks solely so its reorg walk-back had local hashes to compare:

  • prover-node: its handler ignores blocks-added entirely. Every block from the partially-proven epoch
    onward was downloaded at startup (via startingBlock), and every block at the tip thereafter, all unused
    — its checkpoint catch-up is cursor-driven since refactor(stdlib)!: thin chain-checkpointed event, collapse sync #24007.
  • PXE (default syncChainTip: 'proposed'): used only the last header of each blocks-added batch to
    update its anchor, downloading the whole unfinalized range per sync to keep one header.

This PR introduces a tips-only stream mode so neither consumer downloads blocks it never uses.

Stacked on #24042, which hardens the block-mode walk-back and adds the pass-completion gate this PR's
chain-proposed emission relies on. The walk over sparse history additionally relies on #24042's
missing-local-hash rule (correctness) and finalized-tip walk floor (bounds the walk to the unproven
window).

Approach

  • New thin chain-proposed event, emitted in both modes after the blocks, symmetric with the other
    tier events. A pass emits chain-pruned → blocks-added* → chain-proposed → chain-checkpointed → chain-proven → chain-finalized (tiers highest-to-lowest so the local tip-ordering invariant holds
    mid-pass). It fires iff the proposed tip changed during the pass, compared against the pre-pass local tip
    rather than the post-prune re-read — otherwise it would never fire on prune-then-download passes.
    Emission is gated on the pass-completion check from fix(stdlib): fix race conditions in L2BlockStream #24042, so a chain-proposed for an undelivered tip
    cannot corrupt block-mode tips cursors.
  • tipsOnly mode: skips the download loop entirely (zero getBlocks calls, blocks-added never
    emitted). Reorg detection reuses the existing walk-back over sparse history: the tips store records one
    (number, hash) entry per tip-moving poll from chain-proposed, and the tier handlers' recorded tips
    densify it. Steady-state cost per poll drops to one getL2Tips plus at most one getBlockData.
  • Witness registration keeps sparse-history prune targets honest. The walk-back rolls back to the
    nearest recorded hash at or below the true divergence, so a reorg landing inside a witness gap would
    produce an over-deep prune event (reporting more pruned blocks than were rolled back). A recorded hash
    is the only thing tying local per-height state to a specific fork, so consumers that materialize
    per-height state now record witnesses for those heights via the new
    L2TipsStoreBase.recordBlockHashes(blocks): hash-index writes only, no tip cursor moves, and a witness
    on an orphaned fork is compared rather than trusted (it fails the hash comparison and the walk continues),
    so the API cannot cause under-deep prunes. Where no witness covers a height, over-deep prunes remain the
    accepted fallback (idempotent checkpoint reprocessing; note re-sync).
  • prover-node: consumes the stream in tipsOnly mode. The startingBlock resolution is deleted
    (computeStartupStateresolveLastFullyProvenEpoch), taking the prover-node out of the startingBlock
    hazard class entirely (p2p is now its sole user). Cursor seeding, at-least-once retry ordering, and the
    expiry ticker are unchanged. It records a witness for every block of each checkpoint it registers.
  • prover-node prune handling re-keyed off the prune target. The chain-pruned handler used
    event.checkpointed.checkpoint — the source's current checkpointed tip, which sits above the prune
    target whenever the source re-checkpointed past the divergence before the prune was observed (a restart
    spanning a prune+rebuild, or a same-number checkpoint replacement seen atomically). Number-keyed marking
    then left orphaned provers canonical and under-clamped the catch-up cursor, permanently skipping a rebuilt
    checkpoint with the same number. It now keys off event.block (the highest surviving block):
    markPrunedAboveBlock marks every prover whose checkpoint holds a block above the target (no source
    fetch; catches partially-orphaned straddling checkpoints), and the cursor clamps to one below the
    target's checkpoint — over-clamping by one only re-registers a checkpoint (registration is at-least-once,
    A-1041), whereas under-clamping is the dangerous direction.
  • PXE: consumes the stream in tipsOnly mode; anchors on the proposed tip by fetching the header by
    hash (reorg-safe — it can no longer anchor on a header from a stale batch). The handler now updates the
    tips store after consumer-side work, so a thrown anchor update or prune rollback is retried on the next
    sync instead of silently lost; a missing proposed header throws (node inconsistency) to keep the cursor
    retryable. It records a witness at the anchor height whenever the anchor advances — the rollback boundary
    for the note and private-event stores — keeping prune events at the true divergence.

API changes

Internal only (no RPC surface changes):

  • L2BlockStreamEvent gains { type: 'chain-proposed'; block: L2BlockId }; all event switches updated.
  • L2BlockStream gains the tipsOnly option; the constructor throws if it is combined with
    startingBlock, batchSize, or skipFinalized.
  • L2TipsStoreBase (and the L2TipsStore type) gain recordBlockHashes(blocks: L2BlockId[]).
  • CheckpointStore.markPrunedAfter(checkpointNumber)markPrunedAboveBlock(blockNumber) (prover-node).
  • TraceableL2BlockStream (telemetry-client) deleted — zero production users and a stale opts type.

Changes

  • stdlib: chain-proposed event + emission (gated on the fix(stdlib): fix race conditions in L2BlockStream #24042 completion check); tipsOnly mode +
    constructor validation; handleChainProposed in the tips store base; recordBlockHashes witness API.
  • stdlib (tests): tips-only suite (zero-getBlocks, sparse-history prunes, constructor validation),
    pre-pass baseline / block-mode chain-proposed, event ordering, sparse anchors vs finalized hash
    deletion, witness-tightened prune targets (with vs without a recorded witness). (The walk-back,
    prune-payload-freshness, re-read, and pass-atomicity regressions live in fix(stdlib): fix race conditions in L2BlockStream #24042.)
  • prover-node: tipsOnly; startingBlock resolution deleted; handler arms updated; prune handling
    re-keyed off event.block (block-range marking + cursor clamp to cpAtTarget - 1, with prune-to-genesis
    and missing-target fallbacks); witness recording at checkpoint registration; tests adjusted, including a
    same-number checkpoint-replacement regression the old checkpoint-number keying fails.
  • pxe: tipsOnly; by-hash proposed anchoring with throw-on-miss; handler ordering flipped to
    store-last (at-least-once); anchor-height witness recording; tests including a retry-after-throw
    red/green regression.
  • p2p / world-state: no-op chain-proposed arms (world-state also gains an exhaustiveness default);
    block-mode behavior untouched.
  • kv-store (tests): shared tips-store suite covers the new event and the witness API (records hashes,
    moves no cursors).

Fixes A-1209

@spalladino
spalladino force-pushed the spl/a978-remove-proposed-checkpoint-tip branch from 292945f to a0f4da4 Compare June 11, 2026 23:17
@spalladino
spalladino force-pushed the spl/tips-only-block-stream branch from 44ea60c to 949ebe3 Compare June 11, 2026 23:17
@spalladino
spalladino force-pushed the spl/tips-only-block-stream branch from 949ebe3 to 732522f Compare June 12, 2026 02:16
@spalladino
spalladino changed the base branch from spl/a978-remove-proposed-checkpoint-tip to spl/block-stream-pass-atomicity June 12, 2026 02:16
@spalladino
spalladino force-pushed the spl/tips-only-block-stream branch from 732522f to f3fe2bf Compare June 12, 2026 02:20
@spalladino
spalladino force-pushed the spl/block-stream-pass-atomicity branch 2 times, most recently from dd04dc1 to adc83f6 Compare June 12, 2026 13:10
Add a thin `chain-proposed` event to the L2BlockStream union, reporting the
new proposed tip once per pass against the pre-pass local baseline, and a
`tipsOnly` mode that skips block downloads entirely (no getBlocks calls,
no blocks-added) for consumers that only track tips. The constructor rejects
tipsOnly combined with startingBlock/batchSize/skipFinalized.

chain-proposed is emitted in both modes after the blocks, gated on the
download-plan completion check so an undelivered tip cannot advance block-mode
tips cursors. tipsOnly reuses the existing reorg walk-back over the sparse
history recorded by the tier handlers and handleChainProposed.

Add handleChainProposed to L2TipsStoreBase (records the proposed tip into the
walk-back hash index). Migrate consumers: no-op arms in p2p and world-state
(plus a never-default in world-state), and delete the unused
TraceableL2BlockStream wrapper.

The block-mode pass-atomicity, post-divergence reorg-payload re-read,
missing-local-hash walk-back rule, and proposed-only cache seeding this builds
on live in the base PR.
Pass tipsOnly: true to the L2BlockStream and stop downloading block
payloads the prover-node never uses. The handler drops the blocks-added
arm (never emitted in tips-only mode), treats chain-proposed as a no-op
(the tips store records it as walk-back history), and gains a never
default. computeStartupState is renamed to resolveLastFullyProvenEpoch
and no longer resolves a starting block — it only derives the last
fully-proven epoch that seeds the catch-up cursor and lastExpiredEpoch.
…wnloads

Run the L2BlockStream in tips-only mode so PXE stops downloading the
unfinalized block range every sync just to keep one header. The proposed
anchor now updates from the thin chain-proposed event by fetching the tip
header by hash (warn-and-skip on a transient reorg miss), mirroring the
existing chain-checkpointed arm. blocks-added becomes a no-op arm (never
emitted in tips-only mode) and a never default guards the switch.

Flip the handler ordering to advance the tips store cursor after the
anchor update / prune rollback rather than before, matching the
prover-node's handle-first/tips-last ordering: a thrown anchor write
leaves the cursor unadvanced so the stream re-emits the event next pass
(at-least-once). The SerialQueue makes the reorder safe.
The chain-proposed arm warn-and-skipped when the by-hash getBlockData
returned undefined, but the tips-store update (now at the end of the
handler) still advanced the proposed cursor. On a quiet chain or a node
inconsistency the tip would then never change, chain-proposed would never
re-emit, and the anchor would stay stale indefinitely.

Throw on the missing header instead (same node-inconsistency reasoning as
the existing chain-pruned throw). The throw propagates before the
tips-store update, so the cursor stays put and the next sync() re-emits
chain-proposed — at-least-once retry. The other tier arms keep their
pre-existing warn-and-skip semantics.
In tips-only mode the L2BlockStream never delivers blocks, so a tips store's hash
history is sparse: one anchor per tip-moving poll. The reorg walk-back rolls back to
the nearest recorded hash at or below the true divergence, so a reorg below the nearest
anchor produces a prune event that is over-deep by the witness gap.

Add L2TipsStoreBase.recordBlockHashes(blocks), which writes (number -> hash) into the
walk-back hash index without moving any tip cursor, so consumers that materialize
per-height state can pin those heights to their fork and keep prune events at the true
divergence. Expose it on the L2TipsStore type. Wire it into PXE's tips-only block
synchronizer: record a witness at the anchor height whenever the anchor advances, so the
note and private-event stores' rollback boundary is tied to a specific fork.
The prover-node keyed prune handling off event.checkpointed — the source's CURRENT
checkpointed tip — which sits above the prune target whenever the source re-checkpointed
past the divergence before the prune was observed (a restart spanning a prune+rebuild, or
a same-number checkpoint replacement seen atomically). Number-keyed marking then left
partially or fully orphaned checkpoints canonical and under-clamped the catch-up cursor,
so a rebuilt checkpoint with the same number was permanently skipped and its epoch never
proven by this node.

Key both off event.block (the prune target, the highest surviving block, which by
construction survives on the source):

- Marking: replace CheckpointStore.markPrunedAfter(checkpointNumber) with
  markPrunedAboveBlock(blockNumber), which marks every prover whose last block is above
  the target. This needs no source fetch and correctly marks a checkpoint whose range
  straddles the target (partially orphaned).
- Cursor: resolve the checkpoint at the target via getBlockData(...).checkpointNumber and
  clamp lastProcessedCheckpoint to min(current, cpAtTarget - 1). Over-clamping by one only
  re-registers a checkpoint (registration is at-least-once, A-1041); under-clamping is the
  dangerous direction. Prune-to-genesis clamps to 0 and marks every prover; a missing
  target block (source race) clamps conservatively to 0.

Also record a walk-back witness for every block in a checkpoint as it is registered, so
the prover-node's sparse tips-only history pins those heights and prune events land at the
true divergence rather than over-deep by the witness gap.
@spalladino
spalladino force-pushed the spl/tips-only-block-stream branch from f3fe2bf to aa03add Compare June 12, 2026 18:52
- Remove unused config param from PXE createBlockStream
- Drop redundant anchor-height witness recording in PXE: the tips store
  already records the same (number, hash) when handling the event
- Inline the prover-node cursor clamp into handlePruneEvent, resolving the
  floor before marking provers, and throw on missing prune-target data so
  the prune is retried next pass instead of over-clamping to genesis
- Condense block-stream and prover-node comments
@PhilWindle
PhilWindle merged commit 261b529 into spl/block-stream-pass-atomicity Jun 16, 2026
12 checks passed
@PhilWindle
PhilWindle deleted the spl/tips-only-block-stream branch June 16, 2026 11:03
PhilWindle pushed a commit that referenced this pull request Jun 16, 2026
## Human-written summary

Fixes three potential issues in the block-stream:

1. We snapshot the source tips, then download blocks until the proposed
tip and pass them to the consumer, and then emit the checkpointed and
proven tips. If there's a prune while we're downloading blocks, we may
not be able to fetch blobs until the proposed tip. And worse, we may be
then emitting checkpointed and proven events that reference blocks _we
could never pass onto the consumer_. This PR fixes it by aborting all
events if we fail to collect all the blocks we expected, and waiting
until the next iteration to heal.

2. When we detect a prune, we walk back to find the common ancestor. But
that walk back may take time. During that walk back, the checkpointed
and proven tips may have shifted, so we re-fetch them to advertise the
proper latest tips.

3. When walking back to find the common ancestor during a prune, we
compare block hashes of source vs consumer. If the consumer didn't see a
block for whatever reason, but that block was also removed from the
source, then both block hashes compared are undefined, and we stop the
walk-back, instead of continuing until finding an actual existing common
ancestor.

4. When walking back, if the consumer for some reason did not have the
block hashes available, a walk back to genesis would be triggered. We
now cap the walk back to the local finalized chain tip.

## Motivation

This PR hardens the block-mode `L2BlockStream` against three
pre-existing hazards. It is split out of
#24034 (the `chain-proposed`/`tips-only` feature work) so the
correctness fixes can be reviewed on their
own — #24034's `chain-proposed` emission depends on the pass-completion
gate introduced here, so this PR
sits below it in the stack.

The three hazards, all latent in block mode today:

- **Tier cursors advance from an incomplete download pass.** The
download loop broke on an empty
`getBlocks` result and end-of-pass reconciliation still ran, so
`chain-checkpointed`/`chain-proven`/
`chain-finalized` could reference blocks the consumer never received via
`blocks-added` — exactly the
  transient-inconsistency class #24007 set out to remove.
- **`chain-pruned` carries stale clamp tips.** The prune event's
`checkpointed`/`proven` payload came from
the pass snapshot, which on a stale pass describes the chain *before*
the prune. p2p feeds
`event.checkpointed.checkpoint` into `isEpochPrune`, whose contract is
"the checkpoint id **after** the
prune", and the result drives the irreversible `deleteAllTxs` mempool
wipe — a misclassification risk.
- **The walk-back stops above the true divergence.**
`areBlockHashesEqualAt` ended with
`localHash === sourceHash`, so a height missing on both sides
(`undefined === undefined`) counted as
agreement. With a store that has gaps in its hash index (p2p keeps none
below its `startingBlock`), a gap
meeting a source-pruned height stopped the walk above the divergence —
an under-deep prune that left
  old-fork state in place with no later re-detection.

## Approach

- **Pass atomicity.** Extract the block download loop into
`downloadBlocks()`, which returns whether the
download plan completed; tier reconciliation is skipped for the pass
when it did not. Two staleness
terminations: an empty `getBlocks` below the target, and a delivered
proposed block whose hash differs
from the snapshot's (a mid-pass same-height fork swap). Intentional
skips — the loop never running
because `startingBlock`/caught-up put the cursor past the proposed tip —
are trivially complete and still
  reconcile, preserving the A-1061 fix.
- **Fresh reorg payload.** On walk-back divergence, re-read `getL2Tips`
and drive the `chain-pruned` clamp
tips, the #13471 prune-target clamp, the download target, and tier
reconciliation from the re-read rather
than the pre-prune snapshot. The pass does not abort if the re-read's
proposed tip moved on (a busy chain
  advances every pass and prune emission would otherwise starve).
- **Walk-back missing-local-hash rule.** A missing **local** hash now
compares unequal regardless of the
source side, so the walk continues to a recorded matching height or
genesis (block 0 always resolves via
the store's `initialBlockHash`). Prunes become over-deep-only, which
consumers tolerate. The
  `skipFinalized` AbortError path is unchanged.
- The walk-back hash cache is seeded with the **proposed tip only**.
Seeding the tier tips would let a
snapshot entry that went stale (a reorg between the snapshot and the
walk) fake agreement at a reorged
height and stop the walk early; the re-read appends only its proposed
tip after the walk is over.

**Behavior change worth calling out:** a *permanently* missing source
range previously advanced tier
cursors past undelivered blocks (masking the source bug). With pass
atomicity it now wedges loudly — a
warn each pass, p2p stays SYNCHING — instead of silently diverging. Loud
failure is the better behavior.

## Changes

- `stdlib/src/block/l2_block_stream/l2_block_stream.ts`:
`downloadBlocks()` extraction + completion gate;
post-divergence `getL2Tips` re-read driving the prune
payload/target/download/reconciliation; missing
  local-hash → unequal; proposed-only cache seeding.
- `stdlib/src/block/l2_block_stream/l2_block_stream.test.ts`:
pass-atomicity terminations (empty
`getBlocks`, delivered-hash mismatch, A-1061 fast-forward still
reconciles), prune-payload freshness,
re-read download target + reconciliation, and the walk-back
both-undefined / stale tier-seed regressions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants