Follower sync deadlock: failed multi-height fetch skips a height, consensus wedges permanently (v0.7.3)

Summary

A follower node (CL in --follow mode) can deadlock permanently when a batched sync fetch fails at the wrong moment. On a failed fetch, sync's sync_height is advanced past all heights in the failed range, but only the first height of the range is re-requested. If the range covered more than one height, the later height is never requested again. Consensus then waits forever at that height while sync fills all max_parallel_requests=5 slots with later blocks that can never be delivered, so no new requests are ever issued. The services stay active — the node just silently stops advancing.

Observed on v0.7.3 (built from source), Arc testnet, chain id 5042002. The node wedged at height 52893874 and sat stalled for ~18 hours until the CL was manually restarted (restart fully recovers it).

Environment

arc-node v0.7.3, built from the release tag via cargo install
CL: arc-node-consensus start ... --follow --follow.endpoint https://rpc.drpc.testnet.arc.network,... --execution-persistence-backpressure
EL: arc-node-execution node --chain arc-testnet --disable-discovery ... (IPC to CL)
Ubuntu 22.04.5, EL/CL on the same host

Timeline (2026-07-21, UTC)

The node had been sync-following normally. Certificate not found (-32004) fetch failures are chronic in our logs and are normally harmless because the failed height gets re-requested — until this sequence:

05:08:00.949949  sync{tip=52893872 sync_height=52893873}: Requesting sync from peer range=52893873..=52893873 peer=1Ak9...
05:08:00.957722  WARN Fetch failed request_id=39161 error="Failed to fetch certificates: JSON-RPC error: code -32004: Certificate not found"
05:08:00.957760  sync{tip=52893872 sync_height=52893874}: Re-requesting values from peer request_id=39161 except=1Ak9...
05:08:01.185749  sync{tip=52893872 sync_height=52893873}: Requesting sync from peer range=52893873..=52893874 peer=1Ak9...   <-- 2-height range
05:08:01.195753  WARN Fetch failed request_id=39162 error="... code -32004: Certificate not found"
05:08:01.195778  sync{tip=52893872 sync_height=52893875}: Re-requesting values from peer request_id=39162 except=1Ak9...     <-- sync_height jumped PAST 52893874
05:08:01.195788  sync{tip=52893872 sync_height=52893875}: Requesting sync from peer range=52893873..=52893873 peer=1AWR...   <-- only 52893873 re-requested
05:08:01.321114  ✅ Fetch completed, sending blocks to consensus request_id=39163 start_height=52893873 count=1
05:08:01.348720  consensus{height=52893873}: Decided round=0 height=52893873
05:08:01.362328  consensus{height=52893874}: Starting new height height=52893874
05:08:01.689196  sync{tip=52893873 sync_height=52893875}: Requesting sync from peer range=52893875..=52893875 ...            <-- 52893874 never requested again

The failed request 39162 covered 52893873..=52893874. Its failure bumped sync_height to 52893875, but the retry (39163) only covered 52893873..=52893873. Height 52893874 fell through the gap.

Sync then fetched 52893875–52893879 into its 5 parallel slots. Consensus can't decide 52893874 (no value, and as a follower with role=None it can't propose), those 5 pending deliveries never complete, and sync refuses to issue new requests:

05:08:04.364143  consensus{height=52893874 round=0}: Timeout elapsed step=Propose
05:08:06.364464  consensus{height=52893874 round=0}: Timeout elapsed step=Rebroadcast    (repeats forever)
...
22:52:21.033505  sync{tip_height=52893873 sync_height=52893880}: SYNC REQUIRED: Falling behind tip_height=52893873 sync_height=52893880 peer_height=52987354
22:52:21.033517  sync{...}: Maximum number of parallel requests reached, skipping request for values max_parallel_requests=5 pending_requests=5

Meanwhile the EL just waits (its view of the incident):

22:47:11  WARN Beacon client online, but no consensus updates received for a while. ... period=63549.767870051s

Expected behavior

Either (a) a failed range fetch re-requests the entire failed range, not just its first height, or (b) sync detects tip_height + 1 < min(pending heights) and re-requests the gap, or (c) pending requests time out and free their slots so the gap can eventually be re-fetched. Any of these would prevent the permanent wedge.

Possibly related

During the same session (but ~2h earlier, and recovered automatically via the serial fallback) the EL hit a panic in the parallel state root task on v0.7.3:

thread 'tokio-rt' (1593146) panicked at .../reth-e231042ee7db3fb7/d6324d6/crates/trie/sparse/src/parallel.rs:3049:36:
node at path Nibbles(0x71) does not exist
02:50:17.736866  WARN Failed to compute state root in parallel

Happy to provide fuller logs for either issue.

Confirmed on Arc Testnet (client-side observations) + fix recommendation + mitigation

We've been running a keeper service against Arc Testnet v0.7.3 for several weeks making frequent eth_call / eth_getLogs / waitForTransactionReceipt calls via rpc.drpc.testnet.arc.network. A few things that may be useful context:

On the -32004 trigger rate

-32004 Certificate not found is chronic on both the public RPC and the DRPC endpoint — we see it in our keeper logs roughly every 3–5 minutes during normal operation at ~10 req/s. The key observation is it's usually harmless because the retry succeeds on the next peer. But the timing window your trace shows — where a 2-height range is batched right at the moment the first height's cert becomes available but the second doesn't yet — narrows the bug to a short race window. Given how frequently -32004 fires, hitting that window within hours is plausible.

On the three proposed fixes

Of your three proposals, (a) re-request the entire failed range is the most precise fix — it eliminates the gap at the source without relying on downstream detection. (b) gap detection (tip_height + 1 < min(pending_heights)) is worth implementing as a safety net regardless, since it would catch any future source of a skipped height. (c) pending-request timeouts would also help free the 5-slot deadlock as a last resort but doesn't address the gap itself.

The combination of (a) + (b) seems like the right approach: (a) prevents the gap at the source, (b) heals it if any other code path produces one.

Mitigation until the fix ships

Restart fully recovers the node (confirmed). A simple watchdog that detects consensus height stagnation works in the meantime:

#!/usr/bin/env bash
# Restart CL if consensus height hasn't advanced in 5 minutes
PREV=$(curl -sf http://localhost:31000/consensus/height | jq '.height')
sleep 300
CURR=$(curl -sf http://localhost:31000/consensus/height | jq '.height')
if [ "$CURR" -le "$PREV" ]; then
  echo "Consensus wedged at height $CURR — restarting CL"
  systemctl restart arc-node-consensus
fi

Run as a cron job every 5 minutes and it catches the wedge before it becomes an 18-hour gap.

On the EL parallel trie panic (possibly related)

The node at path Nibbles(0x71) does not exist panic in parallel.rs:3049 is a separate Reth upstream issue in the sparse parallel trie. The fact that it logged WARN Failed to compute state root in parallel and then recovered via the serial fallback is the intended degradation path — Reth's parallel trie is a performance optimization, not the primary path. The EL continuing normally after that warn is expected behavior. Worth filing upstream with Reth separately, but it's not part of the CL sync deadlock described here.

Version question

Is this known to be addressed in the upcoming v0.8.0? The dev-v0.8.0 branch appears to be in progress from the open PRs but there's no public release yet — asking in case a workaround note in the docs or a patch on v0.7.3 is feasible in the meantime.

Thanks for flagging this and for the detailed report!

This is a known bug in the version of Malachite that Arc v0.7 uses. It’s already fixed upstream and will be rolled out in Arc v0.8.

Let's leave this issue open until Arc v0.8 is out so anyone else running into it can find it easily.

Good to have the fix confirmed and the upstream Malachite connection identified — that explains why the -32004 window alone was enough to trigger a permanent wedge rather than a transient stall.

Two quick follow-ups if useful:

1. Upstream Malachite PR/commit reference?
If there's a specific PR or commit in the Malachite repo that contains the range-re-request fix, linking it here would help anyone building Arc from source know exactly which point on main includes the fix — and would give the community a place to review the fix approach (a vs b vs c from the original report).

2. Watchdog remains relevant until v0.8 ships
The cron-based height-stagnation watchdog in my earlier comment is a reasonable interim mitigation for anyone running follower nodes on v0.7.3 who can't wait for v0.8. Worth keeping that visible in the issue for operators who find this page in the meantime.

Thanks for the quick turnaround on confirmation.

Recurrence data: two more wedges on the same follower node (v0.7.3) within one day — 2026-07-22

Same signature both times as the original report (tip frozen, sync_height a few heights past tip_height, all 5 parallel request slots pinned, Maximum number of parallel requests reached, skipping request for values max_parallel_requests=5 pending_requests=5 spam until restart):

went quiet (UTC)	tip_height	sync_height	recovered
~13:27	53,088,995	53,089,002	CL restart @ 13:32:42
~23:02	53,157,165	53,157,172	CL restart @ 23:09:36

So under current testnet conditions this is hitting roughly twice a day, not a rare race — worth factoring into how urgently v0.8 reaches followers.

For anybody reading this before v0.8: the height-stagnation watchdog approach discussed above works reliably — both wedges fully recovered without a resync or snapshot needed. We've since tightened our check interval from 5 minutes to 1 minute (with a short post-restart grace period before re-checking), which caps the stall outage at ~1–2 minutes.

Happy to attach full log excerpts from either occurrence if useful for validating the fix.

docs: add sync-wedge analysis accompanying the BATCH_SIZE=1 patch

The recurrence data is useful for the Arc team — twice a day is well into "reliably reproducible on demand" territory, not a rare race, which should make fix validation straightforward once v0.8 ships.

One pattern worth flagging across all three wedge observations (including the original report):

Event	tip_height	sync_height	gap
2026-07-21 original	52,893,873	52,893,880	7
2026-07-22 ~13:27	53,088,995	53,089,002	7
2026-07-22 ~23:02	53,157,165	53,157,172	7

The gap is consistently exactly 7 across all three: max_parallel_requests (5) + failed batch width (2) = 7. This is the expected mechanical outcome when the failed range covers exactly 2 heights — sync advances sync_height past both, retries only the first, then fills all 5 parallel slots with the following heights.

A few things this is useful for:

Watchdog precision. Height stagnation alone (tip hasn't moved in N minutes) is the right primary trigger, but adding a secondary check — sync_height - tip_height >= 6 at stagnation time — immediately distinguishes this specific bug from a general slow-sync or peer-connectivity issue. The latter would show a much smaller or larger gap. The 1-minute check interval you've settled on is tight enough that this combined check could drop the false-positive restart rate if you're seeing any.

Log excerpt value. If the gap in the new log excerpts is also exactly 7 in both cases, that confirms the failing batch is always 2 heights wide under current testnet conditions — consistent with the original sequence. If either shows a gap of 8 or 9, the batch width is occasionally 3+, which would mean the fix also needs to handle wider ranges (and is worth the Arc team knowing before v0.8 cut). That's the specific thing worth checking in those excerpts beyond confirming the bug recurred.

EL cleanliness across all three. All three recoveries were pure CL restarts — no EL intervention, no resync, no snapshot. That consistency confirms the wedge state is entirely in the CL's in-memory sync machine. EL state is intact throughout, which is also a useful fix-validation criterion: post-v0.8, a node that previously would have wedged here should simply continue advancing without any intervention, and the EL should see no disruption.

Interim mitigation result: BATCH_SIZE=1 (single-height sync requests) — 0 wedges in 41h+ on a previously-2x/day follower

Following up on my earlier data point with a concrete mitigation and its results, in case it's useful to other operators running v0.7.3 followers.

What we changed. Since the range-split can only skip a height when a multi-height range fetch partially fails, we pinned the sync batch size to 1 so every request covers exactly one height — a fetch then either delivers its height or fails whole and is re-requested; there's no partial case to lose a height in. On v0.7.3 that's a one-line change to malachite-app's hardcoded sync config (BATCH_SIZE: 10 → 1).

Deployed build (v0.7.3 + this patch, CL only, EL stock), with the reasoning and the residual trigger classes: https://github.com/arjun215-eng/arc-node/tree/v0.7.3-single-height-sync

Results (~41.4h after deploy, follower on Arc Testnet):

	Stock v0.7.3 (baseline)	With `BATCH_SIZE=1`
Wedges	~2/day (2 in the 24h before the patch)	0 in 41h+
CL restarts needed	per wedge	0 — single continuous process
Sync lag vs public tip	—	0, held across ~137k blocks/day

Cost: single-height fetches mean ~10x more RPC round trips during catch-up (2 per block instead of ~2 per 10-block batch). On a follower this was invisible — lag stayed pinned at 0 and it never tripped provider rate limits in steady state. It would matter more for a node doing a long cold catch-up.

Caveats. This only removes the range-split trigger. As noted in the writeup, a dead pending-request entry can in principle still form via other paths (a fetched value dropped from a full buffer, a processing-error attribution miss, timeout/response races) — none observed here, but the terminal signature would be identical, so we kept a height-stagnation watchdog running alongside regardless. This is why the tip_height + 1 < min(pending_heights) gap-detection safety net discussed earlier would be valuable upstream even with the range re-request fix in place.

Useful data — 41h+ clean is a meaningful baseline given the previous 2x/day rate.

A few notes from looking at the fork diff:

Watchdog simplification on the patched build

The gap-fingerprint secondary check I mentioned in an earlier comment — sync_height - tip_height >= 6 at stagnation time — is now structurally irrelevant on BATCH_SIZE=1. That specific signature requires a multi-height range to partially fail, which can't happen when every request covers exactly one height. Operators running this patch can simplify the watchdog back to pure height-stagnation detection and drop the gap check:

if [ "$CURR" -le "$PREV" ]; then
  systemctl restart arc-node-consensus
fi

The gap check is still worth keeping on stock v0.7.3 if distinguishing "this specific bug" from "general slow-sync" matters for alerting. On the patched build it adds noise.

The watchdog itself is still worth running alongside the patch, for the residual paths you flagged. If one of those alternate sources fires, the gap signature would differ from 7 (no range-split means no fixed-width hole before the 5-slot fill), which would itself be a useful diagnostic signal for the Arc team — a wedge with sync_height - tip_height != 7 post-patch would point to one of the non-range-split paths rather than a regression.

The git_branch addition

The fork also exposes git_branch in the /version RPC response (alongside the existing git_commit, git_short_hash, cargo_version). Small thing, but it means a quick GET /version confirms which build is active — "git_branch": "v0.7.3-single-height-sync" vs "main" vs a tag checkout — without needing to inspect binary metadata or check the systemd unit. Useful when coordinating with a monitoring system or confirming a rollout across multiple hosts.

On the July 21 recovery

The SYNC_WEDGE_ANALYSIS.md notes the first incident used a snapshot restore rather than a plain restart. That makes sense for an 18h gap (~100k+ blocks at Arc's ~1.6 block/s rate) — restart is always sufficient to unblock consensus, but snapshot avoids the catch-up time when the lag is large. The distinction is worth noting for anyone hitting this for the first time: CL restart is the minimal fix; snapshot is optional if the downtime was long enough that re-syncing from scratch is preferable.

This looks like it's already fixed upstream in the malachite fork, just not yet pulled into arc-node's pin.

Root cause lives in crates/sync/src/handle.rs. When a batched fetch fails, re_request_values_from_peer_except rewinds sync_height back toward the start of the failed range so the untried heights can be re-requested — but the unconditional forward set at the end of send_and_track_request_to_peer:

// Update sync_height to the next uncovered height after this range
set_sync_height(state, final_range.end().increment());

clobbers that rewind. The stranded tail of the range ends up below sync_height and outside pending_requests, so request_values never re-issues it, the parallel slots fill with later heights, and the follower wedges — exactly the 52893874-fell-through-the-gap sequence in the report.

The fix is already merged: circlefin/malachite #1567 — fix(sync): preserve rewound sync_height after concurrent re-request (merged 2026-06-22) makes that forward set_sync_height conditional so it no longer overwrites a rewind. Two related sync-robustness fixes also landed after the current pin and are worth getting in the same bump: #1543 (reject non-contiguous sync responses) and #1556 (ignore stale invalid-value responses).

Why v0.7.3 still wedges: arc-node currently pins malachite at rev = 8ee5d998… (2026-05-22), which predates all three of those PRs. So the fix exists on main but isn't in the built binary.

Suggested resolution: bump the circlefin/malachite git dependency to a commit that includes #1567 (and ideally #1543 / #1556), then re-pin. Happy to open that dependency-bump PR if it's useful, though I suspect the periodic fork-sync process may already cover it — flagging so the gap is visible. (Separately, the reth parallel-state-root panic mentioned at the end looks unrelated and would be worth its own issue.)

@aproskill81

Good investigation — the root cause and the #1567 fix are correctly identified. A couple of corrections on the other two PRs after checking the repo directly:

#1543 is already in the current pin

fix(sync): reject sync responses with non-contiguous heights merged 2026-04-08 (9109e96e), six weeks before the 8ee5d998 pin (2026-05-22). Comparing the two on the GitHub API (8ee5d998...9109e96e) returns status=behind, ahead_by=0, behind_by=54 — 9109e96e is an ancestor of 8ee5d998, so the non-contiguous rejection is already present in the v0.7.3 build. It doesn't prevent the wedge (it validates incoming responses; the stranded height never reaches the response stage because sync stops issuing requests for it), but it's not a missing piece for the bump.

#1556 was never merged

fix(sync): ignore stale invalid value responses was auto-closed on creation (2026-05-12) because the contributor (@aproskill81) wasn't pre-assigned to issue #1555 — the Malachite bot immediately closed it on contribution-process grounds, with zero review. merge_commit_sha being populated is GitHub's speculative merge preview, not evidence it landed. Whether the underlying fix reached main via the downstream fork port (#1561, the commit that is 8ee5d998) is plausible — #1561's body says it "brings 3 commits forward from a downstream fork" and the timing (port merged 2026-05-22, #1556 created 2026-05-12) lines up — but that's independent of #1556 and would need to be verified against the 3 commits in #1561 directly.

Bump target

The only fix missing from the current pin is #1567. Its merge commit 5cd137fb158fc52b55044eee9b090fd0d34996b0 (2026-06-22) is also the current tip of malachite main — no commits have landed since. So the bump is clean: one commit beyond 8ee5d998, and bumping to current main and pinning to 5cd137fb are equivalent right now. All 13 malachite packages in Cargo.toml use the same rev = "8ee5d998" so it's a mechanical find-replace to 5cd137fb158fc52b55044eee9b090fd0d34996b0.

A dep-bump PR targeting exactly that SHA would be self-contained and straightforward to validate against the twice-a-day recurrence rate we're seeing on testnet.

A candidate fix for the bug — running for 30h on the follower that was wedging 2×/day, no stalls so far, and the failure it targets has occurred twice and been handled both times.

The remaining bug in v0.7.3 (which ships circlefin/malachite#1567)

When an in-flight range request fails, re_request_values_from_peer_except picks a replacement peer via random_peer_with_except, which returns the peer and the portion of the range that peer can serve. If no peer covers the whole range, that portion is a trimmed prefix (range.start()..=peer.tip_height). The trimmed prefix gets requested — and the rest of the range is dropped: no code path re-requests it. The dropped height sits below sync_height (which only ever scans upward) with no pending request covering it, so it is never re-issued; the parallel slots fill with later heights that consensus can't apply until the gap is filled, and the follower stalls permanently. The same trim-and-drop exists in request_values_range, the helper used to re-request the remainder of partial responses — it's single-shot, so its own request can be trimmed the same way.

This is untouched by circlefin/malachite#1567 (which guards sync_height in the no-peer-available branch — a different path). It also explains the constant stall signature: a 2-height request that loses a 1-height tail, with 5 slots filling behind it, gives sync_height − tip = 7 every time.

The fix

One commit, two functions in crates/sync/src/handle.rs, mirroring what on_valid_value_response already does for partial responses:

request_values_range now loops until the requested range is fully covered across peers, or rewinds sync_height to the uncovered part if no peer can serve it yet.
re_request_values_from_peer_except re-requests the remainder (peer_range.end()+1 ..= range.end()) after sending a trimmed prefix, instead of dropping it.

Includes a regression test (test_re_request_covers_trimmed_suffix_to_other_peer) that reproduces the stall on unpatched code — remainder dropped, sync_height stuck past it — and passes with the fix. Full malachite sync suite passes (46/46).

The patch: arjun215-eng/malachite@378b0e5 — one commit on top of 5cd137fb (current malachite main / the #1567 merge commit); branch 5cd137fb-arc-trimdrop-fix

Results on Arc Testnet (same follower as the earlier reports; stock `BATCH_SIZE=10`)

metric (first 30h)	result
stalls	0 (this node on stock v0.7.3: ~2/day)
re-request path executions	~72,000, no regressions, node at chain tip throughout
trimmed re-requests — the exact failure that used to wedge	2, both handled, zero impact

Both occurrences match the incident shape from the original report: a 2-height request failed on two peers in a row, the third peer's tip was mid-range, and the re-request came back trimmed. On stock code that is the wedge. With the fix, the remainder was re-requested ~100µs later and all heights committed within ~350ms. From the first occurrence (2026-07-28, peer IDs shortened):

07:50:40.683200  Requesting sync range=54056816..=54056817 -> peer B     (re-hand after peer A failed)
07:50:40.857178  Fetch failed request_id=143415 (-32004)                 (peer B failed too)
07:50:40.857242  Re-requesting request_id=143415 except=peer B           (only peer C left; its tip = ...816)
07:50:40.857256  Requesting sync range=54056816..=54056816 -> peer C     (trimmed prefix)
07:50:40.857344  Requesting sync range=54056817..=54056817 -> peer A     (the remainder — previously dropped, now re-requested)
07:50:40.918…41.210  heights 54056816 / 817 / 818 committed

Two occurrences in 30h matches the rate at which this node used to wedge — it is hitting the same failure at the same frequency and absorbing it.

Upstream

As far as the public repos show, malachite main has no commit beyond 5cd137fb (#1567), so there is currently nothing for a version bump to pick up that would address this — it needs a fix like the above to land in malachite first. Happy to open this as a PR against circlefin/malachite; per CONTRIBUTING it needs an issue assignment first, so let us know how to route it. Until something lands, operators' options are this patch, or the BATCH_SIZE=1 workaround from the earlier comment. We recommend this patch.

@romac

The fix looks mechanically correct. A few observations from reading the diff, the test, and the malachite CONTRIBUTING, in case they're useful before routing upstream.

Loop termination in `request_values_range` — verified

The commit message states the invariant: random_peer_with only returns peers whose coverage starts at range.start(), so peer_range.end() >= range.start(). From the patched handle.rs (line 743):

"at range.start(), so peer_range.end() >= range.start() and each iteration's suffix starts strictly higher."

That holds because random_peer_with filters to peers where history_min_height <= range.start() <= tip_height. The returned peer_range is always range.start()..=min(peer.tip_height, range.end()), so peer_range.end() >= range.start(). The next iteration's range.start() = peer_range.end().increment(), strictly greater than the previous start. The height domain is a finite well-ordered set, so the loop terminates in at most range.len() iterations. ✓

One thing for the upstream reviewer: transient `max_parallel_requests` overshoot

The existing comment in request_values_range justifies not enforcing max_parallel_requests because "the original request is not on the wire anymore" — one freed slot for one sub-request. That reasoning covers the first loop iteration, but subsequent iterations each add a net-new pending entry (no slot was freed for them). For a range split across N lagging peers, N pending entries are created and only one slot was freed, so the pending count exceeds the budget by N−1 transiently.

In practice on Arc Testnet (3 public RPC peers, batch_size=10): N ≤ 3, so worst case +2 over budget for the duration of those requests, each of which has its own timeout and self-corrects. Not a correctness issue — temporary overshoot is strictly better than dropping a height — but worth a comment in the loop so the reviewer understands the intent and a future maintainer doesn't "fix" it by adding an early return that re-introduces the bug.

Excluded peers not propagated to suffix — correct, worth documenting

In re_request_values_from_peer_except, the suffix is re-requested via request_values_range(suffix), which uses random_peer_with (no exclusion list). The peer that was just excluded (the one that failed) is not excluded from the suffix request. In the field trace this was correct: the suffix went to peer A, which had recently failed on the original request but was available and could serve the suffix height when it was reissued ~100µs later. The peer scorer's penalty already suppresses the failed peer where possible.

The only scenario where this matters is a 2-peer topology where only the excluded peer has the suffix height — in which case it goes back to the recently-failed peer rather than being dropped. That's the right trade-off. A comment on the call site (// excluded_peers not propagated: suffix goes to any peer including the failed one if it's the only option) would make the intent clear to reviewers.

Test topology is precise

peer_suffix has history_min_height = 12, making it ineligible for the full 11..=12 range (can't serve range.start() = 11) but eligible for the 12..=12 suffix. This means random_peer_with skips peer_suffix in the first iteration and finds it in the second. The test would fail on unpatched code exactly as described: suffix dropped, sync_height stays at 13, height 12 stranded. The two-tier assertion (weaker suffix_pending || suffix_reachable_via_sync_height, then stronger suffix_pending) is also right — it distinguishes "genuinely re-requested" from "accidentally reachable via rollback". ✓

Upstream routing per `circlefin/malachite` CONTRIBUTING

The CONTRIBUTING.md is strict about unsolicited PRs:

We do not accept unsolicited contributions. PRs submitted without prior issue assignment or maintainer approval will be closed immediately.
The submitter must be a core contributor OR explicitly assigned to the issue.

The process it requires:

Open an issue on circlefin/malachite — the root-cause section of this comment is already a clean issue body. The field reproduction data (circlefin/arc-node#214, ~2 wedges/day, 30h clean with the fix) and the commit message are ready to paste.
Comment requesting assignment and wait for maintainer approval before submitting the PR.
Target branch: malachite has both main (currently at 5cd137fb) and sync/dev-v0_8_0. The fix is on top of 5cd137fb (directly applicable to both). Worth asking @romac in the issue which branch they want it against — if sync/dev-v0_8_0 is where the v0.8 sync work is landing, targeting that branch avoids a rebase later.

Alternatively, since @romac already confirmed this is a known bug fixed in v0.8 — it's possible the fix is being developed internally and the malachite issue tracker is the right place to ask whether they want an external PR or prefer to handle it on their side. Either way, the issue-first path is required.

The patch commit message is already well-formatted for an issue/PR body (root cause, distinctness from #1567, fix description, test, field reproduction). The only thing to add would be a note on the two transient-overshoot and excluded-peer items above so the reviewer doesn't have to rediscover them.

Opened #227 — bumps the malachite pin to 5cd137fb (current main tip), which includes #1567.

@arjun215-eng

Follow-up on where the actual fix has to land. I checked circlefin/malachite and there is no open PR carrying this fix upstream yet — the only open sync PR is #1577 (stale invalid-value responses, a different bug), and #1567 (merged) only guarded the no-peer branch. So the arc-node side (#227) can't close this until the loop fix is in malachite main and the pin points at that commit.

@arjun215-eng — your fork commit 378b0e5 is the fix; it just needs to be a PR against circlefin/malachite:main. I re-read it against 5cd137fb and it's the right shape end to end: both one-shot paths (re_request_values_from_peer_except @ handle.rs:933 and request_values_range @ handle.rs:750) become loops over the peer-trimmed suffix, and the regression test strands height 12 in a 3-peer topology — which fails on unpatched code and passes after. Termination holds (random_peer_with always returns coverage starting at range.start(), so each suffix starts strictly higher; finite height domain ⇒ ≤ range.len() iterations).

Since it's your work, best if you open it. A ready-to-paste draft so it's zero-friction:

Title: fix(sync): re-request the untried suffix when a re-request is trimmed to a prefix

Base: circlefin/malachite:main · Closes: the follower sync wedge tracked in circlefin/arc-node#214

Body:
When a failed or partial range is re-requested, filter_peers_by_range returns only the prefix a lagging peer can serve (range.start()..=peer.tip_height). The two one-shot re-request paths — re_request_values_from_peer_except and request_values_range — sent only that prefix and dropped the untried suffix. Because the original request had already advanced sync_height past the range end, the dropped suffix sits below sync_height with no covering pending request and is never re-issued (request_values only scans upward); the parallel slots fill with higher heights consensus can't decide, and the follower wedges permanently.
Fix: loop each path over the peer-trimmed suffix so the whole range is eventually covered across peers, preserving the existing no-peer sync_height rollback as the loop's exit. Adds a regression test that strands a suffix height in a multi-peer topology (fails before, passes after).
Repro trigger: any failed multi-height fetch where the replacement peer's tip_height is below the failed range's end.

Happy to comment on the malachite PR with the same trace once it's up, and to help verify a repro against testnet. If you'd rather not carry it, say so and I'll coordinate getting it in front of the malachite maintainers.

Following up on the verification offer above: I spun up an Arc Testnet node to help soak-test the fix once it lands, but realized the standard follow-mode setup (--follow.endpoint ...) does not exercise the P2P sync path at all (it fetches finalized blocks over RPC, bypassing malachitebft-sync entirely) so it cannot reproduce or verify this bug.

Per docs/running-an-arc-node.md, real P2P/gossip participation (CL P2P port 27000) is restricted to peer IPs provided during onboarding as a network sentry. Is there a path for an external node operator to get onboarded as a P2P/gossip peer on Arc Testnet specifically to help verify sync fixes like this one? Happy to run a 48-72h soak once the malachite fix (arjun215-eng's commit) lands and the pin is bumped here, if there's a way to actually join the gossip layer.

@ygd58

@ygd58 Good news — you don't need P2P onboarding for this one. The premise that follow mode "bypasses malachitebft-sync entirely" isn't right, and the issue itself is the proof: the original wedge in this report happened on a follow-mode node (see the environment section — --follow --follow.endpoint https://rpc.drpc.testnet.arc.network,...).

How it actually works (from crates/malachite-app/src/rpc_sync/mod.rs on current main): follow mode replaces the libp2p transport, not the sync logic. The module doc spells it out:

RPC Network Actor: … sends NetworkEvent::Status when peer heights update, and handles OutgoingRequest from malachite's sync to fetch blocks via RPC.

Malachite's Default Sync: Receives Status events, tracks peers, and makes parallel batch requests based on configuration.

RPC Peers: Maps RPC endpoints to malachite PeerIds.

So the buggy code path — sync_height advancing past a failed multi-height range while only the prefix is re-requested (re_request_values_from_peer_except / request_values_range in malachite's handle.rs) — runs identically in follow mode. Each --follow.endpoint is just a "peer" whose tip is tracked via WS subscription (PeerHeightUpdate), and the chronic -32004 Certificate not found failures from the drpc endpoint are exactly the fetch-failure trigger from the timeline above.

Practical soak-test setup, no onboarding needed:

Run your CL in follow mode against multiple endpoints (comma-separated) so the sync actor has several "peers" with independently-updating tips — that's what makes range trimming/re-request races possible.
The trigger is any failed multi-height fetch, which on testnet occurs organically via -32004 (the reporter saw them chronically for weeks before one landed on a 2-height range).
Watch for the wedge signature: sync_height > consensus height with no new Requesting sync from peer lines, services still active — or just alert on decided-height staleness (>N minutes with peers ahead).

Caveat on determinism: the wedge needs the failure to land on a multi-height range at the wrong moment, so a live soak is probabilistic (~weeks of exposure in the original report). For a deterministic check, the regression test in arjun's malachite commit (strands a suffix height in a multi-peer topology) is the fast path — a soak in follow mode is then good complementary validation that the pin bump ('#227'-style) behaves under real testnet conditions.

On the actual onboarding question: docs/running-an-arc-node.md only describes CL P2P (27000) as restricted to sentry IPs provided during onboarding, and I don't see any documented self-serve path to join the gossip layer — that's a question for the Circle team. But for this bug, follow mode is the honest reproduction environment anyway, since that's where it was observed.

You are right, and I was wrong — thanks for the correction. I misread --follow as bypassing malachite's sync layer entirely; it only swaps the libp2p transport for RPC/WS, and rpc_sync/network.rs confirms OutgoingRequests from malachite's sync (malachitebft_sync::Request::ValueRequest) are handled the same way, so the buggy re-request path runs identically. Retracting the onboarding question — follow mode against multiple --follow.endpoint values is the right (and honest) reproduction environment, as the original report already showed. Will set up a multi-endpoint follow-mode soak once the malachite suffix-loop fix lands, watching for the sync_height-stall signature you described, as complementary validation alongside the regression test.

@ygd58

@ygd58 Two additions that should make the soak more effective when the time comes:

1. Alert on metrics, not log-grepping. The CL exports a Prometheus registry under the arc_malachite_app prefix (crates/malachite-app/src/metrics/app.rs), and a couple of these map directly onto the wedge signature:

arc_malachite_app_block_time (histogram) — its sample count stops increasing the moment consensus stops deciding. rate(..._count[5m]) == 0 while your follow endpoints keep advancing is the cleanest single alert.
arc_malachite_app_sync_fell_behind_count — incremented on the InSync → CatchingUp transition (handlers/decided.rs). A fell-behind event not followed by resumed block-time samples within a few minutes is exactly the wedge (a healthy catch-up resumes; a wedged one never does).
arc_malachite_app_height_restart_count — should stay flat during the soak; a manual-restart recovery shows up here, which also makes it a good "did anyone touch the node" tripwire for a 48–72h unattended run.

This is the same signal set arc-node's own quake harness uses for health checks (quake/src/tests/health.rs asserts zero deltas on height_restart_count and sync_fell_behind_count over a healthy run), so alerting on them keeps your soak consistent with the project's internal definition of healthy.

2. Timing: there's nothing to soak against yet. I re-checked circlefin/malachite open PRs today — the suffix-loop fix (arjun's fork commit) still hasn't been opened as a PR; the only open sync PR remains your #1577 (stale invalid-value responses, a different bug). So the dependency chain is: malachite PR opens → merges to main → arc-node #227's pin is retargeted at that commit → soak becomes meaningful. Worth watching #227 rather than this issue for the "pin now includes the fix" moment, since a pin bump to a commit without the loop fix (like the current #227 target) would soak clean for the wrong reason — the bug is probabilistic enough (~weeks of exposure in the original report) that a green 72h run on an unfixed pin proves nothing.

One suggestion for making the soak stronger given that probabilism: pair the two builds. Run the pre-fix and post-fix pins side by side against the same --follow.endpoint set for the same window. If neither wedges, the run is inconclusive (expected); but if the control wedges and the fixed node doesn't, that's a genuinely strong field result rather than just an absence of failure.

Makes sense, thanks — verified all three metric names and the quake health-check reference against current main, all check out. Revised plan:

Watching chore(deps): bump malachite pin to current main — hardens no-peer sync branch #227 (not this issue) for the pin retarget once the malachite suffix-loop fix lands, since a soak against the current 5cd137fb pin would prove nothing either way.
When it does: A/B soak, old pin as control vs. new pin as test, same --follow.endpoint set, same window, rather than a single-build run.
Alerting on rate(arc_malachite_app_block_time_count[5m]) == 0 (with follow endpoints still advancing) as the primary wedge signal, sync_fell_behind_count transitions not followed by resumed block_time samples as secondary, and height_restart_count flat as an unattended-run tripwire — matching quake's own healthy-run definition.

Will report back here once the malachite PR is up and the pin is retargeted.

@ygd58

@ygd58 Plan looks right — one pin-bookkeeping correction so the A/B builds are cut from the right commits:

5cd137fb isn't the current pin. arc-node main still pins malachite at 8ee5d998 (all [workspace.dependencies.malachitebft-*] entries in Cargo.toml); 5cd137fb is only what #227 proposes to move to. So the pin ladder is:

8ee5d998 — current main. No suffix-loop fix.
5cd137fb — chore(deps): bump malachite pin to current main — hardens no-peer sync branch #227's target (malachite main tip incl. #1567's no-peer guard). Still no suffix-loop fix.
(future) malachite main after arjun's suffix-loop PR merges — the first commit worth soaking as "test".

Which means for the A/B run, the control can be either of the first two (they're equivalent with respect to this bug — #1567 only guarded the no-peer branch, not the prefix-trim drop), and the test build has to be a third pin that doesn't exist yet. Practically: control = whatever release/pin you'd run in production anyway, test = same tree with only the malachite rev changed, so the delta is exactly the fix.

On the remaining critical path: still no upstream PR carrying the fix as of today. @arjun215-eng, the offer stands — the draft PR text is in my comment above, ready to paste. If it hasn't been opened in the next few days I'll file a malachite issue instead, linking your fork commit 378b0e5 and the trace from this thread, so the maintainers have a review target while the authorship stays clearly yours.

chore(deps): bump malachite pin to current main — hardens no-peer syn…

ci: validate Arc circlefin#214 Malachite pin

Fixed as of #285

Navigation Menu

Repository navigation

Follower sync deadlock: failed multi-height fetch skips a height, consensus wedges permanently (v0.7.3) #214

Summary

Environment

Timeline (2026-07-21, UTC)

Expected behavior

Possibly related

Confirmed on Arc Testnet (client-side observations) + fix recommendation + mitigation

On the -32004 trigger rate

On the three proposed fixes

Mitigation until the fix ships

On the EL parallel trie panic (possibly related)

Version question

4 remaining items

The remaining bug in v0.7.3 (which ships circlefin/malachite#1567)

The fix

Results on Arc Testnet (same follower as the earlier reports; stock `BATCH_SIZE=10`)

Upstream

Loop termination in `request_values_range` — verified

One thing for the upstream reviewer: transient `max_parallel_requests` overshoot

Excluded peers not propagated to suffix — correct, worth documenting

Test topology is precise

Upstream routing per `circlefin/malachite` CONTRIBUTING

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Follower sync deadlock: failed multi-height fetch skips a height, consensus wedges permanently (v0.7.3) #214

Description

Summary

Environment

Timeline (2026-07-21, UTC)

Expected behavior

Possibly related

Activity

osr21 commented on Jul 22, 2026

Confirmed on Arc Testnet (client-side observations) + fix recommendation + mitigation

On the -32004 trigger rate

On the three proposed fixes

Mitigation until the fix ships

On the EL parallel trie panic (possibly related)

Version question

romac commented on Jul 22, 2026

romac commented on Jul 22, 2026

osr21 commented on Jul 22, 2026

arjun215-eng commented on Jul 22, 2026

osr21 commented on Jul 23, 2026

arjun215-eng commented on Jul 24, 2026

osr21 commented on Jul 24, 2026

danilaverbena commented on Jul 27, 2026

osr21 commented on Jul 27, 2026

4 remaining items

arjun215-eng commented on Jul 28, 2026

The remaining bug in v0.7.3 (which ships circlefin/malachite#1567)

The fix

Results on Arc Testnet (same follower as the earlier reports; stock BATCH_SIZE=10)

Upstream

osr21 commented on Jul 29, 2026

Loop termination in request_values_range — verified

One thing for the upstream reviewer: transient max_parallel_requests overshoot

Excluded peers not propagated to suffix — correct, worth documenting

Test topology is precise

Upstream routing per circlefin/malachite CONTRIBUTING

ygd58 commented on Aug 6, 2026

osr21 commented on Aug 8, 2026

ygd58 commented on Aug 9, 2026

osr21 commented on Aug 9, 2026

ygd58 commented on Aug 9, 2026

osr21 commented on Aug 9, 2026

ygd58 commented on Aug 9, 2026

osr21 commented on Aug 9, 2026

romac commented on Aug 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Results on Arc Testnet (same follower as the earlier reports; stock `BATCH_SIZE=10`)

Loop termination in `request_values_range` — verified

One thing for the upstream reviewer: transient `max_parallel_requests` overshoot

Upstream routing per `circlefin/malachite` CONTRIBUTING