Repository navigation
node stopped getting new blocks at 7083588 #51
Description
Node stopped getting new blocks at 7083588 (Wed Feb 26 2025 14:59:59 GMT+0000 (UTC)
I did upgrade the docker images with the newest in v.1.2.0 a day before needed (I triple checked everything).
We are atleast 3-4 dudes on Ink's dev-general channel on discord, that experienced the same problem.
I have tried restarting docker compose, also completely removing the instance with "docker compose down -v" and let it resync from scratch with and without the rock torrent download thingy. even with other settings than archive, like full and snap (snap is still syncing though) - but archive, after it has finished downloading and begun to catch up, it always keeps around ~43329 blocks behind the official top.
I have tried different L1-rpc's for execution and beacon, usually I use my own local hosted (nethermind, reth, lighthouse), but I have also tried online services like chainstack for a full unpruned beacon node.
software: ubuntu 22 lxc-container on proxmox, and followed the guide on this github.
errors:
protocols not supported: [/opstack/req/payload_by_number/57073/0]
healthcheck-1 | {"level":30,"time":1740640127229,"targetHeight":7098275,"referenceHeight":7141714,"heightDifference":43439,"minBlockNumber":7098275,"msg":"latest block heights"}
healthcheck-1 | {"level":50,"time":1740640127448,"extra":{"blockNumber":7098275,"referenceHash":"0xeafdc9982d3aef8529fdfe6a01fdd58527cdcbc20f0c612d6e5a276ab0264577","targetHash":"0xb7567a7e6907d1767778d043f2c80f0b95eaf7b5ebfbf816e4fea06fe8798224"},"msg":"reference client has different hash for block"}
healthcheck-1 | {"level":30,"time":1740640127448,"msg":"beginning binary search to find first mismatched block"}
op-node-1 | t=2025-02-27T08:02:38+0000 lvl=warn msg="L1 head signal indicates a possible L1 re-org" old_l1_head=0x2a8eefb8c0d5c16116fe3778e0ab44a85b27594c9f252104d8cc5a1656420ed3:21936344 new_l1_head_parent=0x3a56745e071a91f12c88ce52c919dc4d43a6cd49da6ef3fee81fe2b48d086708 new_l1_head=0x91fec02644b1bc1bd05a15543254ce283d360a40de455fd9e4e7eb5c7cec669c:21936346
op-node-1 | t=2025-02-27T08:02:40+0000 lvl=warn msg="dropping future span batch" origin=0x52c65466f01c6181e1bbf61285390818c8b1b19d55fa0a3ec86f8c7d87692358:21934660 epoch=0x74bae72d1bc891c1cf4ee2a7f4bc42b07767d3ffef2f43ba1b22c882bae4f60a:21932561 batch_type=SpanBatch batch_timestamp=1740617012 parent_check=0xe5afc7ab0f1e97e4ba9a53684b76fe915d541938 origin_check=0x8f42a6f7d0b0c3c66d194bd0d623221a932c04ea start_epoch_number=21934159 end_epoch_number=21934160 block_count=10 txs=0 next_timestamp=1740597671
op-node-1 | t=2025-02-27T08:02:40+0000 lvl=warn msg="Dropping invalid span batch, flushing channel" origin=0x52c65466f01c6181e1bbf61285390818c8b1b19d55fa0a3ec86f8c7d87692358:21934660 epoch=0x74bae72d1bc891c1cf4ee2a7f4bc42b07767d3ffef2f43ba1b22c882bae4f60a:21932561 batch_type=SpanBatch batch_timestamp=1740617012 parent_check=0xe5afc7ab0f1e97e4ba9a53684b76fe915d541938 origin_check=0x8f42a6f7d0b0c3c66d194bd0d623221a932c04ea start_epoch_number=21934159 end_epoch_number=21934160 block_count=10 txs=0
op-node-1 | t=2025-02-26T15:12:39+0000 lvl=warn msg="Dropping invalid unsafe payload" hash=0xe614f2437ad350142a3a461e2b299579efd00a650b74db3cf86ed87a5f3482aa number=7083589 timestamp=1740582000
op-node-1 | t=2025-02-26T15:12:39+0000 lvl=warn msg="Payload was invalid" block=0xe614f2437ad350142a3a461e2b299579efd00a650b74db3cf86ed87a5f3482aa:7083589 err="execution payload 0xe614f2437ad350142a3a461e2b299579efd00a650b74db3cf86ed87a5f3482aa:7083589 was INVALID! Latest valid hash is <nil>, ignoring bad block: 0xc011d16080" timestamp=1740582000
op-node-1 | t=2025-02-26T15:12:39+0000 lvl=warn msg="Engine temporary error" err="temp: cannot process unsafe payload: new - 0xe614f2437ad350142a3a461e2b299579efd00a650b74db3cf86ed87a5f3482aa:7083589; parent: 0x8362847a2fbd7e0291892836c1761af953ad3979abd2e3d9ca1972e387bae502:7083588; err: execution payload 0xe614f2437ad350142a3a461e2b299579efd00a650b74db3cf86ed87a5f3482aa:7083589 was INVALID! Latest valid hash is <nil>, ignoring bad block: 0xc011d16080"
Activity
algotradingstrategies commentedon Feb 28, 2025
I see the same.
For full node (using snap sync). I am using quicknode as L1.
It started exactly at 2025-02-26T15:00:00+0000 (which interestingly is mentioned in the release note of v1.2 here ).
Specificially, I keep seeing lines like this in my log
437ad350142a3a461e2b299579efd00a650b74db3cf86ed87a5f3482aa:7083589 was INVALID! Latest valid hash is , ignoring bad block: 0xc010fa2c20" ```