fix(core): reset core state when the stored chain differs from genesis

[rickyrombo]

Problem

A validator node moving to audius-mainnet-beta changes its image tag and restarts. The new image's genesis names a new chain ID, so CometBFT starts from an empty data directory, but the node's PostgreSQL still holds the old chain's core state (core_blocks at ~34M on audius-mainnet-alpha-beta).

Before #591, the genesis-generation path in SetupNode set RunDownMigration whenever it had to write a genesis file, and startup ran core's migrations down. That emptied core_blocks, so the node state-synced onto the new chain. Runbook step 10 ("operators set an explicit image tag") relies on this.

#591 removed that reset, correctly for the case it targeted: a restore that brought PostgreSQL back without its CometBFT files was wiping its own database. It added checkCometDataFiles to refuse that case instead. But a migrating node looks the same to that check, so on current main it stops core with:

PostgreSQL contains block 34xxxxxx but CometBFT database ".../audius-mainnet-beta/data/state.db" is missing; restore PostgreSQL and the full chain data directory from a consistent backup ...

Even without the check, Info would report the old chain's height and app hash, and the handshake would refuse it.

Fix

Tell a chain change from a restore by the chain ID stored on the latest block. Every row in core_blocks records the chain that produced it.

• Latest block names another chain: clear core's consensus state, then continue as an unsynced node, so it state-syncs.
• Latest block names this chain: unchanged. A restore without CometBFT files is still refused by checkCometDataFiles.
• Empty chain_id: not a positive identification, so never reset.

What gets cleared is the state-sync snapshot table set (stateSyncSnapshotTables), minus core_db_migrations, using the same no-CASCADE TRUNCATE the restore uses (#551). That is all of core's consensus state (TestSnapshotTablesCoverCoreSchema already enforces that every core_* table is either in the set or explicitly exempt), so a node that block-syncs instead of state-syncing doesn't replay the new chain over the old chain's validators and auth state. Mediorum and ETL tables are untouched. core_db_migrations is kept because, unlike a restore, nothing reloads it, and clearing it would make the next startup re-run every migration.

This does not reinstate running migrations down. On a populated old-chain database some down steps can fail (e.g. 00019 re-adds UNIQUE (cid) on sound_recordings), and a truncate of the known set does the same job without that risk.

Tests

• TestIsForeignChainBlock: previous chain is foreign; same chain and empty chain_id are not.
• TestForeignChainResetStmtClearsCoreConsensusState: every snapshot table except core_db_migrations is cleared.
• TestForeignChainResetStmtKeepsMigrationsAndNonCoreTables: core_db_migrations, mediorum and ETL tables are not named, and there is no CASCADE.

make test-unit passes, including #591's TestCheckCometDataFiles and the state-sync truncate tests.

Rollout

Needs to be in the new-network build (#553) before runbook step 10. A migrating node also needs a snapshot to sync from: the bootstrap's first is at height 20,000 with BlockInterval=20000, and ProdStateSyncRpcs must point at nodes serving snapshots.

🤖 Generated with Claude Code