Skip to content

feat(genesis-writer): emit a portable chain dump and manifest - #595

Open
rickyrombo wants to merge 1 commit into
mainfrom
feat/genesis-writer-portable-dump
Open

rickyrombo wants to merge 1 commit into
mainfrom
feat/genesis-writer-portable-dump

Conversation

@rickyrombo

Copy link
Copy Markdown
Contributor

Summary

The genesis writer now produces a self-contained artifact instead of leaving a live Postgres database for the operator to seal and export.

  • <data-dir>/chain.dump: after a successful write (after genesis.json, state.db and the index rebuild), the writer runs pg_dump --format=directory --jobs=N on the chain DB before it exits, so nothing else has connected first. It dumps to chain.dump.partial and renames on success. It leaves out the writer's own genesis_writer_progress table, plus owners and ACLs. On by default; --no-dump (or --dump=false) skips it, and --dump-jobs sets the parallelism.
  • <data-dir>/MANIFEST.json: chain_id, end_height, first_live_height, source_chain_id, source_last_indexed_block (MAX(height) in the source's core_indexed_blocks for audius-mainnet-alpha-beta, read at the start of the run), new_chain_flush_from_block (= source + 1), genesis_sha256, genesis_validator_address, genesis_migration_address, dump.{pg_dump_version, pg_dump_major, server_version, server_major, restore_min_pg_major, size_bytes}, writer_commit, and exclude_from_bootstrap (node_key / priv_validator_key / addrbook / priv_validator_state, each with a reason and whether it was present).
  • Version pinning with --target-pg-major, which defaults to 15, the node image's bundled Postgres:
    • The managed Postgres uses exactly that major instead of the newest one installed. A cluster left by an earlier run on a different major is refused with a clear error.
    • With --dst-dsn, the writer checks the server's major before writing anything. A server newer than the target is refused unless you pass --allow-newer-postgres, and then it logs loudly.
    • A pg_dump with server ≤ major ≤ target is resolved up front, so a missing client fails in seconds, not after the run.
    • pg_dump/pg_restore never get the password on argv; it goes through PGPASSWORD.
  • ROLLOUT.md:
    • Step 1: serve and ship from the dump.
    • Appendix B is rewritten as "The artifact": why a dump, version pinning, and sealing kept as an optional fallback for --no-dump runs and the 2026-08-25 artifact.
    • Step 4: pg_restore into a new database before first start, then restore the mediorum tables, and exclude the manifest's identity files.
    • Steps 7 and 8 read heights from the manifest.
    • Step 11: NewChainFlushFromBlock = source last block + 1, because the flusher deletes confirmed_block < value.
    • Appendix E gets the exclusion table.
  • README: new flags, the output layout, and how the managed Postgres picks its major.
  • make test-unit now includes ./cmd/genesis-writer/.... Before this, no make target ran the package's unit tests. Its DB-backed tests skip without ETL_TEST_DB_URL, as they already did.

The 2026-08-25 artifacts predate this and still need the manual dump. This change is for future writer runs.

Test plan

  • make test-unit passes. It includes the new TestManifestDescribesTheArtifact, TestManifestWithoutDumpRecordsNull, TestParsePgToolMajor, TestPostgresVersionChecks (covers the PG17-server-into-PG15-node case) and TestPgDumpKeepsPasswordOffArgv.
  • Local check with a temporary test that was not committed, against real Homebrew clusters:
    • A PG17 server with target 15 is refused, and goes through with a loud warning under --allow-newer-postgres.
    • A PG15 server resolves PG15 pg_dump and dumps.
    • pg_restore --list (PG15) reads the archive: format 1.14, indexes and constraints included, progress table excluded.
    • A stale chain.dump is replaced.
  • Not run: the full writer integration test (cmd/genesis-writer make test, docker stack). It builds WriterConfig directly without ArtifactDir, so the artifact step is skipped there.

🤖 Generated with Claude Code

After a successful write, pg_dump the chain database (directory format,
parallel) to <data-dir>/chain.dump and write <data-dir>/MANIFEST.json, so
the artifact is self-contained instead of a live database operators must
seal and export by hand. The dump runs before the writer exits, so nothing
else has connected first. --no-dump skips it; the manifest is still written.

The manifest records chain_id, end/first-live height, the source snapshot's
last indexed old-chain block and new_chain_flush_from_block (= that + 1),
genesis sha256 and validator address, pg_dump/server versions, the writer
commit, and the per-node identity files not to seed onto a bootstrap node.

Pin Postgres versions to the node's bundled major (15) via
--target-pg-major: the managed cluster runs exactly that major, and a
--dst-dsn server newer than it is refused before writing unless
--allow-newer-postgres is given. A pg_dump in range is resolved up front.

Run cmd/genesis-writer unit tests in make test-unit, and update ROLLOUT.md
(steps 1, 4, 7, 8, 11; Appendices B, C, E) and README for the new flow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rickyrombo
rickyrombo force-pushed the feat/genesis-writer-portable-dump branch from 1f1a756 to b36dc0b Compare October 9, 2026 05:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant