Skip to content

perf(core): maintain transaction count incrementally - #586

Open
raymondjacobson wants to merge 1 commit into
mainfrom
codex/incremental-transaction-count
Open

raymondjacobson wants to merge 1 commit into
mainfrom
codex/incremental-transaction-count

Conversation

@raymondjacobson

Copy link
Copy Markdown
Contributor

Summary

The transaction-count cache runs COUNT(*) over all of core_tx_stats every five blocks. On v.monophonic.digital, this query and its parallel workers ran for tens of seconds over roughly 68 million transactions while consensus was far behind.

Maintain an exact singleton counter with PostgreSQL statement triggers instead. Inserts count only rows actually inserted, including ON CONFLICT DO NOTHING; deletes decrement the count for rollback; transaction aborts undo both changes. TRUNCATE resets the count. The application reads this constant-time counter immediately at startup and every two seconds, without relying on lossy block notifications.

The migration performs one initial full count while holding a write-conflicting table lock to establish a consistent baseline. This adds a one-time startup/backfill cost, and concurrent writers serialize their counter updates. New snapshots include the counter in the same pg_dump snapshot as transaction stats. After restoring an older snapshot without a counter, seed it once before accepting the snapshot; new snapshots need no scan. The migration is safe to rerun after restoring an older migration ledger.

Tests

  • TEST_DB_URL=... go test -race ./pkg/core/db -run TestTransactionCount -v against an isolated PostgreSQL 16 instance, with pg_dump/pg_restore available
  • go test ./pkg/core/server -run 'Test.*(Snapshot|Truncate)'
  • go test ./pkg/core/server -run '^TestNonexistent$'
  • Regenerated SQL bindings with sqlc v1.29.0
  • git diff --check

The database test covers backfill, bulk inserts, duplicates, aborted inserts/deletes, rollback deletion, upserts, TRUNCATE, concurrent writers, old/new snapshot data, migration rerun/down, and an actual data-only pg_dump/pg_restore round trip with triggers disabled during COPY.

No production migration or deployment has been performed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant