perf(core): maintain transaction count incrementally - #586
Open
raymondjacobson wants to merge 1 commit into
Open
raymondjacobson wants to merge 1 commit into
raymondjacobson wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The transaction-count cache runs
COUNT(*)over all ofcore_tx_statsevery five blocks. On v.monophonic.digital, this query and its parallel workers ran for tens of seconds over roughly 68 million transactions while consensus was far behind.Maintain an exact singleton counter with PostgreSQL statement triggers instead. Inserts count only rows actually inserted, including
ON CONFLICT DO NOTHING; deletes decrement the count for rollback; transaction aborts undo both changes. TRUNCATE resets the count. The application reads this constant-time counter immediately at startup and every two seconds, without relying on lossy block notifications.The migration performs one initial full count while holding a write-conflicting table lock to establish a consistent baseline. This adds a one-time startup/backfill cost, and concurrent writers serialize their counter updates. New snapshots include the counter in the same pg_dump snapshot as transaction stats. After restoring an older snapshot without a counter, seed it once before accepting the snapshot; new snapshots need no scan. The migration is safe to rerun after restoring an older migration ledger.
Tests
TEST_DB_URL=... go test -race ./pkg/core/db -run TestTransactionCount -vagainst an isolated PostgreSQL 16 instance, withpg_dump/pg_restoreavailablego test ./pkg/core/server -run 'Test.*(Snapshot|Truncate)'go test ./pkg/core/server -run '^TestNonexistent$'git diff --checkThe database test covers backfill, bulk inserts, duplicates, aborted inserts/deletes, rollback deletion, upserts, TRUNCATE, concurrent writers, old/new snapshot data, migration rerun/down, and an actual data-only pg_dump/pg_restore round trip with triggers disabled during COPY.
No production migration or deployment has been performed.