Skip to content

Darling store: an hourly row-count ledger for query_stats (V164) (part of #4605) - #5289

Merged
erikdarlingdata merged 9 commits into
devfrom
feat/4605-hour-ledger
Oct 5, 2026
Merged

erikdarlingdata merged 9 commits into
devfrom
feat/4605-hour-ledger

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Oct 5, 2026 •

Copy link
Copy Markdown
Owner

This is plan lane 1 of #4605 only: the V164 rung and the storage class. Other lanes add the writer, the writer census pin and the retention prune on this same branch, and the guard read and route test seeding in a second PR.

Why

The hourly-edges read may take its middle hours from the hourly rollup only when the rollup holds every raw row those hours have. Today the count guard proves that by counting raw collect.query_stats rows per (server, hour), a scan of millions of rows that runs past the panel deadline, so the panel falls back to raw. Option A of the plan (approved) keeps that count at write time instead: one ledger row per (server, hour), written in the same transaction as the collector's COPY. The guard then reads a few thousand ledger rows, and its cost follows the number of (server, hour) pairs and not the number of raw rows. This lane adds the storage for it. Lanes 2 and 6 below write and prune it, nothing reads it until the guard PR (#5290), and a user sees no change.

What changes

  • Darling/PerformanceMonitor.Darling.Storage/QueryStatsHourLedger.cs (new):
    • CreateSql (line 62): both tables and the state row. The rung embeds it (the V123 precedent), and the rung test pins its shape so an edit to a shipped rung fails there.
    • UpsertSql (line 93): $1 server id, $2 the storage name, $3 the batch's collection time (truncated to its hour), $4 the count. Adds to an existing hour. A count of 0 or less writes nothing, because the rollup has no row for an hour with no counted rows and a ledger row of 0 would not equal that.
    • RecountSql (line 112): $1 start, $2 end exclusive. Sets (does not add) the ledger from raw for those hours, deletes ledger rows whose raw rows are gone, and returns (rows written, rows deleted). It counts the rollup's population (sample_interval_seconds IS DISTINCT FROM 0, so restart rows are skipped and NULL intervals are counted), writes server_name as raw holds it (what the COPY wrote), and widens a ragged range to whole hours so a partial hour can never be written. It does not touch the state row. It uses date_trunc('hour', ...), so it also runs on a store without TimescaleDB.
  • PgMigrations.cs: V164Sql (line 656) and new Migration(164, "query-stats-hour-ledger", V164Sql) (line 859). StorageVersion.SchemaVersion is 164.
  • Tables (both plain, in collect, not collectors): query_stats_hour_ledger (server_id, server_name, bucket, n) and query_stats_hour_ledger_state (one row, id = 1, counted_since). The rung sets counted_since to the next whole hour with ON CONFLICT DO NOTHING, so a re-run keeps an earlier value. Hours before it are "uncounted" and the guard will stay on raw for them. Because the writer lands on this same branch, one deploy starts the counting.
  • ViewerDataService.cs: the schema probe's V164 sentinel (line 1090) and its MapProbedSchemaVersion arm (line 1329), so a store at V164 is not shown an upgrade banner.
  • Deliberate deviation from the plan: the primary key is (bucket, server_id, server_name), not (server_id, server_name, bucket). The guard reads a range of hours across every server and the retention prune deletes below one hour, so a key that leads with the hour makes both an index range scan with no second index. The plan's acceptance step (a 24 hour fleet read under 2 k buffers) cannot be met by a scan of the whole ledger. The upsert's conflict target names all three columns, and PostgreSQL matches a conflict target to a unique index without regard to column order.
  • Tests: QueryStatsHourLedgerTests.cs (new); StoreStatementHistoryRungTests (its probe test no longer assumes V163 is the top arm, as V163 did for V162); CaggGroupIndexDropLiveTests (drops the V164 tables when it simulates an older store); StoreSelfMetricsTests (the plain-table census lists both tables with a reason).

Grants evidence (no new grant, no provisioning change)

New collect tables get SELECT for admin, viewer and mcp from two things that provisioning re-asserts at every start, after migration (DarlingManagedRoles.BuildProvisioningSql, sections 2, 4 and 6):

BYO stores get the same from Darling/tools/provision-roles.sql (the ON ALL TABLES form at line 191 and 194, the default privileges at 355 and 370). collect.module_map and module_map_state have no grant of their own either; they rely on this path, and so do the V162, V163 and V123 tables. The collector writes as the owner. No role setting changes, so nothing moves into a migration.

Proved live: TheViewerRolesProvisionedGrants_ReadBothTables_AndWriteNeither creates a no-superuser role, replays exactly the viewer's provisioned GRANT/REVOKE statements (ViewerGrantReplay), then reads both ledger tables as that role and gets SQLSTATE 42501 on an INSERT. That covers the compose read in lane 3. I did not exercise the mcp role directly; its statements are the same lines and are pinned verbatim by DarlingManagedRolesTests.

Test plan

Build: 0 Warning(s), 0 Error(s). Run on a throwaway PostgreSQL 18.6 with TimescaleDB, on a fresh scratch database for each live test (the rung ran on a fresh store in MigratingAFreshStore_...). These are lane 1's numbers at its own commit. Later lanes' numbers are in their sections. The merged-tree numbers are under Review round 1. Each line is Total, Failed:

  • QueryStatsHourLedgerTests 10, 0 (schema pins; upsert adds to an existing hour, starts a new one, ignores 0 and negative; recount from raw skips restart rows, counts NULL intervals, keeps the raw name spelling, sets rather than adds, widens a ragged range, deletes a stale row, leaves rows outside the range alone; the state row is the next whole hour and survives a re-run; the viewer grants proof; the probe arm)
  • MigrationLadderPins 4, 0
  • MigrationDataMovingRungCensusPins 28, 0 (no declaration needed: the tables are empty and the one INSERT is a VALUES row into a table the same rung creates)
  • MigrationUpgradeLadderLiveTests 6, 0 (the generated-schema ladder replay over released stores)
  • StoreSelfMetricsTests 37, 0 (the plain-table census; it needed the two entries)
  • StoreStatementHistoryRungTests 6, 0; SlowReadsRungTests 4, 0; QueryStoreTopDailyRungTests 11, 0; CollectorRunTimeRungTests 19, 0; RawChunkIntervalRungHistoryRungTests 3, 0
  • ViewerSchemaVersionGateTests 34, 0; CaggGroupIndexDropLiveTests 5, 0
  • DarlingRetentionTests 47, 0 (retention walks the collector catalog, and neither table is a collector)
  • DarlingManagedRolesTests 27, 0; ProvisionRolesAclDriftTests 9, 0; ServerTagViewerRoleLiveTests 2, 0
  • LiveCleanupBatchTests 5, 0 (the helper keeps no list of collect tables; the lane-4 route tests add ledger cleanup)
  • DiagnosticsBundleVerbTests 21, 0; DiagnosticsBundleSizeTests 4, 0 (the bundle reads named tables and has no table inventory, and no section reads the ledger)
  • DocCommentHygieneTests 77, 0; PgSchemaGeneratorTests 25, 0; StoreObjectConvergenceTests 7, 0; DarlingObservabilityTests 11, 0; DarlingModuleMapTests 5, 0
  • The full suite: not run, CI runs it
  • The mcp role reading the tables as a live role: covered only by the pinned statements, see above
  • The acceptance measurements on a real store (recount cost, the 24 hour guard read, ledger against raw): they need the writer and the guard lanes first

Lanes after lane 1

Status: lanes 2, 5 and 6 are on this branch (their sections are below). Lanes 3 and 4 are the second PR (#5290).

  • Lane 2, the writer: DarlingCollectorRunner.CopyBatchOnceAsync runs UpsertSql after the COPY, inside its transaction, with the non-restart row count of the batch counted at the write (reset per attempt) and server.StorageName. A fault in the upsert aborts the batch.
  • Lane 5, the writer census pin: QueryStatsWriterCensusPins, expected writer set exactly CopyBatchOnceAsync.
  • Lane 6, retention: prune the ledger below the hourly rollup's horizon in the query_stats purge pass.
  • Then, in a second PR: lane 3 (the guard SQL reads the ledger, a window before counted_since returns -1) and lane 4 (route test seeding with RecountSql, ledger cleanup, LiveCleanupBatch).

For the coordinator to double-check

  • The primary key order deviation above.
  • UpsertSql ignoring a count of 0 or less, in the SQL as well as in the writer. If you want the writer to be the only gate, drop the WHERE $4::bigint > 0 line and its pin.
  • RecountSql on an hour that is still being written can overwrite a batch that commits during the statement. That leaves the ledger below the rollup, which fails the guard safe and never passes it wrongly, and it is stated on the constant. Tests and a backfill use it on closed hours.
  • Comments in QueryStatsHourLedger.cs, V164Sql and the census entry say the retention prune and the writer exist. They do: lanes 2 and 6 are on this branch.

SECTION: None
ENTRY: - No user-visible change in this PR ([#5289]) - This PR writes and prunes the ledger tables. Nothing reads them until the guard PR (#5290), so nothing a user sees changes.

Part of #4605

Lanes 5 and 6: the writer census and retention

Lane F's commit on this branch: 8737f74a3. It touches QueryStatsWriterCensusPins.cs (new), QueryStatsHourLedger.cs (one constant), DarlingRetention.cs (one block) and DarlingRetentionTests.cs. Nothing lane E owns is edited.

Lane 5, the census (Darling/Darling.Tests/QueryStatsWriterCensusPins.cs). The ledger counts rows at one write, so a second writer would put rows into raw and the hourly rollup that no ledger entry accounts for. If those rows are late, the guard cannot see the hole. The pin fails the test run on a literal write to query_stats in the service's sources. It also fails on a COPY start missing from its register, and the message says why. It is a text scan with blind spots, listed below, and it has three arms:

  • A literal scan of every non-test .cs source in the service's project-reference closure for INSERT INTO, COPY, UPDATE or MERGE INTO aimed at query_stats. The schema and quotes are optional, and the match is word-bounded, so query_stats_hour_ledger and the other query_stats_* relations never match. The closure is every Darling/PerformanceMonitor.* project and every project those reference, transitively, so the six root libraries compiled into the same service are included. Comments are stripped and string literals kept, because the SQL lives in them.
  • A register of every COPY start (BeginBinaryImportAsync, BeginTextImport, BeginRawBinaryCopy), because the real COPY text is built at run time (CopyCommandFor appends schema.TargetTable), so the literal scan cannot see it. The register holds the runner's one site (generic definition, and it must sit inside CopyBatchOnceAsync, declared once) and four RDS ingestors, each over a fixed Pg*Collector.Instance with its table named (pg_cpu_utilization, pg_deadlocks, pg_log_events, pg_plan_capture) and compared with the definition's real TargetTable. A fifth pin: QueryStatsCollector is the only catalog definition whose table is query_stats.
  • No migration rung above V164 runs DML (the four verbs, plus DELETE FROM and TRUNCATE) on the table.
  • The scan proves it looked: it asserts it read more than 300 files and that the runner, retention, row writer and ledger files were among them. It also asserts that PerformanceMonitor.Collectors/QueryStatsCollector.cs was read. That file sits outside Darling/, so it shows the walk left the Darling tree. Each arm has positive and negative controls on fabricated sources (TheLiteralScan_*, TheCopyScan_*, TheMigrationArm_*).
  • Blind spots, stated in the class comment: the pin is a text scan. It cannot see SQL built at run time outside a COPY (an interpolated string, a StringBuilder or a concatenation across literals). It cannot see SQL in a file that is not .cs, and none exists in these projects today. It cannot see a writer inside the database, such as a trigger, a rule, a function or a scheduled job. It cannot see a COPY reached through a method group. Projects outside Darling/PerformanceMonitor.* are no longer a blind spot (review round 1). Dynamic SQL inside a rung is already closed by MigrationDataMovingRungCensusPins.TheLadderStillContainsNoDynamicSql.

RED proof: I added Darling/PerformanceMonitor.Darling.Service/ZzPlantedQueryStatsWriter.cs holding "INSERT INTO collect.query_stats (...) VALUES (...)" and re-ran the built class. TheWritersOfQueryStats_AreExactly_CopyBatchOnceAsync failed with Found: {CopyBatchOnceAsync, PerformanceMonitor.Darling.Service/ZzPlantedQueryStatsWriter.cs:5 (INSERT INTO collect.query_stats)} followed by the ledger explanation. I removed the plant (it was never committed) and the class went back to 21 passed. A first version of the pin also failed legitimately on its own bug (it compared the RDS definition expression with the identifier definition, and flagged the four RDS sites as unlisted); the fix is in the commit.

Lane 6, retention (DarlingRetention.cs, QueryStatsHourLedger.PruneSql). A new failure-isolated step in the sweep, after the per-collector loop and outside every timescaleAvailable branch, deletes collect.query_stats_hour_ledger rows with bucket < date_trunc('hour', now - TimescaleSupport.HourlyRetentionSpan). That is the successor hourly rollup's horizon, 90 days, derived from the same constant the rollup's own retention policy and the read router use. It is not raw's four days: the guard compares the ledger with the rollup, and a ledger pruned at raw's horizon would be missing hours the rollup still holds, so the guard would fail every window past four days. It is one unsliced DELETE (about servers x 24 x 90 rows, primary key leads with bucket), so it dispatches SingleShotStatement like the fleet-sweep children. A store without TimescaleDB has no rollup and no guard; the same constant bounds the table there.

What the 90 day horizon promises (review round 1). The ledger keeps 90 days, the rollup's retention horizon, and no read reaches past that horizon. The read router sends a window to the hourly rollup only up to 89 days back (RetentionTierRouter.HourlyMaxAge, the span less a one day margin). The prune does not claim to keep every hour the rollup holds. The rollup drops whole chunks, and only when the daily rollup covers them (RequireSuccessorDailyOf in TimescaleSupport.cs). So it can keep hours older than 90 days that the ledger already pruned, and no read uses them.

  • Live: EndToEnd_PurgeDeletesExpiredRowsAndKeepsFreshOnes_AgainstDevPostgres now seeds ledger buckets at 100 days, 80 days and 1 hour under its test server id. Only the 100-day bucket goes. The 80-day one survives although it is past raw's horizon and the 30-day and 60-day horizons, which is what fails a prune wired to the wrong horizon. Its cleanup removes the test ledger rows.
  • Unit: HourLedgerPrune_UsesTheSuccessorHourlyRollupsHorizon_NotRaws_AndDispatchesSingleShot pins that the successor hourly's policy is HourlyRetentionInterval, that the rollup outlives raw, the PruneSql text, and the call's shape (derived horizon, single shot, no raw span, after the collector loop).

Test plan for lanes 5 and 6

Build: 0 Warning(s), 0 Error(s). Live classes ran on a throwaway PostgreSQL 18.6 with TimescaleDB, timezone = 'UTC', pg_stat_statements preloaded. Each line is Total, Failed:

  • QueryStatsWriterCensusPins 21, 0
  • DarlingRetentionTests 48, 0 (live, none skipped)
  • QueryStatsHourLedgerTests 10, 0 (the first run failed TheViewerRolesProvisionedGrants_... with database "darling" does not exist: my rig lacked the default database. After creating it the class passed)
  • QueryStoreWriterSetPinTests 6, 0; MigrationDataMovingRungCensusPins 28, 0; LiveCleanupConversionRatchetTests 15, 0; StorageCommandTimeoutTests 24, 0; LivePostgresCollectionHygieneTests 11, 0; RepoFileAdoptionTests 2, 0; DocCommentHygiene 77, 0
  • Merge with lane E's runner change and a re-run of QueryStatsWriterCensusPins after it: lane E had not pushed when lane F pushed. The merge is done, and the census ran on the merged tree after review round 1: Total 21, Failed 0.
  • The prune against a store whose hourly rollup policy is armed (the live test runs timescaleAvailable: false; the prune does not depend on the flag)
  • The full suite: not run, CI runs it

For the coordinator: the squash body no longer says the writer, the census pin and the prune are the next lanes. It now says the collector writes the ledger, a census pin guards that one writer, and the sweep prunes the ledger. No CHANGELOG change: both pieces are internal until the guard PR.

Lane 2: the writer

Plan lane 2 of #4605. The collector now adds each query_stats batch's row count to the hour ledger. It does this inside the same transaction as the batch's COPY.

What it does

DarlingCollectorRunner.CopyBatchOnceAsync (DarlingCollectorRunner.cs:4251-4254, 4293, 4372-4374, 4437-4439) does this for a query_stats batch:

  • A batch is a ledger batch by the table its COPY writes (definition.TargetTable, line 4251), not by the collector's CLR type. A one-off definition aimed at query_stats is counted too, or fails loudly. IntervalPayloadIndex throws for a definition without the column, and the tally check catches any other overload.
  • It starts a tally on the attempt's own row writer before the first row.
  • After the COPY and the dimension flush, still inside the batch's transaction and before the commit, it runs QueryStatsHourLedger.UpsertSql. The call goes through the new QueryStatsHourLedgerWriter.AddBatchAsync (QueryStatsHourLedgerWriter.cs:94).
  • The arguments are server.ServerId, server.StorageName, the batch's stored collection_time and the tally. server.StorageName is the exact text the COPY writes into server_name.
  • The transaction condition gained a || ledgerBatch term. query_stats already diverts its text and plans, so this changes nothing today. It makes sure the upsert can never lack the transaction its atomicity depends on.

The count rule

  • The count is the number of rows written with a sample_interval_seconds other than 0. A NULL counts, the same as the rollup's IS DISTINCT FROM 0.
  • It is counted at the exact write. It is not worked out again.
  • PgCollectorRowWriter already numbers every payload value. It now tallies the integer it sends at one position (PgCollectorRowWriter.cs:100-125 and the int and int? overloads). It tallies only inside a row's payload, between BeginPayload (line 75) and EndPayload. The runner writes each row's prefix (collection_time, server_id) through the same overloads before the payload opens, and those writes are never counted (review round 1).
  • The position is looked up by name, sample_interval_seconds, in the same PayloadColumns list that the COPY's column list is built from (QueryStatsHourLedgerWriter.cs:55).
  • I chose the writer over a counter on CollectorContext. That leaves the shared Collectors project and Lite untouched.
  • A new writer is built for every attempt (DarlingCollectorRunner.cs:4228). So the tally starts from zero on every attempt by construction.
  • If a row writes the interval through some other overload, the tally misses it. The batch then fails before the COPY completes, instead of undercounting. The check is QueryStatsHourLedgerWriter.EnsureEveryRowTallied (QueryStatsHourLedgerWriter.cs:77), called at DarlingCollectorRunner.cs:4374, and it has a test of its own. It cannot trip today.
  • A count of 0 skips the upsert, with no round trip. AddBatchAsync returns before it builds a command. The SQL also keeps its own WHERE $4 > 0.
  • One batch is one hour. CopyBatchOnceAsync makes one storedCollectionTime (DarlingCollectorRunner.cs:4298) and writes that one value into every row (line 4359). It takes no per-row time. So one upsert is exact, and no per-bucket counting is needed.

The abort rule

The upsert has no savepoint and no catch. A fault in it throws out of the batch. The transaction rolls back, and the batch's raw rows are not stored. Raw and the ledger commit together or not at all.

The fault lands in the unstamped region of CopyBatchOnceAsync, which StoreWriteReattempt declines. So the collector's next cycle is the retry. The batch's sample is lost, as it already is for a dimension-flush fault.

Tests

QueryStatsHourLedgerWriterLiveTests has 4 facts. Each uses its own scratch store and drives the product's own WriteBatchAsync, with real rows and the real delta calculator.

  • Counts: a first-sighting batch (all restart rows) leaves no ledger row. A later batch with two measured rows and one new statement adds exactly 2. More batches in the same hour add to it, and the next hour gets its own bucket. After every batch the ledger equals the raw rows with an interval other than 0. The ledger's server_name is the raw rows' name, a named-instance spelling with a backslash.
  • Failed upsert: a trigger refuses the write. The batch throws, and raw and the ledger are both unchanged. Once the trigger is gone, the next batch counts normally.
  • Failed COPY: a CHECK constraint refuses one row after two measured rows were tallied. Raw and the ledger are unchanged. The next batch moves the ledger by exactly its own 2, not 4.
  • The tally on a real binary COPY: 0 is not counted. 60 and 5 are, and a NULL is. A long written at that position is not tallied, which is what the guard above catches. Starting a tally again resets it. The fact also writes each row's prefix first, on a fresh writer, in the runner's order. The prefix integers are not counted. Before review round 1 that case counted rows + 1.

QueryStatsHourLedgerWriterTests has 6 source and seam pins. Comments and strings are blanked first.

  • The upsert sits inside if (transaction is not null), after CompleteAsync and the dimension flush and before CommitAsync. No catch, savepoint or rollback sits between.
  • It is only for a batch whose COPY writes query_stats, by target table and not by the collector's type. It is keyed on server.StorageName and the writer's tally.
  • The tally starts on the attempt's own writer before the first row.
  • EnsureEveryRowTallied passes when the tally equals the rows written. It throws on an undercount, on an overcount and on a zero tally for a non-empty batch. A source pin requires the runner's call directly inside if (ledgerBatch), after the row loop and before the COPY completes.
  • A zero count returns before a command is built.
  • The interval position is the column the COPY names at that position.

Red first: I wrote the live facts before the writer. Against the tree without it, all 3 facts that existed then failed on the first ledger check (expected 2, got no row). The raw-row checks before it had passed.

Two items from the brief that the tests cover differently

  • A NULL interval adds 1. The real collector cannot write a NULL interval. ResolveWorkerDelta returns an int, and WritePayload writes that int. So the NULL rule is proven on the writer's seam with a real COPY. The SQL side, where the recount counts a NULL, is lane 1's RecountSql fact.
  • A retried batch counts once. No test induces a transport fault that the re-attempt gate accepts. The evidence is structural, and the pins hold it. The tally lives on a writer built for each attempt (one new PgCollectorRowWriter(), pinned). A start-phase fault has sent no row, so it tallied nothing. The failed-COPY fact shows that a failed attempt's tally of 2 does not reach the next batch.

Test plan

  • Build: 0 warnings, 0 errors
  • QueryStatsHourLedgerWriterLiveTests: Total 4, Failed 0 (merged tree, throwaway rig)
  • QueryStatsHourLedgerWriterTests: Total 6, Failed 0 (it was 5 before review round 1)
  • QueryStatsHourLedgerTests (lane 1): Total 10, Failed 0
  • QueryStatsWriterCensusPins (lane F, merged tree): Total 21, Failed 0
  • DarlingRetentionTests (lane F, merged tree): Total 48, Failed 0
  • Neighbouring classes on the merged tree, in one run, before review round 1: Total 476, Failed 0, Skipped 3. The classes are QueryStatsHourLedgerTests, QueryStoreWriteFencePinTests, StorageCommandTimeoutTests, CollectionSweepCommandTimeoutTests, every *CommandTimeoutTests, DarlingCollectorRunnerTests, StoreCopyPhaseTests, StoreCopyPhaseLivePostgresTests, DocCommentHygiene, PayloadDimensionTests, QueryStatsRowCoherentResetTests, QueryStatsUnknownCpuTests and LivePostgresCollectionHygieneTests. The 3 skips are DarlingCollectorRunnerTests end-to-end facts that need a live SQL Server.
  • Full Darling.Tests and Lite.Tests suites: not run here, because CI runs them. No Lite file and no shared Collectors file changed.

For the coordinator to double-check

  • The CHANGELOG entry above was written with lane 1 and said "The ledger tables are not read or written yet". It is rewritten: this PR writes and prunes the ledger, and nothing reads it until the guard PR (Custom Views: the query_stats count guard reads the hour ledger (part of #4605) #5290). It is still not user-visible, so the section stays None.
  • The fail-loud guard (QueryStatsHourLedgerWriter.EnsureEveryRowTallied, called at DarlingCollectorRunner.cs:4374) is my addition beyond the plan. Tell me if you prefer that the batch store with an undercount instead.
  • The ledger write costs one round trip for each query_stats batch with measured rows. It runs on the transaction that already exists.

Review round 1

Four review findings, fixed on this branch. Each is its own commit, and no migration changes.

  1. The writer census scans the service's whole project-reference closure, not just Darling/PerformanceMonitor.*. A second writer in one of the six root projects is now seen (QueryStatsWriterCensusPins, 9483f72).
  2. The ledger batch is keyed on the table the COPY writes (definition.TargetTable), not on definition is QueryStatsCollector. A one-off definition aimed at query_stats is counted or fails loudly (c0cd3aa).
  3. The fail-loud tally check is now QueryStatsHourLedgerWriter.EnsureEveryRowTallied, tested for equality, undercount, overcount and zero. The runner pin requires the call directly inside if (ledgerBatch) (36a9b89).
  4. The row writer's tally observes only inside the payload, so the prefix's server_id is never counted as the interval. A live case writes the prefix first on a fresh writer, as the runner does (b06d642).

Claims corrected in this body: the census is a text scan that fails on a literal write or a new COPY start. It cannot see SQL built at run time or writers inside the database. The retention prune keeps 90 days, the hourly rollup's retention horizon, and no read reaches past it. The earlier wording that the ledger keeps every hour the rollup holds was too strong. The live tally test now covers the runner's prefix-then-payload order. The changelog entry no longer says the tables are unwritten.

Results on the merged tree, on a throwaway PostgreSQL 18.6 with TimescaleDB in UTC and pg_stat_statements preloaded. Build: 0 Warning(s), 0 Error(s). Each line is Total, Failed:

  • QueryStatsWriterCensusPins 21, 0
  • QueryStatsHourLedgerWriterTests 6, 0
  • QueryStatsHourLedgerWriterLiveTests 4, 0
  • QueryStatsHourLedgerTests 10, 0
  • DarlingRetentionTests 48, 0
  • DocCommentHygieneTests 77, 0
  • MigrationLadderPins 4, 0
  • The full suite: not run, CI runs it

…t of #4605)

Migration V164 adds two plain collect tables: query_stats_hour_ledger (one row per hour, server id and server name with the count of non-restart query_stats rows) and its one-row state table, whose counted_since is the next whole hour after the rung ran. QueryStatsHourLedger holds the DDL the rung embeds plus UpsertSql (adds a batch's count to its hour, a count of 0 or less writes nothing) and RecountSql (sets a range of whole hours from raw, counting the rollup's population and writing raw's name spelling). The viewer's schema probe gains the V164 arm. The compose roles read the tables through the collect schema's existing blanket SELECT and default privileges, so no grant changes. The writer, the guard read, the writer census pin and the retention prune are the next lanes of #4605.

Part of #4605
…ceAsync, and prune the hour ledger at the hourly rollup's horizon (part of #4605)

The hour ledger counts rows at one write, so a census now fails if any
other code writes collect.query_stats, and the retention sweep deletes
ledger buckets below the successor hourly rollup's 90-day horizon (not
raw's four days).
… to the hour ledger inside its COPY transaction (part of #4605)

Plan lane 2 of #4605. CopyBatchOnceAsync now adds each query_stats batch's count of rows written with a
sample_interval_seconds other than 0 (a NULL counts) to collect.query_stats_hour_ledger, on the COPY's own transaction,
after the COPY and the dimension flush and before the commit. A fault in the upsert has no savepoint and no catch: it
aborts the batch, the transaction rolls back, and raw and the ledger commit together or not at all.

The count is the PgCollectorRowWriter's tally of the integer it sends at the interval column's payload position, which
QueryStatsHourLedgerWriter resolves by name from the same PayloadColumns list the COPY column list is built from. It is
taken at the write, never re-derived, and the writer is built per attempt, so every attempt starts from zero. A batch
whose rows did not all write the interval as an integer fails before the COPY completes rather than undercounting.
All rows of a batch carry the one stored collection_time, so a batch is one hour and one upsert. A count of 0 skips the
upsert. The ledger key uses server.StorageName, the exact spelling the COPY writes.

Tests: QueryStatsHourLedgerWriterLiveTests (four live facts: counts and restart rows through the real WriteBatchAsync, a
failed upsert, a failed COPY, and the writer's tally on a real binary COPY including NULL and 0) and
QueryStatsHourLedgerWriterTests (source pins on the upsert's position and the tally's start).
…he collector's CLR type (#4605)

A definition aimed at query_stats that is not the catalog's sealed QueryStatsCollector (a one-off import or backfill handed to WriteBatchAsync) went through the same COPY site with ledgerBatch false, so its rows were never counted and no census arm saw it. Keying on the COPY's target table makes that case counted, or a loud failure: IntervalPayloadIndex throws for a definition without the column, and the tally check catches any other overload. The source pin follows.
…t just Darling/PerformanceMonitor.* (#4605)

QueryStatsWriterCensusPins read only Darling/PerformanceMonitor.*, so it never
saw the six root projects compiled into the same service binary (Alerting,
Analysis, Collectors, Common, Notifications, PlanAnalysis). A second writer of
collect.query_stats added to any of them would have left every census fact
green, and the hour ledger's count guard could then have served a short rollup
as exact.

LoadSources now walks the ProjectReference graph from every
Darling/PerformanceMonitor.* project and reads the non-test sources of each
project it reaches. File keys keep the ProjectName/relative/path.cs form, so
the runner file and the fixed-site register still match. The known-files list
gains PerformanceMonitor.Collectors/QueryStatsCollector.cs, which is outside
Darling/, to show the walk left the Darling tree. The class doc now says what
the scan covers and lists the blind spots that remain (run-time-built DML, SQL
outside .cs files, database-side writers, a COPY reached by method group).
…d, with a test that fails without it (#4605)

The check that every row wrote the interval through a tallied overload lived inline in CopyBatchOnceAsync, and its only pin was a substring between the row loop and CompleteAsync. A mutation that kept the text but never threw (an added && false, or a log line in place of the throw) left every test green and the silent undercount back. The comparison is now QueryStatsHourLedgerWriter.EnsureEveryRowTallied, unit-tested for equality, undercount, overcount and zero, and the runner source pin looks for the call directly inside if (ledgerBatch).
… so a prefix integer is never counted (#4605)

The runner writes each row's prefix (collection_time, server_id) through the same overloads before BeginPayload resets the position. On a fresh writer the first row's server_id, a Value(int), landed at payload position 1 or 2, so a watched column at either position would have been tallied once too often and the batch refused by the tally check. The writer now sets a flag in BeginPayload, clears it in EndPayload, and Observe returns unless the flag is set. A live case writes the prefix first on a fresh writer, as the runner does, and counts rows (it counted rows + 1 before the change).
@erikdarlingdata erikdarlingdata added the merge-train In a merge train: its CI ran on the train PR label Oct 5, 2026
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review October 5, 2026 22:54
@erikdarlingdata
erikdarlingdata merged commit 9a089ff into dev Oct 5, 2026
13 of 17 checks passed
@erikdarlingdata
erikdarlingdata deleted the feat/4605-hour-ledger branch October 5, 2026 22:54
erikdarlingdata added a commit that referenced this pull request Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-train In a merge train: its CI ran on the train PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant