Repository navigation
Darling store: an hourly row-count ledger for query_stats (V164) (part of #4605) - #5289
Merged
Merged
Conversation
…t of #4605) Migration V164 adds two plain collect tables: query_stats_hour_ledger (one row per hour, server id and server name with the count of non-restart query_stats rows) and its one-row state table, whose counted_since is the next whole hour after the rung ran. QueryStatsHourLedger holds the DDL the rung embeds plus UpsertSql (adds a batch's count to its hour, a count of 0 or less writes nothing) and RecountSql (sets a range of whole hours from raw, counting the rollup's population and writing raw's name spelling). The viewer's schema probe gains the V164 arm. The compose roles read the tables through the collect schema's existing blanket SELECT and default privileges, so no grant changes. The writer, the guard read, the writer census pin and the retention prune are the next lanes of #4605. Part of #4605
9 of 10 tasks
…ceAsync, and prune the hour ledger at the hourly rollup's horizon (part of #4605) The hour ledger counts rows at one write, so a census now fails if any other code writes collect.query_stats, and the retention sweep deletes ledger buckets below the successor hourly rollup's 90-day horizon (not raw's four days).
… to the hour ledger inside its COPY transaction (part of #4605) Plan lane 2 of #4605. CopyBatchOnceAsync now adds each query_stats batch's count of rows written with a sample_interval_seconds other than 0 (a NULL counts) to collect.query_stats_hour_ledger, on the COPY's own transaction, after the COPY and the dimension flush and before the commit. A fault in the upsert has no savepoint and no catch: it aborts the batch, the transaction rolls back, and raw and the ledger commit together or not at all. The count is the PgCollectorRowWriter's tally of the integer it sends at the interval column's payload position, which QueryStatsHourLedgerWriter resolves by name from the same PayloadColumns list the COPY column list is built from. It is taken at the write, never re-derived, and the writer is built per attempt, so every attempt starts from zero. A batch whose rows did not all write the interval as an integer fails before the COPY completes rather than undercounting. All rows of a batch carry the one stored collection_time, so a batch is one hour and one upsert. A count of 0 skips the upsert. The ledger key uses server.StorageName, the exact spelling the COPY writes. Tests: QueryStatsHourLedgerWriterLiveTests (four live facts: counts and restart rows through the real WriteBatchAsync, a failed upsert, a failed COPY, and the writer's tally on a real binary COPY including NULL and 0) and QueryStatsHourLedgerWriterTests (source pins on the upsert's position and the tally's start).
…he collector's CLR type (#4605) A definition aimed at query_stats that is not the catalog's sealed QueryStatsCollector (a one-off import or backfill handed to WriteBatchAsync) went through the same COPY site with ledgerBatch false, so its rows were never counted and no census arm saw it. Keying on the COPY's target table makes that case counted, or a loud failure: IntervalPayloadIndex throws for a definition without the column, and the tally check catches any other overload. The source pin follows.
…t just Darling/PerformanceMonitor.* (#4605) QueryStatsWriterCensusPins read only Darling/PerformanceMonitor.*, so it never saw the six root projects compiled into the same service binary (Alerting, Analysis, Collectors, Common, Notifications, PlanAnalysis). A second writer of collect.query_stats added to any of them would have left every census fact green, and the hour ledger's count guard could then have served a short rollup as exact. LoadSources now walks the ProjectReference graph from every Darling/PerformanceMonitor.* project and reads the non-test sources of each project it reaches. File keys keep the ProjectName/relative/path.cs form, so the runner file and the fixed-site register still match. The known-files list gains PerformanceMonitor.Collectors/QueryStatsCollector.cs, which is outside Darling/, to show the walk left the Darling tree. The class doc now says what the scan covers and lists the blind spots that remain (run-time-built DML, SQL outside .cs files, database-side writers, a COPY reached by method group).
…d, with a test that fails without it (#4605) The check that every row wrote the interval through a tallied overload lived inline in CopyBatchOnceAsync, and its only pin was a substring between the row loop and CompleteAsync. A mutation that kept the text but never threw (an added && false, or a log line in place of the throw) left every test green and the silent undercount back. The comparison is now QueryStatsHourLedgerWriter.EnsureEveryRowTallied, unit-tested for equality, undercount, overcount and zero, and the runner source pin looks for the call directly inside if (ledgerBatch).
… so a prefix integer is never counted (#4605) The runner writes each row's prefix (collection_time, server_id) through the same overloads before BeginPayload resets the position. On a fresh writer the first row's server_id, a Value(int), landed at payload position 1 or 2, so a watched column at either position would have been tallied once too often and the batch refused by the tally check. The writer now sets a flag in BeginPayload, clears it in EndPayload, and Observe returns unless the flag is set. A live case writes the prefix first on a fresh writer, as the runner does, and counts rows (it counted rows + 1 before the change).
…losure) into r/5289-fixC (#4605)
erikdarlingdata
added a commit
that referenced
this pull request
Oct 5, 2026
erikdarlingdata
marked this pull request as ready for review
October 5, 2026 22:54
This was referenced Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is plan lane 1 of #4605 only: the V164 rung and the storage class. Other lanes add the writer, the writer census pin and the retention prune on this same branch, and the guard read and route test seeding in a second PR.
Why
The hourly-edges read may take its middle hours from the hourly rollup only when the rollup holds every raw row those hours have. Today the count guard proves that by counting raw
collect.query_statsrows per (server, hour), a scan of millions of rows that runs past the panel deadline, so the panel falls back to raw. Option A of the plan (approved) keeps that count at write time instead: one ledger row per (server, hour), written in the same transaction as the collector's COPY. The guard then reads a few thousand ledger rows, and its cost follows the number of (server, hour) pairs and not the number of raw rows. This lane adds the storage for it. Lanes 2 and 6 below write and prune it, nothing reads it until the guard PR (#5290), and a user sees no change.What changes
Darling/PerformanceMonitor.Darling.Storage/QueryStatsHourLedger.cs(new):CreateSql(line 62): both tables and the state row. The rung embeds it (the V123 precedent), and the rung test pins its shape so an edit to a shipped rung fails there.UpsertSql(line 93):$1server id,$2the storage name,$3the batch's collection time (truncated to its hour),$4the count. Adds to an existing hour. A count of 0 or less writes nothing, because the rollup has no row for an hour with no counted rows and a ledger row of 0 would not equal that.RecountSql(line 112):$1start,$2end exclusive. Sets (does not add) the ledger from raw for those hours, deletes ledger rows whose raw rows are gone, and returns (rows written, rows deleted). It counts the rollup's population (sample_interval_seconds IS DISTINCT FROM 0, so restart rows are skipped and NULL intervals are counted), writesserver_nameas raw holds it (what the COPY wrote), and widens a ragged range to whole hours so a partial hour can never be written. It does not touch the state row. It usesdate_trunc('hour', ...), so it also runs on a store without TimescaleDB.PgMigrations.cs:V164Sql(line 656) andnew Migration(164, "query-stats-hour-ledger", V164Sql)(line 859).StorageVersion.SchemaVersionis 164.collect, not collectors):query_stats_hour_ledger(server_id, server_name, bucket, n) andquery_stats_hour_ledger_state(one row,id = 1,counted_since). The rung setscounted_sinceto the next whole hour withON CONFLICT DO NOTHING, so a re-run keeps an earlier value. Hours before it are "uncounted" and the guard will stay on raw for them. Because the writer lands on this same branch, one deploy starts the counting.ViewerDataService.cs: the schema probe's V164 sentinel (line 1090) and itsMapProbedSchemaVersionarm (line 1329), so a store at V164 is not shown an upgrade banner.(bucket, server_id, server_name), not(server_id, server_name, bucket). The guard reads a range of hours across every server and the retention prune deletes below one hour, so a key that leads with the hour makes both an index range scan with no second index. The plan's acceptance step (a 24 hour fleet read under 2 k buffers) cannot be met by a scan of the whole ledger. The upsert's conflict target names all three columns, and PostgreSQL matches a conflict target to a unique index without regard to column order.QueryStatsHourLedgerTests.cs(new);StoreStatementHistoryRungTests(its probe test no longer assumes V163 is the top arm, as V163 did for V162);CaggGroupIndexDropLiveTests(drops the V164 tables when it simulates an older store);StoreSelfMetricsTests(the plain-table census lists both tables with a reason).Grants evidence (no new grant, no provisioning change)
New
collecttables get SELECT for admin, viewer and mcp from two things that provisioning re-asserts at every start, after migration (DarlingManagedRoles.BuildProvisioningSql, sections 2, 4 and 6):GRANT SELECT ON ALL TABLES IN SCHEMA collect TO admin, viewer;andGRANT SELECT ON ALL TABLES IN SCHEMA collect TO mcp;ALTER DEFAULT PRIVILEGES FOR ROLE <owner> IN SCHEMA collect GRANT SELECT ON TABLES TO admin, viewer;and the same for mcp (Compose and bring-your-own: the web dashboard and MCP server connect as the store owner, not the least-privilege viewer/mcp roles the README describes #3914).BYO stores get the same from
Darling/tools/provision-roles.sql(theON ALL TABLESform at line 191 and 194, the default privileges at 355 and 370).collect.module_mapandmodule_map_statehave no grant of their own either; they rely on this path, and so do the V162, V163 and V123 tables. The collector writes as the owner. No role setting changes, so nothing moves into a migration.Proved live:
TheViewerRolesProvisionedGrants_ReadBothTables_AndWriteNeithercreates a no-superuser role, replays exactly the viewer's provisioned GRANT/REVOKE statements (ViewerGrantReplay), then reads both ledger tables as that role and gets SQLSTATE 42501 on an INSERT. That covers the compose read in lane 3. I did not exercise the mcp role directly; its statements are the same lines and are pinned verbatim byDarlingManagedRolesTests.Test plan
Build:
0 Warning(s),0 Error(s). Run on a throwaway PostgreSQL 18.6 with TimescaleDB, on a fresh scratch database for each live test (the rung ran on a fresh store inMigratingAFreshStore_...). These are lane 1's numbers at its own commit. Later lanes' numbers are in their sections. The merged-tree numbers are under Review round 1. Each line is Total, Failed:QueryStatsHourLedgerTests10, 0 (schema pins; upsert adds to an existing hour, starts a new one, ignores 0 and negative; recount from raw skips restart rows, counts NULL intervals, keeps the raw name spelling, sets rather than adds, widens a ragged range, deletes a stale row, leaves rows outside the range alone; the state row is the next whole hour and survives a re-run; the viewer grants proof; the probe arm)MigrationLadderPins4, 0MigrationDataMovingRungCensusPins28, 0 (no declaration needed: the tables are empty and the one INSERT is a VALUES row into a table the same rung creates)MigrationUpgradeLadderLiveTests6, 0 (the generated-schema ladder replay over released stores)StoreSelfMetricsTests37, 0 (the plain-table census; it needed the two entries)StoreStatementHistoryRungTests6, 0;SlowReadsRungTests4, 0;QueryStoreTopDailyRungTests11, 0;CollectorRunTimeRungTests19, 0;RawChunkIntervalRungHistoryRungTests3, 0ViewerSchemaVersionGateTests34, 0;CaggGroupIndexDropLiveTests5, 0DarlingRetentionTests47, 0 (retention walks the collector catalog, and neither table is a collector)DarlingManagedRolesTests27, 0;ProvisionRolesAclDriftTests9, 0;ServerTagViewerRoleLiveTests2, 0LiveCleanupBatchTests5, 0 (the helper keeps no list of collect tables; the lane-4 route tests add ledger cleanup)DiagnosticsBundleVerbTests21, 0;DiagnosticsBundleSizeTests4, 0 (the bundle reads named tables and has no table inventory, and no section reads the ledger)DocCommentHygieneTests77, 0;PgSchemaGeneratorTests25, 0;StoreObjectConvergenceTests7, 0;DarlingObservabilityTests11, 0;DarlingModuleMapTests5, 0Lanes after lane 1
Status: lanes 2, 5 and 6 are on this branch (their sections are below). Lanes 3 and 4 are the second PR (#5290).
DarlingCollectorRunner.CopyBatchOnceAsyncrunsUpsertSqlafter the COPY, inside its transaction, with the non-restart row count of the batch counted at the write (reset per attempt) andserver.StorageName. A fault in the upsert aborts the batch.QueryStatsWriterCensusPins, expected writer set exactlyCopyBatchOnceAsync.counted_sincereturns -1) and lane 4 (route test seeding withRecountSql, ledger cleanup,LiveCleanupBatch).For the coordinator to double-check
UpsertSqlignoring a count of 0 or less, in the SQL as well as in the writer. If you want the writer to be the only gate, drop theWHERE $4::bigint > 0line and its pin.RecountSqlon an hour that is still being written can overwrite a batch that commits during the statement. That leaves the ledger below the rollup, which fails the guard safe and never passes it wrongly, and it is stated on the constant. Tests and a backfill use it on closed hours.QueryStatsHourLedger.cs,V164Sqland the census entry say the retention prune and the writer exist. They do: lanes 2 and 6 are on this branch.SECTION: None
ENTRY: - No user-visible change in this PR ([#5289]) - This PR writes and prunes the ledger tables. Nothing reads them until the guard PR (#5290), so nothing a user sees changes.
Part of #4605
Lanes 5 and 6: the writer census and retention
Lane F's commit on this branch:
8737f74a3. It touchesQueryStatsWriterCensusPins.cs(new),QueryStatsHourLedger.cs(one constant),DarlingRetention.cs(one block) andDarlingRetentionTests.cs. Nothing lane E owns is edited.Lane 5, the census (
Darling/Darling.Tests/QueryStatsWriterCensusPins.cs). The ledger counts rows at one write, so a second writer would put rows into raw and the hourly rollup that no ledger entry accounts for. If those rows are late, the guard cannot see the hole. The pin fails the test run on a literal write toquery_statsin the service's sources. It also fails on a COPY start missing from its register, and the message says why. It is a text scan with blind spots, listed below, and it has three arms:.cssource in the service's project-reference closure forINSERT INTO,COPY,UPDATEorMERGE INTOaimed atquery_stats. The schema and quotes are optional, and the match is word-bounded, soquery_stats_hour_ledgerand the otherquery_stats_*relations never match. The closure is everyDarling/PerformanceMonitor.*project and every project those reference, transitively, so the six root libraries compiled into the same service are included. Comments are stripped and string literals kept, because the SQL lives in them.BeginBinaryImportAsync,BeginTextImport,BeginRawBinaryCopy), because the real COPY text is built at run time (CopyCommandForappendsschema.TargetTable), so the literal scan cannot see it. The register holds the runner's one site (generic definition, and it must sit insideCopyBatchOnceAsync, declared once) and four RDS ingestors, each over a fixedPg*Collector.Instancewith its table named (pg_cpu_utilization,pg_deadlocks,pg_log_events,pg_plan_capture) and compared with the definition's realTargetTable. A fifth pin:QueryStatsCollectoris the only catalog definition whose table isquery_stats.DELETE FROMandTRUNCATE) on the table.PerformanceMonitor.Collectors/QueryStatsCollector.cswas read. That file sits outsideDarling/, so it shows the walk left the Darling tree. Each arm has positive and negative controls on fabricated sources (TheLiteralScan_*,TheCopyScan_*,TheMigrationArm_*).StringBuilderor a concatenation across literals). It cannot see SQL in a file that is not.cs, and none exists in these projects today. It cannot see a writer inside the database, such as a trigger, a rule, a function or a scheduled job. It cannot see a COPY reached through a method group. Projects outsideDarling/PerformanceMonitor.*are no longer a blind spot (review round 1). Dynamic SQL inside a rung is already closed byMigrationDataMovingRungCensusPins.TheLadderStillContainsNoDynamicSql.RED proof: I added
Darling/PerformanceMonitor.Darling.Service/ZzPlantedQueryStatsWriter.csholding"INSERT INTO collect.query_stats (...) VALUES (...)"and re-ran the built class.TheWritersOfQueryStats_AreExactly_CopyBatchOnceAsyncfailed withFound: {CopyBatchOnceAsync, PerformanceMonitor.Darling.Service/ZzPlantedQueryStatsWriter.cs:5 (INSERT INTO collect.query_stats)}followed by the ledger explanation. I removed the plant (it was never committed) and the class went back to 21 passed. A first version of the pin also failed legitimately on its own bug (it compared the RDS definition expression with the identifierdefinition, and flagged the four RDS sites as unlisted); the fix is in the commit.Lane 6, retention (
DarlingRetention.cs,QueryStatsHourLedger.PruneSql). A new failure-isolated step in the sweep, after the per-collector loop and outside everytimescaleAvailablebranch, deletescollect.query_stats_hour_ledgerrows withbucket < date_trunc('hour', now - TimescaleSupport.HourlyRetentionSpan). That is the successor hourly rollup's horizon, 90 days, derived from the same constant the rollup's own retention policy and the read router use. It is not raw's four days: the guard compares the ledger with the rollup, and a ledger pruned at raw's horizon would be missing hours the rollup still holds, so the guard would fail every window past four days. It is one unslicedDELETE(about servers x 24 x 90 rows, primary key leads withbucket), so it dispatchesSingleShotStatementlike the fleet-sweep children. A store without TimescaleDB has no rollup and no guard; the same constant bounds the table there.What the 90 day horizon promises (review round 1). The ledger keeps 90 days, the rollup's retention horizon, and no read reaches past that horizon. The read router sends a window to the hourly rollup only up to 89 days back (
RetentionTierRouter.HourlyMaxAge, the span less a one day margin). The prune does not claim to keep every hour the rollup holds. The rollup drops whole chunks, and only when the daily rollup covers them (RequireSuccessorDailyOfinTimescaleSupport.cs). So it can keep hours older than 90 days that the ledger already pruned, and no read uses them.EndToEnd_PurgeDeletesExpiredRowsAndKeepsFreshOnes_AgainstDevPostgresnow seeds ledger buckets at 100 days, 80 days and 1 hour under its test server id. Only the 100-day bucket goes. The 80-day one survives although it is past raw's horizon and the 30-day and 60-day horizons, which is what fails a prune wired to the wrong horizon. Its cleanup removes the test ledger rows.HourLedgerPrune_UsesTheSuccessorHourlyRollupsHorizon_NotRaws_AndDispatchesSingleShotpins that the successor hourly's policy isHourlyRetentionInterval, that the rollup outlives raw, thePruneSqltext, and the call's shape (derived horizon, single shot, no raw span, after the collector loop).Test plan for lanes 5 and 6
Build:
0 Warning(s),0 Error(s). Live classes ran on a throwaway PostgreSQL 18.6 with TimescaleDB,timezone = 'UTC',pg_stat_statementspreloaded. Each line is Total, Failed:QueryStatsWriterCensusPins21, 0DarlingRetentionTests48, 0 (live, none skipped)QueryStatsHourLedgerTests10, 0 (the first run failedTheViewerRolesProvisionedGrants_...withdatabase "darling" does not exist: my rig lacked the default database. After creating it the class passed)QueryStoreWriterSetPinTests6, 0;MigrationDataMovingRungCensusPins28, 0;LiveCleanupConversionRatchetTests15, 0;StorageCommandTimeoutTests24, 0;LivePostgresCollectionHygieneTests11, 0;RepoFileAdoptionTests2, 0;DocCommentHygiene77, 0QueryStatsWriterCensusPinsafter it: lane E had not pushed when lane F pushed. The merge is done, and the census ran on the merged tree after review round 1: Total 21, Failed 0.timescaleAvailable: false; the prune does not depend on the flag)For the coordinator: the squash body no longer says the writer, the census pin and the prune are the next lanes. It now says the collector writes the ledger, a census pin guards that one writer, and the sweep prunes the ledger. No CHANGELOG change: both pieces are internal until the guard PR.
Lane 2: the writer
Plan lane 2 of #4605. The collector now adds each query_stats batch's row count to the hour ledger. It does this inside the same transaction as the batch's COPY.
What it does
DarlingCollectorRunner.CopyBatchOnceAsync(DarlingCollectorRunner.cs:4251-4254, 4293, 4372-4374, 4437-4439) does this for a query_stats batch:definition.TargetTable, line 4251), not by the collector's CLR type. A one-off definition aimed atquery_statsis counted too, or fails loudly.IntervalPayloadIndexthrows for a definition without the column, and the tally check catches any other overload.QueryStatsHourLedger.UpsertSql. The call goes through the newQueryStatsHourLedgerWriter.AddBatchAsync(QueryStatsHourLedgerWriter.cs:94).server.ServerId,server.StorageName, the batch's storedcollection_timeand the tally.server.StorageNameis the exact text the COPY writes intoserver_name.|| ledgerBatchterm. query_stats already diverts its text and plans, so this changes nothing today. It makes sure the upsert can never lack the transaction its atomicity depends on.The count rule
sample_interval_secondsother than 0. A NULL counts, the same as the rollup'sIS DISTINCT FROM 0.PgCollectorRowWriteralready numbers every payload value. It now tallies the integer it sends at one position (PgCollectorRowWriter.cs:100-125 and theintandint?overloads). It tallies only inside a row's payload, betweenBeginPayload(line 75) andEndPayload. The runner writes each row's prefix (collection_time, server_id) through the same overloads before the payload opens, and those writes are never counted (review round 1).sample_interval_seconds, in the samePayloadColumnslist that the COPY's column list is built from (QueryStatsHourLedgerWriter.cs:55).CollectorContext. That leaves the shared Collectors project and Lite untouched.QueryStatsHourLedgerWriter.EnsureEveryRowTallied(QueryStatsHourLedgerWriter.cs:77), called at DarlingCollectorRunner.cs:4374, and it has a test of its own. It cannot trip today.AddBatchAsyncreturns before it builds a command. The SQL also keeps its ownWHERE $4 > 0.CopyBatchOnceAsyncmakes onestoredCollectionTime(DarlingCollectorRunner.cs:4298) and writes that one value into every row (line 4359). It takes no per-row time. So one upsert is exact, and no per-bucket counting is needed.The abort rule
The upsert has no savepoint and no catch. A fault in it throws out of the batch. The transaction rolls back, and the batch's raw rows are not stored. Raw and the ledger commit together or not at all.
The fault lands in the unstamped region of
CopyBatchOnceAsync, whichStoreWriteReattemptdeclines. So the collector's next cycle is the retry. The batch's sample is lost, as it already is for a dimension-flush fault.Tests
QueryStatsHourLedgerWriterLiveTestshas 4 facts. Each uses its own scratch store and drives the product's ownWriteBatchAsync, with real rows and the real delta calculator.server_nameis the raw rows' name, a named-instance spelling with a backslash.longwritten at that position is not tallied, which is what the guard above catches. Starting a tally again resets it. The fact also writes each row's prefix first, on a fresh writer, in the runner's order. The prefix integers are not counted. Before review round 1 that case counted rows + 1.QueryStatsHourLedgerWriterTestshas 6 source and seam pins. Comments and strings are blanked first.if (transaction is not null), afterCompleteAsyncand the dimension flush and beforeCommitAsync. No catch, savepoint or rollback sits between.query_stats, by target table and not by the collector's type. It is keyed onserver.StorageNameand the writer's tally.EnsureEveryRowTalliedpasses when the tally equals the rows written. It throws on an undercount, on an overcount and on a zero tally for a non-empty batch. A source pin requires the runner's call directly insideif (ledgerBatch), after the row loop and before the COPY completes.Red first: I wrote the live facts before the writer. Against the tree without it, all 3 facts that existed then failed on the first ledger check (expected 2, got no row). The raw-row checks before it had passed.
Two items from the brief that the tests cover differently
ResolveWorkerDeltareturns anint, andWritePayloadwrites that int. So the NULL rule is proven on the writer's seam with a real COPY. The SQL side, where the recount counts a NULL, is lane 1'sRecountSqlfact.new PgCollectorRowWriter(), pinned). A start-phase fault has sent no row, so it tallied nothing. The failed-COPY fact shows that a failed attempt's tally of 2 does not reach the next batch.Test plan
*CommandTimeoutTests, DarlingCollectorRunnerTests, StoreCopyPhaseTests, StoreCopyPhaseLivePostgresTests, DocCommentHygiene, PayloadDimensionTests, QueryStatsRowCoherentResetTests, QueryStatsUnknownCpuTests and LivePostgresCollectionHygieneTests. The 3 skips areDarlingCollectorRunnerTestsend-to-end facts that need a live SQL Server.For the coordinator to double-check
QueryStatsHourLedgerWriter.EnsureEveryRowTallied, called at DarlingCollectorRunner.cs:4374) is my addition beyond the plan. Tell me if you prefer that the batch store with an undercount instead.Review round 1
Four review findings, fixed on this branch. Each is its own commit, and no migration changes.
Darling/PerformanceMonitor.*. A second writer in one of the six root projects is now seen (QueryStatsWriterCensusPins, 9483f72).definition.TargetTable), not ondefinition is QueryStatsCollector. A one-off definition aimed atquery_statsis counted or fails loudly (c0cd3aa).QueryStatsHourLedgerWriter.EnsureEveryRowTallied, tested for equality, undercount, overcount and zero. The runner pin requires the call directly insideif (ledgerBatch)(36a9b89).server_idis never counted as the interval. A live case writes the prefix first on a fresh writer, as the runner does (b06d642).Claims corrected in this body: the census is a text scan that fails on a literal write or a new COPY start. It cannot see SQL built at run time or writers inside the database. The retention prune keeps 90 days, the hourly rollup's retention horizon, and no read reaches past it. The earlier wording that the ledger keeps every hour the rollup holds was too strong. The live tally test now covers the runner's prefix-then-payload order. The changelog entry no longer says the tables are unwritten.
Results on the merged tree, on a throwaway PostgreSQL 18.6 with TimescaleDB in UTC and
pg_stat_statementspreloaded. Build:0 Warning(s),0 Error(s). Each line is Total, Failed:QueryStatsWriterCensusPins21, 0QueryStatsHourLedgerWriterTests6, 0QueryStatsHourLedgerWriterLiveTests4, 0QueryStatsHourLedgerTests10, 0DarlingRetentionTests48, 0DocCommentHygieneTests77, 0MigrationLadderPins4, 0