Build a BRIN index on query_store_interval_wide (collection_time) in the background

[erikdarlingdata]

Refs #4605

What

A Custom Views Query Store panel over 12 hours or more reads collect.query_store_interval_wide on collection_time. Nothing serves that column, so the read scans the whole table. On a large production monitoring store (28 GB, 3.1 of 9 days held) a 24-hour fleet panel read 3.18M blocks, 98.8 s cold.

This PR adds QueryStoreIntervalWideBrinIndex (new file in the Storage project) and schedules it from the worker. It builds collect.ix_query_store_interval_wide_collection_time_brin (USING brin (collection_time) WITH (autosummarize = on)) with CREATE INDEX CONCURRENTLY. It changes no read, no migration rung and no existing file except a launch and a drain in DarlingWorker.cs.

Measured on that store with a temporary copy of the index (random_page_cost 1.1, warm): 24 h 4.3 s (1.14M blocks), 12 h 1.6 s, 6 h 0.8 s, 48 h 9.2 s. Cold, the 24 h read is about 8.7 GiB, roughly 37 s at the volume's throughput cap, against 98.8 s cold for the full scan. The build took 108 s on the 25 GB heap, the index was 856 kB, and no writes were blocked.

These figures are from before the table reached its retention horizon (the store held 3.1 of its 9 days). Once the 9-day purge starts, freed pages near the start of the heap are reused by new rows, and a BRIN range's summary only ever widens. Windows that end at the present, which is what the panels read, should stay selective. A window several days back will match more ranges, and the planner's collection_time correlation falls, which can move some windows back to a sequential scan for part of the purge cycle. Results stay exact, no read gets slower than without the index, and the writer's cost is unchanged. If the steady state needs it, the remedy is a periodic REINDEX INDEX CONCURRENTLY or re-summarizing the recycled ranges; this PR does neither.

Most of the read gain needs a low random_page_cost. On that store, at the default of 4, a 12-hour window still sequential-scans with the index present, while a 24-hour window already uses the index. This threshold is a production measurement: on the 150k-row test fixture the planner picks the BRIN at 4 as well, so pin (c) proves the index is usable, not where the cost threshold sits. The 12-hour and shorter windows gain at about 1.1 or lower: on the managed store once the random_page_cost change lands (#4863), or on a bring-your-own store whose operator sets it.

Design choices

• BRIN, not btree. The writer's upsert rewrites collection_time on every update, and 99.9% of this table's updates are HOT today. A btree on collection_time would make each one non-HOT (the The Query Store liveness touch costs ~13% of the Query Store–ingesting store's WAL (~33 GB/day) at its 6-hour guard: every touch is a non-HOT update #4250 class). PostgreSQL 16 and later keep an update HOT when only summarizing (BRIN) columns change. On that store, in one rolled-back transaction: updating collection_time and execution_count on 20 rows was 20/20 HOT, and the control, first_execution_time (btree-indexed), was 0/20. Physical correlation of collection_time is 0.9998.
• PostgreSQL 16 floor. Decide(serverVersionNum, tableIsHypertable) is a pure function. Below 160000 the ensure skips the build and logs at Information: the version and "a BRIN index on collection_time would make the upsert non-HOT below PG 16". A bring-your-own store on 14 or 15 keeps today's plans and HOT rate.
• Hypertable guard. The table is a plain heap today, and TimescaleDB refuses CONCURRENTLY on a hypertable. If timescaledb_information.hypertables lists the table, the ensure skips with a Warning and never retries. The view is read only after to_regclass('timescaledb_information.hypertables') says it exists, so a store without TimescaleDB never parses it.
• No migration rung. A rung runs in one transaction (CONCURRENTLY fails with 25001 there), and a plain in-rung build would block the writer for the whole heap read. An applied rung also never changes. So the ensure is idempotent and runs each start. A fresh store builds an empty table's index instantly; no table-creation change is needed. The schema-equivalence tests (fresh vs migrated index sets) are unaffected because no migration changes; those live in Lite.Tests, which cannot run on macOS, and nothing in this diff touches Lite or PgMigrations.cs.
• INVALID leftovers. It reads pg_index.indisvalid first. Valid: nothing (Debug). Invalid (an interrupted build): DROP INDEX CONCURRENTLY IF EXISTS, a Warning, then build. Plain IF NOT EXISTS would keep an invalid leftover.
• Own connection, generous deadline. The worker opens a connection from its data source (autocommit, no transaction). Build and drop use a 7200 s command timeout; catalog reads use 30 s. The cancellation token stops it at shutdown.
• 20-minute start delay (a default I chose; easy to change): after an install or restart a big store's volume sits at its IOPS cap for about 15 minutes (cold cache, migrations, retention purge, aggregate refresh), and a full-heap read on top slows all of it. The delay lives only in RunDelayedAsync (cancellable); tests call EnsureAsync directly. One attempt per start; a failure is a Warning and is retried at the next start. The task is launched after migrations, is never awaited on the startup path, and is drained only at shutdown.
• Build duration is logged at Information (name, seconds, "valid").

Pins

Live pins ran on a local TimescaleDB 2.30.1 / PG 18 rig (QueryStoreIntervalWideBrinIndexLiveTests, 5 tests), the unit pins in QueryStoreIntervalWideBrinIndexTests (15 cases). Each mutation is a temporary code change, reverted.

• (a) exists, valid, brin, autosummarize, second call no-op (same oid). RED with "ensure returns before building": FAIL. GREEN: pass.
• (b) HOT. Upsert-shaped SET (collection_time, last_execution_time, execution_count, avg_duration_us) on 200 rows: 200/200 HOT with the BRIN. Control (first_execution_time update): 0 HOT. Mutation, swap the BRIN for a btree on collection_time: 0 HOT, the assert fails. RED with the ensure building a btree ((collection_time) instead of brin): FAIL. GREEN: pass.
• (c) plan. 150k rows over 9 days, ANALYZE, SET LOCAL random_page_cost = 1.1, EXPLAIN (FORMAT JSON) of a 12 h collection_time window. Before the ensure: Seq Scan and no Bitmap Index Scan (asserted; this is the dev behaviour). After: Bitmap Heap Scan over Bitmap Index Scan, no Seq Scan. This pins the planner's use of the index; the product reads' end-to-end effect is the before/after on Custom Views panels over a recent window scan raw hypertables with no bound; a Query Store panel spilled ~23 GB and tore collector writes #4605.
• (d) INVALID leftover. UPDATE pg_index SET indisvalid = false fakes an interrupted build (the rig and CI roles are superuser). The ensure rebuilds: valid, brin definition, new oid. Mutation, skip the validity read: FAIL. GREEN: pass.
• (e) source and decision pins. Decide: build at 160000 and 180006; skip at 159999, 150013, 140000; hypertable skip at 160000 and 180006; version wins over hypertable at 150000; reasons named. SQL contains CONCURRENTLY, IF NOT EXISTS, USING brin (collection_time), autosummarize = on. Source pin: the hypertable view is read only behind hasHypertableView, and the skip logs a Warning. Worker pin: launch after PgMigrations.MigrateAsync(migrateConnection, one await at shutdown after the collection loop. Migration pin: no index in PgMigrations.cs on the wide table includes collection_time.
• (f) the whole upsert SET list stays HOT-safe. NoNonBrinIndex_CoversAColumnTheUpsertSets_SoTheUpsertStaysHotForItsWholeSetList parses the DO UPDATE SET columns from QueryStoreIntervalWide.UpsertSql itself (at least 50 names, collection_time and execution_count among them, no identity column) and asserts that no non-BRIN index on the table covers any of them: key and INCLUDE columns from indkey, expression and predicate columns from pg_depend. In-test probes show it sees what it must: a btree on execution_count is reported, an INCLUDE (query_hash) WHERE plan_type IS NOT NULL index reports both columns, a BRIN on last_execution_time is not. Mutation, a btree on execution_count before the scan: RED (Assert.Empty() Failure: ["execution_count"]). A later btree on any SET column therefore fails a test instead of silently making every upsert non-HOT; pin (b) keeps its hand-written subset, now without the no-op query_text = query_text.
• (g) the shutdown log. A stop during the 20-minute start delay logs at Debug ("cancelled before it started"); only a stop after the delay keeps the Information line about retrying and dropping a half-built leftover. A fake-logger unit pin covers the first branch and a source pin the second. Mutation, the filter flipped: both RED.

Also run and green: DocCommentHygieneTests, StorageCommandTimeoutTests, StartupCommandTimeoutTests, McpReadCommandTimeoutTests, CommentFilterAdoptionTests, RepoFileAdoptionTests, CommandDeadlineScannerAdoptionTests, LiveCleanupConversionRatchetTests, McpPayloadContractCensusTests, McpToolsListBudgetTests, QueryStoreIntervalWideRungTests, QueryStoreIntervalWideMinSchemaVersionPinTests, and the DarlingWorker source-pin classes I found by grep (StartupFailureTriageTests, ScheduledPurgeLaunchTests, StoreObjectConvergenceTests, SharedBaselineCacheTests, RawPurgeNowWiringTests, AlertReadFailureSurfaceTests, and seven more). Build: 0 warnings, 0 errors.

Not run: LivePostgresCollectionHygieneTests.EveryClassUsingTheSharedStore_IsSerializedOrDocumentsWhyNot loads WPF types and cannot run on macOS (10 of its 11 tests pass). The new live class carries the own-store comment, so CI should pass it. Lite.Tests was not changed.

Not changed

• No migration rung, no edit to the table's DDL or fillfactor, no read edits.
• No README edit and no random_page_cost change; both belong to the managed-conf PR.

Findings: 0 substantial. 5 hygiene or polish fixed here: a source pin used the LF reader, which the adoption census restricts; the HOT guard covered only a hand-written subset of the upsert's SET list (now the catalog pin (f)); a no-op column in that subset; the shutdown log (g); and the measured figures now say they predate the purge horizon. 0 dropped.

CHANGELOG

None: collect.query_store_interval_wide is new in 3.9 (#4341; not in v3.8.0), so no 3.8 user has read it and nothing a 3.8 user has seen changes. The 3.9 entry for #4341 carries this index: built in the background about 20 minutes after the service starts, without blocking writes; skipped on PostgreSQL older than 16 and on a hypertable; used by 12-hour-plus Query Store windows where random_page_cost is about 1.1 or lower.