After a big retention drain, warm each server's Query Store floor read in the background (#5581)

[erikdarlingdata]

Claude posting for Erik Darling.

Fixes #5581.

Why

#5572's row-capped drain removed the oldest rows of every server. The read gate's floor read (MIN(first_execution_time) WHERE server_id = $1) walks the index from the oldest entry. It makes a heap fetch for each dead entry that vacuum did not remove and no read marked. On a large store that was 19.6 s and 854,535 heap fetches for one server, against 1.8 s before the drain. The first walk marks the dead entries, so the re-run took 15 ms. Each server paid this on its first Query Store read after the drain.

What changes

• New QueryStoreIntervalFloorWarmUp (Darling.Service). After the retention pass's last purge and its run-record, it runs the shipped QueryStoreIntervalWide.PlainTableFloorSql and QueryStoreIntervalLatest.PlainTableFloorSql once per server, one server at a time. It executes the constants by reference, so it walks the same index with the same plan as the gate.
• It runs for a table only when that table's interval purge may have left dead entries. That means the purge deleted rows, or a part of it failed (a failed part can leave committed batches). See "Wiring after Query Store interval tables are partitioned by day #5573" below. There is no size threshold: when no row is dead, each read is one index probe (milliseconds).
• Server ids come from collect.servers (a tiny registry table). There is no SELECT DISTINCT server_id on the big table.
• Each server gets its own connection and a SET lock_timeout of 3 s. The per-read command timeout is a named constant (FloorReadTimeoutSeconds = 300, about 7x the slowest measured read of 38.9 s). The read takes only ACCESS SHARE. So a lock timeout means an ACCESS EXCLUSIVE request is queued (a schema upgrade step, or Query Store interval tables are partitioned by day #5573's partition step). That server is skipped and logged. A read that passes its command timeout is logged and skipped, but it has already marked the dead entries it walked past. The next pass continues from there.
• A timeout, cancel or failure is logged and never counts as a failed purge. Per table it logs one line: servers warmed, skipped on a lock, failed, total time, slowest server and its time.
• Stale comments fixed. The PlainTableFloorSql docs in both storage files and the wide-table purge comment now say the floor read is not free right after a big row delete. They also say the dead entries are in the legacy table and the DEFAULT partition.

Where it runs (ruling 2)

Both callers of PurgeAsync already run it in the background _purgeTask slot. Neither is awaited on the sweep loop:

• Daily purge: DarlingWorker.cs:3691 TryStartScheduledPurge(... RunScheduledPurgeAsync ...) (fire-and-track), which calls PurgeAsync at DarlingWorker.cs:11778.
• purge_now: RunPurgeNowBackgroundAsync, PurgeAsync call at DarlingWorker.cs:11927, same slot.

So the warm-up sits at the end of PurgeWithPacerAsync, and the loop and the collectors never wait for it. It uses one connection, one server at a time, with no parallel reads. The cost of this placement is that the chores after PurgeAsync in the same task wait behind the warm-up. Those are findings cleanup, the log-file sweep, the module-map refresh and the chunk-interval reconcile. They wait once after a big drain (about 11 min on the large store). I kept it simple instead of threading the drain counts out through PurgeSummary. Tell me if you want it moved to the end of both tasks.

The limit

PostgreSQL marks an index entry dead only when its row is dead to every open snapshot. A transaction that was open before the drain ended keeps the entries live. Then the warm-up pays the walk and the next read pays it again. The warm-up cannot see or end such a transaction.

Autovacuum scale factor (item 6): no change

Autovacuum did start on the large store (17.9 M dead against a threshold of 14.1 M, which is 50 + 0.2 x 70.5 M live). The wide table keeps 9 days, so steady state is about one day of rows (about 11% of the table) dead per day. That is below the 20% trigger, so it fires roughly every second day. That is already frequent for a table with 15 GB of indexes.

A lower factor scans all 15 GB of indexes more often. It does not fix the reported problem, which was the read latency until the index pass ends (42 to 49 min on the large store). The warm-up fixes that directly. Space reclaim stays with autovacuum, and PgTableTuning.cs is untouched.

Merge order

#5573 merged first (as 92cdf50), and this PR carries the wiring to its PurgeIntervalTableAsync.

Wiring after #5573

Dev's merge was clean as text but did not compile against this PR: intervalWideDeleted and intervalLatestDeleted no longer exist.

• DarlingRetention.cs:1120: the call now passes intervalWidePartitions and intervalLatestPartitions, the two IntervalPartitionPurge results.
• QueryStoreIntervalFloorWarmUp.cs:115: RunAfterDrainAsync takes the two IntervalPartitionPurge results instead of two nullable counts. It still never throws and never fails the purge.
• QueryStoreIntervalFloorWarmUp.cs:149: new LeftDeadEntries. A table warms when RowsDeleted > 0 or Failed > 0. A pass that only dropped whole day partitions skips it, because a dropped partition takes its indexes with it.
• DarlingRetention.cs:823 and :1110: comments now describe the partitioned tables (dead entries sit in the legacy table and DEFAULT, and the warm-up gate).
• QueryStoreIntervalWide.cs:459 and QueryStoreIntervalLatest.cs:349: the PlainTableFloorSql doc comments say the dead entries are in the legacy table and DEFAULT after Partition the two Query Store interval tables by day on first_execution_time, so retention drops a day instead of deleting its rows #5571.
• QueryStoreIntervalFloorWarmUp.cs:61: class doc gains a "Partitioned tables" paragraph.
• Floor read on the partitioned parent: the shipped constants are unchanged. On the parent the MIN runs over each partition's index. The 3 s lock_timeout skip still works against an ACCESS EXCLUSIVE lock on the parent, and the locked-table test passes on the partitioned tables.
• Tests (QueryStoreIntervalFloorWarmUpTests.cs): seeding sets autovacuum_enabled = false on the legacy table (:479), since a partitioned parent has no storage options. The sweep pin follows the new call (:115). New OnlyAPurgeThatDeletedRowsOrFailed_LeavesDeadEntries (:143, 6 cases) and APassThatOnlyDropsWholePartitions_DoesNotWarm (:378).

Comparison (large-store copy, #5581 comments)

	Time	Notes
Plain VACUUM (autovacuum at default 2 ms / 200 throttle)	Heap pass 11.89 M pages (91 GB) in about 29 min. Index pass ended 42 to 49 min after the start. Second heap pass still running at 49 min.	Floor reads stay slow until the index pass ends. A manual VACUUM is never cancelled by a waiting lock, so #5571's ACCESS EXCLUSIVE steps queue behind it, and inserts and reads behind them. Earlier autovacuum runs were cancelled by the arm's ADD CONSTRAINT and by ANALYZE.
Warm-up (this PR)	43 servers, one at a time, 644.9 s total (about 11 min). Slowest 38.9 s. 26 of 41 over 10 s.	17.18 M heap fetches, matching autovacuum's 17.53 M dead item ids. Afterwards the floor read for 41 servers took 16 ms in total, max 0.75 ms.

Test plan

• QueryStoreIntervalFloorWarmUpTests (17 tests now). The rig is on port 55550 with two servers and 20,000 expired rows each. The rows are seeded through the partitioned parent, so they land in the attached legacy table:
  • Source pins: the warm-up runs only the two shipped PlainTableFloorSql constants, one reference each. It builds three NpgsqlCommands (lock timeout, registry read, floor read), with no DISTINCT and no other FROM collect.. The sweep calls it once, after the last purge, the last prune and the run-record.
  • Premise: a drain with no warm-up leaves the first floor read walking the dead entries. Fix: through the retention pass, which row-purges the legacy table, the first read per server is cheap and the pass logs the warm-up line. Both tables.
  • A pass that only drops whole expired day partitions (promoted store, one row in the expired day) logs no warm-up line for either table.
  • Locked parent: an ACCESS EXCLUSIVE lock is taken right after the sweep's summary line. Each server skips within the lock timeout and is logged. The purge still returns its row count with no Error logged.
  • Server-list failure and a cancelled token: logged, nothing thrown. A pass that deleted nothing logs no warm-up line.
• Red-plant: I replaced the legacy purge's row count with 0 (rows += 0). 4 of the 17 failed: TheRetentionPass_WarmsEachServersFloorRead... for both tables, the locked-table test and the server-list test. Restored.
• On the final head (01d3790, includes the merge of origin/dev): *Retention*, *QueryStoreInterval*, *FloorWarmUp*, *IntervalPartition*: Total 494, Failed 0, Skipped 2. *Census*, *Ratchet*, *DocCommentHygiene*: Total 570, Failed 0. Stage=Guard: Total 1420, Failed 0. Build 0 warnings.
• The full suite was not run (laptop rule). Not run against a large store. The Comparison numbers are from the seat's rehearsal.

The 2 skips are TheRetentionPass_WarmsEachServersFloorRead... (both tables) in the big run. Another class's open transaction on the same cluster held the horizon, so nothing could be marked (the limit above). The test skips by design. Run alone, the class passes 17 of 17 with 0 skipped.

Note on the measure: on the test tables the planner picks a plain Index Scan, which reports no "Heap Fetches" figure. So the tests use the plan's total buffers (heap fetch plus index page per dead entry) and also bound "Heap Fetches" when a plan reports it. Only the first read per server counts, since EXPLAIN ANALYZE marks entries too. The index is ordered by first_execution_time across servers, so the first server's walk also marks the others' entries.

Coordinator double-checks

• Warm-up placement versus the chores that follow PurgeAsync (above).
• Failed > 0 warms even when the failure was a drop or a relation probe that deleted no row. That is the brief's rule (a failed part can leave committed batches), at the cost of an occasional cheap warm-up.

Proposed CHANGELOG entry (Fixed): The first Query Store read after a big retention drain is no longer slow ([#PR]): after the retention purge removed the oldest Query Store interval rows, each server's first floor read walked every dead index entry (19.6 s on a large store). The purge now runs that read once per server in the background when it finishes. It goes one server at a time and skips a server whose table is locked.