Query Store interval tables stop cancelling their own autovacuum (#5589, #5594)

[erikdarlingdata]

Claude posting for Erik Darling.

Fixes #5589.

Fixes #5594. The #5594 section is below the #5589 test plan.

Why

While a Query Store interval table is not promoted, its legacy table is the whole live table. A monitored clock can put the newest first_execution_time more than one retention horizon ahead. In that state the arm took LOCK TABLE ... IN ACCESS EXCLUSIVE MODE before it read the newest time. It then found the time too far ahead and rolled back. Both the hourly convergence step and the background loop did this every hour. Each lock request cancels a running autovacuum (unless it is an anti-wraparound run), so autovacuum on that table never finished.

What changes

• QueryStoreIntervalPartitions.ArmCoreAsync now reads the legacy table's newest time before it opens the lock transaction. A new ReadLegacyMaxUnlockedAsync runs the existing ReadLegacyMaxAsync in its own short transaction and rolls it back. The read keeps the same index checks and the same 5 s statement timeout, and it takes only an ACCESS SHARE lock.
• When IsLegacyMaxBeyondHorizon is true for that read, the arm calls WarnFarFutureMax (once per table and time, as before). It then returns the same NotReady result and requests no lock.
• A null read (no usable index, a timeout or an empty table) goes on to the locked path. The read under the lock still decides the arm.
• Doc comments updated: the class Locks paragraph, and the IsLegacyMaxBeyondHorizon and ArmCoreAsync comments.

What else runs in this state

• Promotion returns at the arm (NotReady), so no VALIDATE runs.
• The analyze step returns NotReady for an unpromoted table and takes no lock.
• The Query Store index ensure runs once per process start. On a partitioned table it locks only the parent (CREATE INDEX ... ON ONLY) and builds on the leaves that pg_partition_tree lists. V171 attaches the legacy table as MINVALUE..MAXVALUE, so it is a leaf and pg_partition_tree lists it. V171 also attaches each valid legacy index ({name}_legacy) to the parent's index. So on an upgraded store whose indexes were valid, the ensure sees a child on the legacy table and skips it. It builds on the legacy table only when no valid index existed there (V171 drops an invalid one), and then once: the same build the plain table would have had. AnInvalidLeftover_IsDroppedAndRebuilt pins that case.
• Lite has no interval partitions, so there is no Lite change.

Changelog

No separate entry. The partitioning (#5573) ships in the same release, and its entry stays true.

Test plan

• New live test AFarFutureLegacyRow_AnHourOfBothLoops_RequestsNoLock_WhileAnAutovacuumLikeLockIsHeld. A second connection holds SHARE UPDATE EXCLUSIVE on the legacy table, standing in for autovacuum. The test runs three ConvergeUnpromotedAsync calls. It then runs three passes of the background loop body through RunDelayedAsync, with RunPromotionAsync and AnalyzeIfDueAsync as in production. Every call returns NotReady in under 3 s. A third connection polls pg_locks every 20 ms and never sees an ungranted lock on the legacy table. One far-future warning is logged, and no CHECK exists afterwards. The test takes about 2 s.

• Red-plant: with the pre-check disabled, the test fails with "converge ended RetryLater" after 348 s of lock waits. Restored.

• New live test ALegacyRowInsideTheHorizon_StillTakesTheArmLock_AndIsRetryLaterWhileAnAutovacuumLikeLockIsHeld. A newest time inside the horizon still goes to the locked path. It returns RetryLater while the lock is held and arms once the lock is released.

• Source pin TheRound2Fixes_AreInTheProductText updated. The read under the lock is now the last IsLegacyMaxBeyondHorizon call. A new assertion pins the unlocked read and its horizon test (with WarnFarFutureMax) before the lock transaction.

• On head ffb3e28, PostgreSQL 18:

  Tests	Passed
  *QueryStoreIntervalPartitions*	37
  *QueryStoreIntervalConvergence*	23
  *IntervalPartition*	50
  *Census*	477
  *Ratchet*	15
  *DocCommentHygiene*	78
  Stage=Guard	1420

  Build: 0 warnings, 0 errors.

• Not run locally: the rest of Darling.Tests and Lite.Tests. CI runs them.

#5594: BRIN autosummarize cancelled the interval tables' autovacuum

Why

ix_query_store_interval_wide_collection_time_brin was built with autosummarize = on (#4862). Each new block range queues an autovacuum "BRIN summarize" work item. The item waits for SHARE UPDATE EXCLUSIVE on the table, which a running VACUUM holds. After deadlock_timeout (1 s) PostgreSQL cancels that VACUUM. On the store in the issue, a big drain left the table's vacuum cancelled every ~2 minutes and never finished.

What I measured (throwaway PostgreSQL 18.6 rig)

Setup: a 1 GB table of 3 M rows with a BRIN on collection_time. I deleted 90% of the rows and throttled autovacuum (autovacuum_naptime = 1, cost delay 10 ms, cost limit 100). A loop inserted 3,000 rows every 0.7 s, so new block ranges kept landing.

	Before (autosummarize on)	After (off)
"canceling autovacuum task" lines	69 in 240 s (about 17 a minute)	0 in the 16 minutes until the vacuum finished
Completed vacuums	0	1 (141,112 pages scanned, 2,100,640 tuples removed)

• While the loop ran, pg_locks showed a granted SHARE UPDATE EXCLUSIVE for autovacuum: VACUUM ANALYZE and an ungranted one for autovacuum: BRIN summarize, the same pattern as the issue.
• I turned it off on the live index with ALTER INDEX ... SET (autosummarize = off) while the vacuum was running. That one ALTER took 1,008 ms and caused one more cancel (the last one).

The ALTER and the lock it takes

• A partitioned parent index cannot be altered. PostgreSQL 18.6 answers "This operation is not supported for partitioned indexes". The parent index is a template with no storage. A leaf index can be altered.
• A new leaf clones the parent index's options. Measured: both PARTITION OF and CREATE TABLE plus ATTACH give a leaf autosummarize=on from an on parent. Attaching an off child to an on parent works.
• The lock is ACCESS EXCLUSIVE on the index only, not the table (read from pg_locks). A running VACUUM holds ROW EXCLUSIVE on the table's indexes for its whole run, so the ALTER waits for it. After deadlock_timeout PostgreSQL cancels a regular autovacuum and the ALTER goes through. While the ALTER waits, writers and readers of that leaf queue behind it. The planner takes ACCESS SHARE on every index of a leaf it plans against (checked with pg_locks). Round 1 limits that wait, see below.
• I accept that one cancel. It happens once per index and ends a cancel loop. A lock_timeout below deadlock_timeout would rarely get the lock on the store in the issue, where a vacuum is almost always running.
• I ran this on 18.6 only. I did not test PostgreSQL 16 or 17 here.

Rung or runtime: runtime

• A rung cannot fix the parent index, which is the one thing that makes new leaves clone on. The leaves can be fixed at runtime, in the code that already owns them.
• QueryStoreIntervalBrin.TurnOffAutosummarizeAsync reads the catalog for BRIN leaf indexes with autosummarize on, through the table's partition tree, and alters only those. It runs first in the hourly "query store interval partitions" pass, so it also runs on the start path before the collectors.
• Promotion and create-ahead call the same fix inside the transaction that makes the leaf, while nothing else holds it. A promotion holds ACCESS EXCLUSIVE on every partition at that point, so it waits for nothing.
• New builds say autosummarize = off (QueryStoreIntervalWideBrinIndex, so the background ensure's parent and leaf builds too).
• V171's text is edited in round 1 (see below): its parent BRIN now says autosummarize = off. V171 is unreleased (no tag contains it). No version bump. I looked for a migration-text census, hash or ladder pin on that literal and found none (MigrationLadderPins searches column names and a view name only).
• query_store_interval_latest has no BRIN. The DEFAULT partition gets a clone like any leaf, and the fix covers it.

Explicit summarize: added

Without autosummarize, a range is summarized only when a VACUUM finishes, and an unsummarized range is always read. Measured on the rig, a 6 h window of a 24 h, 111 K-block table:

Unsummarized tail	Blocks read
none	27,904
10%	38,895 (+39%)
20%	50,006 (+79%)
after brin_summarize_new_values	27,904

Insert-driven autovacuum fires after 20% growth, and a leaf that stops growing keeps its tail until another vacuum. So the tail is real, and every collection_time read through the BRIN pays for it.

QueryStoreIntervalBrin.SummarizeNewRangesAsync runs hourly on the existing background loop (QueryStoreIntervalPartitions.RunDelayedAsync, which already holds the daily analyze). It is not in DarlingRetention.cs.

• It reads deadlock_timeout at run time and uses a lock_timeout of half of it, at most 2 s. It does nothing when deadlock_timeout is under 200 ms.
• It skips an index whose table pg_stat_progress_vacuum shows being vacuumed, without asking for a lock. A role that cannot see the worker reads NULL there, and then the lock_timeout decides.
• Each index runs in its own transaction with a 60 s statement deadline. A cancelled call keeps the ranges it finished. Measured: after a 25 ms cancel of a tail of about 87 ranges, 22 were left to do. So a huge tail is worked off over several passes.
• Cost: a call with nothing to do takes 0.2 to 0.5 ms. Summarizing 87 ranges (about 11,000 blocks) took 59 ms.
• It summarizes the last, partial range too, as brin_summarize_new_values does (both go through the same function with include_partial).
• I did not list every reader that filters on collection_time. The cost is per block read, whichever reader it is.

Lock steps on these tables

Only the first two rows were measured. The rest are read from the code and the lock modes. Nothing else on an hourly path requests a lock that waits on a vacuum when the state is already converged.

Statement	Lock	How often	Waits on autovacuum?
ALTER INDEX ... SET (autosummarize = off) (new)	ACCESS EXCLUSIVE on the index; queues the leaf's writers and readers while it waits	Once per index; later passes only read the catalog	A regular autovacuum: one cancel after deadlock_timeout. A manual VACUUM or an anti-wraparound autovacuum is never cancelled, so the index is skipped with no lock request. Anything unseen: lock_timeout is min(5 s, deadlock_timeout + 1 s)
brin_summarize_range, one call per range (new, was brin_summarize_new_values)	SHARE UPDATE EXCLUSIVE on table and index, taken and released inside each call	Hourly	No: lock_timeout is half of deadlock_timeout, and a vacuumed table is skipped
ANALYZE of the parent	SHARE UPDATE EXCLUSIVE on each leaf (PostgreSQL 16 and 17 sample every leaf)	Daily per table	Yes, after 1 s; 5 s lock_timeout. Left as is: daily, not hourly
Arm: ADD CONSTRAINT on legacy	ACCESS EXCLUSIVE	When there is no CHECK, or a re-arm near S; #5589 stops the hourly ask for a far-future row	Yes; 5 s, retried 3 times 30 s apart
VALIDATE CONSTRAINT on legacy	SHARE UPDATE EXCLUSIVE	Once per arm	Yes; 5 s
Promote: LOCK TABLE parent (every leaf), DETACH and ATTACH legacy, CREATE partitions	ACCESS EXCLUSIVE	Once per table; retried by the next pass after a lock timeout	Yes; 5 s
Create-ahead: CREATE TABLE, LOCK DEFAULT, ATTACH	SHARE UPDATE EXCLUSIVE on the parent, ACCESS EXCLUSIVE on DEFAULT and the new table	A day at a time (the hourly pass only reads the catalog otherwise)	DEFAULT is empty, so no wait of note
Drop expired (DarlingRetention.cs, not edited)	ACCESS EXCLUSIVE on parent and leaf	Daily	Yes; 5 s
Index ensure: CREATE INDEX IF NOT EXISTS ... ON ONLY parent	SHARE on the parent	Once per process start	No: the parent is not vacuumed
Index ensure: leaf CREATE INDEX CONCURRENTLY	SHARE UPDATE EXCLUSIVE on the leaf	Only for a leaf with no child index, so once	Yes, no lock_timeout (7,200 s deadline). Left: once ever, and it is the build itself
Index ensure: ALTER INDEX ... ATTACH PARTITION	SHARE UPDATE EXCLUSIVE on the leaf table, ACCESS EXCLUSIVE on the child index	Once per leaf that was built	Yes; 5 s
V171 and V172 rename	ACCESS EXCLUSIVE	Once, at the migration, before the collectors	lock_timeout 280 s
Retention purge DELETE, writer upserts, reads	ROW EXCLUSIVE, ACCESS SHARE	Continuous	No conflict

Changelog

Probably no entry. Autovacuum was only starved on a store that had just drained a very large table, and #5573 (the same release) changes the tables anyway. An entry is worth it if 3.10's notes already mention #4862's autosummarize. Suggested text: "Fixed: autovacuum of the Query Store interval tables is no longer cancelled every couple of minutes after a large drain."

Test plan for #5594

• New QueryStoreIntervalBrinLiveTests (10 tests on the rig in the first round, 23 after round 1). An upgraded store with on indexes ends with every BRIN leaf off after the hourly step, promotion, create-ahead and a pass.
• The same class: a fresh store's build and leaves are off.
• The same class: with SHARE UPDATE EXCLUSIVE held on the legacy table, the summarize returns RetryLater in under deadlock_timeout. pg_locks never shows a waiter that long. Once the lock is gone it summarizes the tail, then finds nothing.
• The same class: a deadlock_timeout of 100 ms makes the summarize not run. The lock-wait rule is checked for six deadlock_timeout values.
• Red-plants, all three at once. The build spec back to on failed TheBuildSql_..., both WideBrinIndexLiveTests ensure tests and the fresh-store test. The create-ahead hook removed failed the upgraded-store test. The summarize lock_timeout set to deadlock_timeout plus 500 ms failed the held-lock test. Restored.
• *QueryStoreInterval*, *IntervalPartition*, *BackgroundIndex*, *Brin*: 312 tests, 2 skipped, 2 failed. The two failures were my expected text (pg_get_indexdef prints autosummarize=off without quotes). I fixed it, and the two classes that hold them then passed, 20 of 20.
• *Census*, *Ratchet*, *DocCommentHygiene*, *StorageCommandTimeout*: 602 passed. Stage=Guard: 1,432 passed. Build: 0 warnings, 0 errors.
• Not run: PostgreSQL 16 and 17 (the rig is 18.6 only). CI.
• The "being vacuumed" skip now has live tests, see Review round 1.

Review round 1 (the #5594 part)

Head after this round: 594ad7d. Every finding is fixed. Measured on a throwaway PostgreSQL 18.6 rig (UTC, pg_stat_statements loaded).

M1. The turn-off waited 5 s behind a vacuum it cannot cancel

• Fixed (a): no lock request when the wait cannot end in a cancel. The turn-off's catalog read now also says whether the index's table is being vacuumed by a vacuum the deadlock check never cancels. That is a manual VACUUM (the vacuuming backend is not an autovacuum worker). It is also an autovacuum that is anti-wraparound: age(relfrozenxid) or mxid_age(relminmxid) is past the smaller of the table's own and the server's autovacuum_freeze_max_age or autovacuum_multixact_freeze_max_age, or the worker's activity text ends in (to prevent wraparound) (round 2). Such an index is left on, the step returns RetryLater, and the next pass tries again. A role that cannot see another role's worker reads NULL for pg_stat_progress_vacuum and pg_stat_activity. I checked with a plain role: the row is there, but relid and backend_type are NULL. Then only (b) applies.
• Fixed (b): a shorter wait for what no check can see. The ALTER's lock_timeout is min(5 s, deadlock_timeout + 1 s), read from the server in the same session. A regular autovacuum still gets its deadlock_timeout to be cancelled plus a second to let go. On a default server a futile wait drops from 5 s to about 2 s.
• Can the turn-off and the summarize overlap? Yes, but they do not get in each other's way. They are not one loop. The turn-off is a step of the convergence pass (start path, then hourly). The summarize is on the background task (20 minutes after the start, then hourly), on its own connection. Nothing orders them. But brin_summarize_range takes its SHARE UPDATE EXCLUSIVE locks inside the call and releases them when the call ends: pg_locks shows none held after it. So a summarize holds the index for one range at a time, not for its run. I ran it. An ALTER sent 1.5 s into a 10.7 s summarize of a 156 K-block table took 150 ms. The summarize still finished all 1,220 of its ranges. I first wrote a pg_locks skip for "another backend holds SHARE UPDATE EXCLUSIVE". It could not see a summarize between ranges, so I removed it. The service's own summarize needs no skip, so there is none.
• Remark and lock table fixed. Readers stall too, not only writers. The class remark and the lock table above say so.

L1. One lock timeout left every later index on

The turn-off now moves on to the next index after a lock timeout or a skipped one. It counts them and returns RetryLater after the loop. Its Count is the indexes it did alter, and the maintenance pass now counts those too. It also treats 42P01 (the leaf was dropped between the read and the ALTER) as nothing to do.

L2. A stopped summarize can strand a placeholder range: confirmed, and fixed by stopping between ranges

On the rig: a 150 ms statement_timeout on brin_summarize_new_values over a 47 K-block table left one placeholder = t tuple. A second brin_summarize_new_values (347 ranges) and brin_summarize_range on that very block both left it. Only brin_desummarize_range on that block followed by a summarize healed it. A scan always reads a placeholder range. So each deadline hit costs one 128-block range that every query reads, for good.

So I took the cheap way to stop between ranges. The hourly summarize is now one statement per index. It calls brin_summarize_range for each range of the heap in block order, and each call sits behind a clock check against the 60 s budget. Rig: a 200 ms budget summarized 24 ranges and left no placeholder. The next pass did the other 342. A statement_timeout 30 s past the budget stays as a backstop for a stuck read. Only that can leave a placeholder, and the log says so. Cost of testing summarized ranges each pass: about 3 microseconds a range, so about 0.3 s for the 12.3 M-block table.

When can the budget be hit? Measured speed was 14,600 to 36,000 blocks a second (a 156 K-block table took 10.7 s, a 47 K-block one 1.2 s). One hour of a day leaf's growth is a second or two. Only a tail nothing ever summarized reaches 60 s. The 12.3 M-block legacy table takes 6 to 14 minutes of reading, so several hourly passes. With the old call that was several stranded ranges.

L3. A dropped leaf aborted the table's step

The statement resolves the index once and reads no rows when it is gone, so the result is (0, 0), not a NULL scalar that throws. A drop that lands after that surfaces as XX000 (round 2 corrected this from 42P01), and the step skips the index only when it no longer resolves, so the table's other indexes still run. The first part is tested. The race itself cannot be forced in a test.

L4. Duplicate Information line

The Information log in the summarize is removed. The step runner's line stays.

L5. The two missing live tests, plus the new paths

• A manual VACUUM in a second session, throttled to one page per 100 ms and cancelled at the end of the test. The turn-off leaves the index on and returns RetryLater. pg_locks never shows its request waiting. The summarize does the same ("skipped 1 being vacuumed", nothing waiting), and the next pass does the work once the vacuum is gone.
• The turn-off's lock-timeout path: a reader's open transaction holds the legacy BRIN index. The step takes about deadlock_timeout + 1 s, rolls that index back, still turns off the day leaf that sorts after it (L1), and logs the Warning.
• The summarize stops between ranges. A budget of 0 starts none, the next pass does all of it, and a third finds nothing. A missing index returns (0, 0).
• Text pins: the summarize uses brin_summarize_range with a clock check and not brin_summarize_new_values; the holder read has the freeze-age tests; the wide rung says autosummarize = off.
• Red-plants, all at once. The vacuum EXISTS in the turn-off replaced by false failed the manual-VACUUM turn-off test. The same in the summarize target read failed the summarize test. A return in place of moving on after a lock timeout failed the reader test. Restored.

L6. V171's parent BRIN now says off

PgMigrations.cs creates the parent BRIN with autosummarize = off, so a leaf made by PARTITION OF or ATTACH clones off. No release tag contains V171 (git tag --contains 92cdf50fc is empty). The upgrade-from-V170 rung test now asserts the parent is off. It also asserts the legacy copy stays on until the runtime step turns it off. On a fresh install the rung never creates the parent (it needs an existing legacy BRIN). The background build already says off. A store that already ran V171 keeps an on parent, so the runtime code stays.

Nit

var oid =await in QueryStoreIntervalWideBrinIndexLiveTests.cs now has its space.

Test plan, round 1

• *QueryStoreInterval* 284, *IntervalPartition* 50, *BackgroundIndex* 41, *Brin* 43, *MigrationLadder* 4, *MigrationUpgradeLadder* 7: all pass.
• *QueryStoreIntervalBrinLiveTests*: 23 pass. With the three red-plants in, 3 fail.
• *Census* 485, *Ratchet* 15, *DocCommentHygiene* 78, *StorageCommandTimeout* 24, *StartupCommandTimeout* 15, Stage=Guard 1,432: all pass. Build: 0 warnings, 0 errors.
• Not run: PostgreSQL 16 and 17. A real anti-wraparound autovacuum (a test cannot start one on demand; round 2 tests the age expression on its own instead). The full suite (no full run on this laptop).

What to double-check

• The class remark in QueryStoreIntervalBrin.cs is long. It carries the measured numbers above.
• SummarizeStatementTimeoutSeconds is now SummarizeBudgetSeconds plus SummarizeBackstopSeconds. Nothing else referenced it.
• Changelog: no buffer directory was given in the brief, so I wrote no entry.

Review round 2

Head after this round: 1502a4d. Measured on a throwaway PostgreSQL 18 rig (UTC, pg_stat_statements loaded). Nit 2 is accepted and nothing changed for it. Only the log count of an index whose walk timed out is lost. The ranges it did summarize stay summarized.

L1. The anti-wraparound test could read "not anti-wraparound" for a vacuum that is

• Fixed: the smaller limit. Autovacuum uses the smaller of the table's autovacuum_freeze_max_age setting and the server's (relation_needs_vacanalyze). The age test now uses LEAST(table setting, server setting) for the XID pair and the multixact pair. A per-table value above the server's no longer hides a real anti-wraparound vacuum.
• Fixed: the direct signal. The pg_stat_activity check in the same EXISTS now also accepts query LIKE '%(to prevent wraparound)'. Autovacuum writes that text into its worker's activity for an anti-wraparound vacuum (autovac_report_activity in autovacuum.c, PostgreSQL 16 to 18). It also covers the multixact member-space limit, which lowers autovacuum's real limit and which SQL cannot read.
• The class remark names both signals.

L3. The age leg had no test

• The age expression is now one constant, AntiWraparoundAgeSql, which AutosummarizeOnSql uses. A live test evaluates it alone against three scratch tables. One has the 100,000 minimum, one the 2 billion maximum, and one no setting.
• It reads false for all three, burns 100,200 XIDs, then reads true only for the 100,000 table. The 2 billion table stays false, because autovacuum ignores a per-table limit above the server's. Burning took 0.2 s on the rig, so the test adds about 2 s in all. An open transaction that holds an XID keeps VACUUM from freezing the tables and resetting the age mid-test.
• The smaller-limit leg is tested for real. The server setting cannot change without a restart, so the test puts a function named current_setting earlier in search_path that answers 100,000 for the freeze setting. The 2 billion table then reads true.
• Red-plants, each run alone: flipping > to < in the XID leg fails the test at the first "reads false" check. Replacing LEAST with COALESCE (the table's setting only) fails it at the smaller-limit check. Both were restored.
• No live test covers the multixact leg (burning multixacts is slow) or the activity-text leg (a test cannot start an anti-wraparound autovacuum on demand). Both are pinned by text in TheStatementShapes_StopBetweenRanges_AndCountEveryVacuumThatCannotBeCancelled.

L2. A leaf dropped during the walk aborted the table's summarize

• brin_summarize_range raises XX000 ("could not open relation with OID") when its index is gone, so the old 42P01 catch could never fire. The catch now takes XX000 or 42P01, rolls back, and asks to_regclass for the index on the same connection. When it reads NULL, the index counts as dropped and the walk goes on to the next index. Otherwise the error is rethrown as before.
• The decision is a small static method, IsDroppedIndexError(sqlState, indexStillExists). A unit test covers seven cases: both codes with the index gone (dropped), both with the index still there (rethrow), and other codes. There is no timing-dependent live test.

Nit 1. The partial range comment

brin_summarize_range does summarize the last, partial range (include_partial). The comment in QueryStoreIntervalBrin.cs, the two test comments that said otherwise, and a line in this body are fixed.

Test plan (round 2)

• *QueryStoreIntervalBrin*, *Brin*, *BackgroundIndex*, *Census*, *Ratchet*, *DocCommentHygiene* on a live PostgreSQL rig: 670 passed, 0 failed.
• Stage=Guard: 1,432 passed, 0 failed.
• Build: 0 warnings, 0 errors.
• Not run: the full suite (none runs on this laptop), PostgreSQL 16 and 17, and a real anti-wraparound autovacuum.