Repository navigation
Query Store backfill: the candidate database list reads the newest chunks by index probe, same list (#4662) - #4681
Closed
erikdarlingdata wants to merge 2 commits into
Closed
erikdarlingdata wants to merge 2 commits into
erikdarlingdata wants to merge 2 commits into
Conversation
…ion boundary so the hot half is a SkipScan (#4662)
…ll-candidate-read
This was referenced Sep 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4662.
Why
QueryStoreBackfill.CandidateSqlaggregated the whole raw window (about four days) of one server's Query Store rows on every backfill tick just to list about a dozen database names. On the uncompressed day-chunks that is a full Index Only Scan over every row the server wrote.What changes
Only
QueryStoreBackfill.cs(the SQL, its doc comment, one optional parameter) and the candidate tests. The list is unchanged: same set, same order.The window is read as two halves split at
$3, so each half gets its cheap plan:(floor, $3]: the compressed chunks. Same decompress-and-hash as before (database_nameis a compressed column with no index there).(floor, infinity) and > $3: the newest uncompressed chunks. TimescaleDB plans a SkipScan down the existing(server_id, database_name, ...)index, about one probe per database.$3= start of the UTC dayTimescaleSupport.CompressAfterDaysback (start of yesterday). Day-chunks compress only once their END is older thancompress_after, so yesterday's and today's chunks are uncompressed in steady state.Exact for ANY
$3, by construction: both halves carrycollection_time > $2, so a$3below the floor makes the cold half empty and the hot half the whole old window. A wrong$3can only cost speed.UNION(not ALL), one outerORDER BY database_name, no COLLATE.MergeHoleDatabasesis untouched. It sorts ordinally, so the final list order never came from the SQL; the ORDER BY is kept and pinned anyway.GetCandidateDatabasesAsyncgets a trailing optionalDateTime? hotSince = null(test seam; default derives from the wall clock).Rig measurement (EXPLAIN ANALYZE, BUFFERS)
Local TimescaleDB 2.30.1 / PG 18.6, real
collect.query_store_statsshape: 1-day chunks, segmentbyserver_id, the three real indexes. Server 1: 1,056,000 rows, 20 databases; server 2: 633,600 rows, 12 databases; 5.5 days of 15-minute ticks. Chunks older than yesterday compressed (4 compressed, 2 uncompressed, as the 1-day policy leaves them). Warm, medians of 6-8 runs, default parallel settings.SELECT DISTINCT ... ORDER BYGROUP BY database_nameGROUP BYwithcount(*)in a subqueryDISTINCT ONover the whole window (SkipScan everywhere)\bind, unnamed statement like Npgsql) plans the same: cold half decompresses the compressed chunks only, SkipScan on the two hot chunks.Risk, measured. If yesterday's chunk is ALSO compressed (a
compress_aftershorter than 1 day; not the policy's steady state), the hot half falls off the SkipScan path and the split loses: server 1 98.4 ms / 4,788 buffers vs OLD 73.2 ms / 4,045; server 2 122.5 (69.3 min) ms vs OLD 57.4 ms. Results stay exact; only the speed goes. The split point followsCompressAfterDays, so changing that constant moves it with it. This is why the production EXPLAIN matters.SQL for production EXPLAIN
Correction (found in verification): the floor is
nowUtc - HorizonFor(_timescaleAvailable). On a TimescaleDB store that is 23 hours:RollupStoreHorizonis the smaller ofRawMaxAge(4 days minus the 1-day route margin, so 3 days) and the hourly refresh reach (1 day minus 1 hour). On plain PostgreSQL it is 3 days, with no compression. It is never 4 days, so the rig table above measured a window the service does not read. At 23 hours the split point (the start of yesterday, UTC) is always older than the floor. The first half is then empty by its own predicate, and the second half is the old query. The literal forms below use the 23-hour floor. On a store without TimescaleDB, use 3 days.OLD, bound form:
SELECT DISTINCT database_name FROM query_store_stats WHERE server_id = $1 AND collection_time > $2 ORDER BY database_nameOLD, literal:
NEW, bound form (
$1int,$2and$3timestamp):NEW, literal:
collection_timeistimestampand Npgsql binds UTC, so the literal forms read the floor astimestampin UTC, whatever the session TimeZone. Check in the plan:SkipScanon the chunks from start of yesterday,ColumnarScanonly on older chunks, and that the newest two chunks are uncompressed (timescaledb_information.chunks). A form that is not faster there does not merge.RED runs (each mutation applied to
CandidateSql, then reverted)Failing tests are the shape pin (
Sql_CandidateSql_IsTheExactSplitShape) plus the live ones named.Beta,Gammamissing: first element_x, expectedBeta), and the existingCandidateDatabases_ServicesAQuietHole_...(new_dbmissing).>=for>: shape pin and the new exactness test (floor_dbappears; "Collections differ at index 5").COLLATE "C": shape pin and the new test (raw row sequence differs from the old statement's).lower(): the shape pin only. The live test cannot tell it apart, because this cluster's collation (Windows libc English_United States) already sorts these names case-insensitively, the same aslower(). State that honestly:COLLATE "C"is the live-order RED.server_id(WHERE $1 > 0 AND ...): shape pin, the new test,CandidateDatabases_ServicesAQuietHole_...andCandidateRead_BoundAtItsOwnHorizon_ScansNoCompressedChunk.Lite
Unchanged, on purpose. DuckDB has no chunk or compression tiers, so a split has nothing to exploit and two scans plus a UNION can only add cost. I did not measure DuckDB. The Lite twin pin (
Sql_LiteTwinCarriesTheSameBoundShapes) passes.Test plan
QueryStoreBackfillCandidateLiveTestsandQueryStoreBackfillTests: 18 passed, 0 skipped, on the rig (before the origin/dev merge).MergeHoleDatabasessorts ordinally), the old SQL plus the merge, six split points (the seeded one, the production default, before the floor, at the floor, after every row, on the newest row), and the raw row sequence against the old statement. The existing plan-shape test now binds a literal$3.WaitRateTileReadCountLiveTests.Waits_DetectAnomaliesAsync_ReadsWaitRateTileWindowSqlOnce_NotTwice: passes when its class is re-run alone on a freshly created database (load-sensitive under the full run). (2)DarlingCliCommandsHostCheckTests.CheckSettingsAsync_ManagedStoreWithAStaleBlock_ReturnsStaleSettingsExitCode_Gated: still fails alone on the rig with42P01 relation "collect.managed_conf_verdicts" does not existfromDarlingStoreHostProfile.ReadStoredManagedConfVerdictsAsync. The test creates an EMPTYdarlingdatabase when the cluster has none (this bare rig had none; the CI cluster creates and migrates one), so it looks like a rig difference, and this diff touches neitherDarlingStoreHostProfilenor a migration. I did NOT confirm it against a clean dev checkout or a dev CI run (dev latest Build runs were cancelled or queued when I looked; the last success was c9847bf at 23:03Z), so it needs checking on CI.What to double-check
DateTime.UtcNowinside the method chooses only the split point; results do not depend on it.Addendum: rig pass on the current read and a recursive walk
Rig and design pass on the backfill candidate read (#4662)
This section changes no code.
Result in one paragraph
On the rig, the one plan flip that stale or missing statistics were supposed to cause never happened. A custom plan never flipped, for either form, with fresh, stale (reltuples at 68% of live) or missing (
pg_statisticdeleted) statistics on the newest chunk and the boundary chunk. Two things did make the read bad:force_generic_plan. The old form reproduces the production bad plan exactly: Gather Merge over ChunkAppend over an Index Only Scan onidx_query_store_stats_server_time_forcing, 188-327 ms and 12-21K buffers. The recursive walk with a bound$2is as bad (159-313 ms, 44-50K buffers), because a name-ordered Merge Append gets no runtime chunk exclusion, so every chunk is opened on every step, compressed ones included.Inlining the floor as a
TIMESTAMPliteral removes the first failure in every condition tested (at most 2.74 ms and 614 buffers), because chunk exclusion then happens at plan time in everyplan_cache_mode. The second failure is only reachable if a chunk inside the 23 h window is compressed. The product policy (TimescaleSupport.CompressAfterDays = 1, which compresses a chunk 24 h after its end) never does that. It is still worth checking on the production store (read-only query below), because its signature fits the production numbers better than stale statistics do.Recommendation
TIMESTAMPliteral and keepserver_idbound. Format it in C# with the invariant culture asyyyy-MM-dd HH:mm:ss.ffffff. That truncates to microseconds, which is the same boundary Npgsql'sDateTimebinding produces today, socollection_time > floormeans exactly the same thing. The value is computed internally and is never user input._hasContinuousAggregates).SELECT DISTINCT … ORDER BY database_name, with the literal floor. SkipScan was chosen in every uncompressed condition, in both plan modes. With compressed chunks in the window it degrades to 175-235 ms, not the 1.4-3.4 s the walk takes there.Rig
(server_id, collection_time);(server_id, database_name, query_id, plan_id, collection_time DESC), called (A) below;(server_id, collection_time DESC) INCLUDE (database_name, …), called (B) below.segmentby server_id.ChunkIntervalDays = 1(TimescaleSupport). There is no 12 h or 6 h setting on dev, so the production interval change was made outside the product. I seeded it the way you described: 1-day chunks from 09-22 to 09-24, 12 h chunks on 09-25 and 09-26, and 6 h chunks from 09-27 to 09-28 22:30. Chunks ending by 09-27 12:00 are compressed.db_05stops at 09-27 20:00,db_13at 19:00 anddb_21at 23:00;db_24starts at 09-28 20:00.PREPAREplus threeEXPLAIN (ANALYZE, BUFFERS) EXECUTEcalls. The table gives the median time and the buffers of the last run.Form × condition (server 1, 24 DBs; ms / buffers / plan)
Forms:
$2.$2."c" is
force_custom_planand "g" isforce_generic_plan.Exactness
I compared the walk against the old SQL, after applying the reader's rules (drop NULL, then
MergeHoleDatabases' ordinal sort). The comparison covered 5 servers (24 DBs, 2 DBs, edge cases, NULL-only, nonexistent), 8 floors (including one inside the compressed era) and both stores. Result: 0 mismatches. Edge cases:Beta,betaandBETAare all kept.Ünïcode,unicodeandunicode(trailing space) are all kept.''is kept.NULL handling is the same in effect. The old SQL returns a NULL row, which the reader skips (
IsDBNull). The walk never emits NULL: its anchor hasIS NOT NULL,>never matches NULL, and the final filter drops the terminating NULL. Row order cannot matter, becauseMergeHoleDatabasesre-sorts ordinal. A hole-only database comes from state, not SQL, so it is unaffected.DISTINCTand>both follow the deterministic column collation, so the walk visits every distinct value.Chosen SQL
The product form:
$1isserver_id, and<floor>is inlined by C# asnow − HorizonFor(...), formattedyyyy-MM-dd HH:mm:ss.ffffffwith the invariant culture.TimescaleDB store:
Plain PostgreSQL store (or everywhere, for the single-form option):
The literal production form is the same text with the floor filled in. For a run at 00:30Z on 2026-09-29, it is
TIMESTAMP '2026-09-28 01:30:00'. Recompute it at run time withSELECT (now() AT TIME ZONE 'UTC') - interval '23 hours';.PREPARE / EXPLAIN EXECUTE script (both plan modes)
Replace the floor and
<id>. This is read-only.plan_cache_modeis checked on every EXECUTE.Read-only check on the production store for the compressed-in-window hypothesis:
Why the plan can't flip
The literal floor removes the plan-mode dependency. A generic plan and a custom plan see the same constant, so chunk exclusion happens at plan time in both. With a bound floor, a generic plan cannot exclude chunks inside a name-ordered Merge Append: PostgreSQL's run-time pruning does not apply to inheritance chunks, and TimescaleDB's ChunkAppend does its startup exclusion only for unordered or time-ordered appends. That was the only flip reproduced on uncompressed data. The same thing happens if
plan_cache_modeis set at the role or database level, and after Npgsql auto-prepare switches to a generic plan: even an unnamed statement gets a generic plan underforce_generic_plan.The walk has one sane plan per step. Each step is
ORDER BY database_name LIMIT 1withdatabase_name > prevas an index condition on (A). The only alternative is a (B) range scan plus a Sort, which reads every in-window row of that chunk before it returns one. The planner picks that only if it believes the chunk has almost no rows above the floor, and that belief can only be right where it is actually cheap:Measured: the same plan in every mode and statistics state, on both stores.
DISTINCT with SkipScan is cost-based, and that is why it needs the literal. Once the literal bounds it to the 2-5 chunks in the window, its only alternative reads the window once instead of every chunk. Where it did switch (compressed chunks in the window), the alternative was still the better plan: about 150-235 ms against the walk's 1.4-3.8 s.
What I could not confirm
Addendum: candidate designs for a read that does not walk
Status: design only. No candidate has been measured. Every cost marked "expected" is unmeasured. The seed and bench plan at the end are ready for a measurement run.
Call rate
The backfill loop ticks every 5 minutes (
s_queryStoreBackfillInterval,DarlingWorker.cs:458) and runs the candidate read once per server per tick, so about 500 calls an hour is about 42 servers × 12 ticks. That is a fixed rate, not a loop. One latent loop from the code, not seen in production (A and B have zero calls there): a slice that throws saves no state, so the same database re-runs A, B and the slice on every tick, and the databases after it in name order are never reached.Where the cost comes from
The floor F cuts one chunk, [S, E). Today's plan is SkipScan on index (A),
(server_id, database_name, query_id, plan_id, collection_time DESC). It finds each name with one seek per opened chunk. The exception is the cut chunk: there, a name with no row after F is read entry by entry, becausecollection_timeis the fifth key column and can only be a filter. That per-entry walk is the whole cost. In chunks at or after E every row passes, so nothing walks there.Candidates (expected cost, unmeasured)
Production, for the same read (buffers median / max): end − 20 min 175 / 1,225; end − 60 min 39 / 551; mid 39 / 185.
DISTINCT … collection_time > Fdatabase_name = n AND collection_time > F ORDER BY collection_time DESC LIMIT 1(server_id, database_name, collection_time)EXISTS (… database_name = n AND collection_time > F)DISTINCT … collection_time >= S, S read from the catalogdone:keyDISTINCT … collection_time >= E; part 2DISTINCT … collection_time > F AND collection_time < E, read time-firstCandidate 2: why the extra names are no-ops (
RunServerSliceAsync, origin/dev)done:key costs no store read.collection_time <= F LIMIT 1, which must hit one of those rows.GetStoredFloorAsyncreturnsfloorLimit, :254 savesdone:<name>, and the loop moves on. The total is one A (production median 7 buffers) and one state upsert, once per database. B and the slice never run for it.MergeHoleDatabasesre-sorts ordinal), and an extra name never slices.PruneOrphanedDatabaseStateKeysSql) deletes Done keys only for databases missing fromdatabase_states. A dropped database whose rows are still in the opened chunks costs one A per prune until those rows leave the cut chunk. That is at most one chunk interval longer than today.Candidate 3
database_name <> ALL($done)filter in SkipScan is only a filter (a btree cannot use<>as an index condition), so it walks the same entries as today.collection_time >= S(one seek per chunk). It passes over names that have a Done key without probing them, and probescollection_time > Fonly for the rest.Candidate 4: the boundary comes from the catalog
Read it at run time, never from a constant:
collect(the 6 h era), and [09-27 00:00, 09-28 00:00) incollect1d.set_chunk_time_intervalchanges only new chunks, so a constant is wrong for every chunk created before the last change. I did not list the 12 h era's chunks.Recommendation (provisional: unmeasured on the rig)
collection_time >= TIMESTAMP '<cut_start>'. It removes the walk, adds no index and no write cost, and keeps today's SQL shape and SkipScan plan. By the proof above, the extra names are no-ops. If the behaviour change is not acceptable, use candidate 3: it costs the same seeks and makes exactly today's decisions, but a quiet name with no Done key still walks.Final SQL, TimescaleDB store: first the catalog read above (the floor as today,
now − HorizonFor, formattedyyyy-MM-dd HH:mm:ss.ffffffwith the invariant culture), then:Plain PostgreSQL store:
Production sweep (read-only)
The floor is a literal, so the sweep can use any floor. Take the current cut chunk [S, E) from the first query, then run each form at E − 20 min, E − 60 min and the chunk's midpoint.
A plain statement with literal values gets a custom plan, so no
PREPAREis needed.1b's write cost: not measured
Recipe, on a scratch hypertable in the rig, never on
collect.query_store_stats:scratch_qss (LIKE collect.query_store_stats)as a 6 h hypertable with the product's three indexes.CHECKPOINT, recordpg_current_wal_lsn(), then\copy1M rows in with\timing on.pg_wal_lsn_diff(pg_current_wal_lsn(), <start>).(server_id, database_name, collection_time). Compare the time and the WAL, and read the new index's size withhypertable_index_size.Seed and bench plan
collect1d, where it cuts [09-27 00:00, 09-28 00:00).db_quiet: 30,000 (query_id, plan_id) pairs, with rows at 09-27 18:10, 19:30 and 20:50 only;db_busy: 2,000 pairs at p = 0.5 on 5-minute ticks, from 09-27 18:00 to 09-28 22:30.collect1dandq2plain, thenVACUUM ANALYZE.q2exact.pyFORMS, plus a floor exactly on a chunk boundary (09-27 18:00:00). Check candidate 2 against the proof (every extra name has a row at or below the floor), not against list equality.CHANGELOG
SECTION: Changed
ENTRY:
REF:
[Query Store backfill: the candidate database list reads the newest chunks by index probe, same list (#4662) #4681]: Query Store backfill: the candidate database list reads the newest chunks by index probe, same list (#4662) #4681