You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[BUG] query_store collector: monolithic TVF join is a fixed ~30s+ cost on big catalogs — stage through #temp (sp_QuickieStore architecture) #2133
Field evidence from the dogfood fleet (2026-08-08): after #2125's adaptive shrink deployed, most stragglers converged, but a hard core (AppDatabaseThirteen@multi22, AppDatabaseFourteen@multi-01, AppDatabaseFifteen@multi32, AppDatabaseSixteen@psi, AppDatabaseOne@multi-49, AppDatabaseTwelve@multi22, and rotating peers) kept failing at every window width — 60m, 30m, 15m identically — with zero rows shipped for hours. Width-independence was the tell: some fixed cost eats the 60s command timeout before the window predicate matters.
inner aggregate alone (GROUP BY plan_id, interval_id)
7,346 groups, 81 ms
bare SELECT COUNT(*) sys.query_store_plan
82k rows, 319 ms
bare SELECT COUNT(*) sys.query_store_query
64k rows, 208 ms
aggregate JOIN qsp JOIN qsq (the collector's shape, WITH the LOOP JOIN hint)
timeout ≥30s
same WITHOUT the hint
timeout ≥30s
aggregate joined to qsp ONLY
timeout ≥30s
staged: aggregate INTO #x, then #x JOIN qsp JOIN qsq
stage 56 ms + join 409 ms = 524 ms total
Two relations that each materialize in well under half a second join catastrophically when the optimizer has only TVF fixed-guess cardinalities to work with — the inner side re-materializes per probe (7k probes × ~300 ms ≈ 35 minutes of work against a 60s timeout). The OPTION(LOOP JOIN) hint neither causes nor cures it.
Fix
Restructure QueryStoreCollector.BuildPayloadBody to stage the runtime-stats aggregate into a #temp and run the plan/query/text joins FROM the temp — sp_QuickieStore's architecture, for exactly this reason. Constraints to preserve:
ONE result set at the end of the batch (the reader/byte-budget machinery is single-result-set);
TOP WITH TIES + derived-watermark semantics unchanged on the final SELECT;
the LOOP JOIN hint must NOT survive onto the staged join (looping from the temp into the TVFs is the pathology itself; the 409 ms join is unhinted);
both engine arms (on-prem/RDS sp_executesql nesting and the Azure per-database parameter path — #temp scope lives and dies with the batch in both), both SKUs, live + backfill variants;
while here: the interval pre-filter can resolve ids from sys.query_store_runtime_stats_interval (20 ms) instead of scanning runtime_stats (426 ms) — not the wall, but strictly better and the same tie-group semantics via the existing HAVING.
Expected field effect: the wedged members drop from 'never completes inside 60s at any width' to sub-second, and the adaptive shrink becomes a rarely-needed safety net instead of a perpetually-pinned floor.
Evidence gathered via read-only probes from the monitor box; full transcript in the session that produced #2125/#2132.
Fixed on dev by #2134. The payload stages its slice aggregate into a temp table and joins the plan/query/text catalogs FROM it — real cardinalities instead of TVF fixed guesses, each catalog scanned exactly once, both statements under OPTION(RECOMPILE), the old LOOP JOIN hint gone and pinned absent. Field-validated on the wedged store before merge: staged core 524 ms, the full 55-column batch with plan capture cleared a one-hour backlog in 21.3 s where the old shape never finished inside the 60-second command timeout. Both SKUs and both engine arms through the one shared body. Deploying to the dogfood box next — the acceptance test is watching the wedged members (AppDatabaseOne was 40 consecutive failed cycles deep) converge.
Activity
erikdarlingdata commentedon Aug 8, 2026
Fixed on dev by #2134. The payload stages its slice aggregate into a temp table and joins the plan/query/text catalogs FROM it — real cardinalities instead of TVF fixed guesses, each catalog scanned exactly once, both statements under OPTION(RECOMPILE), the old LOOP JOIN hint gone and pinned absent. Field-validated on the wedged store before merge: staged core 524 ms, the full 55-column batch with plan capture cleared a one-hour backlog in 21.3 s where the old shape never finished inside the 60-second command timeout. Both SKUs and both engine arms through the one shared body. Deploying to the dogfood box next — the acceptance test is watching the wedged members (AppDatabaseOne was 40 consecutive failed cycles deep) converge.
Query Store payload stages its aggregate — the fixed cost big catalog…
Widen query_store's open-interval refresh window: 15min -> 30min