The interval-hourly refresh recomputes one fleet-wide bucket per dirty hour and paid twelve index inserts per row for eleven indexes nothing reads; the capture-down alert read decompressed a server's whole collection_log to learn two statuses (#3597, partial)

[erikdarlingdata]

The question the issue asked first, answered on a rig

What does each hourly refresh of query_store_stats_interval_hourly actually recompute? Measured on a Docker timescale/timescaledb:2.28.1-pg18 (PostgreSQL 18.4 / TimescaleDB 2.28.1 — the production pair), store built by the product's own path (PgMigrations.MigrateAsync → TryEnableAsync → ConvertToHypertablesAsync → EnsureContinuousAggregatesAsync → PgTableTuning.ApplyAsync), seeded to a 1/10 model of the largest production store: 43 servers × 840 interval identities per server per hour × 6 five-minute snapshots = 216,720 raw rows/hour → 36,120 L1 rows/hour, 26 hours, 5.7 M raw rows, 5.3 GB. Every refresh below is the policy's exact window, [now − 1 day, now − 1 hour], run through CALL refresh_continuous_aggregate with client_min_messages = debug1 so TimescaleDB printed the ranges it refreshed and the rows it deleted and inserted.

step	what was written since the last refresh	refreshed range(s)	rows deleted / inserted	wall clock
r0 initial materialization	22 buckets never materialized	3 batches covering 22 buckets	0 / 794,640	31.4 s (938 MB WAL, 955 MB temp)
r1 steady state, hour rolled	live rows only	the newly closed bucket(s): 2 (the hour turned mid-run)	0 / 72,240	2.9 s
r2 steady state, no new rows	nothing	"already up-to-date"	0 / 0	0.0 s
r3 ONE backdated row, 20 h old	1 row, collection_time = now − 20 h	the whole 20-h-old bucket, all 43 servers	36,120 / 36,121	0.8 s
r4 12 backdated rows, one transaction, hours −2…−13	12 rows	one merged range [−13 h, −1 h): 12 buckets, clean hours included	433,440 / 433,452	17.3 s
r5 5 backdated rows, 5 transactions, hours −3,−6,−9,−12,−15	5 rows	5 individual buckets	180,604 / 180,609	7.4 s

Verdict. TimescaleDB refreshes exactly the buckets the invalidation log marks dirty, and the window's width is not the cost (r2: 22-bucket window, nothing dirty, 0 s — #3012's "a refresh's cost is set by the WINDOW it re-scans" is false at the mechanism level; its measured 6,330 → 864 s win came from shrinking the backfill's reach, which QueryStoreBackfill.RollupStoreHorizon derives from the window, i.e. from cutting the dirty-bucket count). Each hour costs (1 + k) × one bucket, where k is the number of distinct past hour-buckets any backdated row landed in since the last run — and a one-row backdated write costs the entire fleet's hour (r3), because invalidation is by time only, never by server. A single write transaction spanning hours dirties every hour between (r4). So:

• (B) is the multiplier and it is real. The one writer of backdated rows is QueryStoreBackfill, by design: WriteBackfillBatchAsync(..., ceilingUtc, ...) stamps every slice's rows at the slice CEILING, up to 23 h back (RollupStoreHorizon), so the rows land beside their own activity (Query Store phase 2: newest-first backfill worker for the history the live path never takes #2022's horizon contract, the BLOCKED: narrowing the hourly CAGG refresh window would reintroduce #1759 — the window is a self-healing repair loop, not just a cost #3100 repair-loop argument). The brief's VERIFIED claim that QueryStoreBackfill.cs:322 stamps CollectionTime = DateTime.UtcNow is false as a description of the row stamp — that is the CollectorContext clock; the write uses ceilingUtc. Nothing stamps rows incorrectly; every backdated row is backdated on purpose, which is why lever 1 as written ("fix the writer's timestamps") is not a lever anyone can pull without a product decision about where backfilled history lands. Each backfill slice (one per server per 5-minute tick, holes and first-contact tails) dirties one bucket; a first-contact tail walks down hour by hour and dirties up to 23; a clamp hole (WatermarkPolicy.MaxCatchup) opened by a slow Query Store pass dirties the hour it was clamped at. That k rises with collector load is the shape the issue measured as "breathes with load".
• (A) is the per-bucket cost, and it was carrying something nothing reads. The steady-state floor is one bucket per hour; on the rig 1.0–1.7 s at 1/10 scale, everything in memory. Production's trough is 276–286 s — for what is, per the rig, one bucket of ~360 K L1 rows from ~4 M raw. That is where the unexpected cost lives, and the rig found it: TimescaleDB's default create_group_indexes had built eleven (column, bucket DESC) btrees on L1's materialization beside the bucket index — twelve indexes on the one aggregate in the file whose cardinality is near-raw. Nothing reads L1 by any of them: its three children (corrected_hourly, corrected_daily, interval_daily) refresh over it by bucket range, RollupCoverageProbeSql / RetentionArmSafetySql read min(bucket), retention drops chunks (grep: no product read names the view outside TimescaleSupport). What they did was tax every re-materialization: EXPLAIN (ANALYZE, BUFFERS, WAL) of one bucket's materialization INSERT (36,121 rows from 216,721 raw, isolated in a rolled-back transaction):

L1 materialization indexes	WAL	WAL records	buffer touches	buffers dirtied	execution
12 (default: 11 group + bucket)	45.5 MB	483,689	1.64 M	6,040	1.48–1.87 s
1 (bucket only)	10.7 MB	72,735	447 K	56	1.24 s

4.3× the WAL, 1.19 M extra buffer touches and 108× the dirtied pages per bucket, from twelve index inserts per row where one suffices. On the rig those touches are shared-buffer hits and the wall clock barely moves; on a store whose L1 materialization is 71.5 GiB they are the leaf pages of eleven cold high-cardinality indexes (query_hash, first_execution_time, query_id, plan_id, …), random-read on every insert — the I/O the issue's alert reads were starved by. The rig can measure the amplification; it cannot reproduce the production I/O regime, so this PR claims no production duration figure. The DELETE half is cheap either way (36 K rows, 16 ms, 2 MB WAL). The other (A) suspect — the 12-column aggregate spilling — is real (external merge sort, 38 MB + 14 MB temp per bucket at work_mem = 4MB; ~520 MB per production bucket) but work_mem = 512MB did not move the rig's wall clock measurably (0.9–1.3 s vs 1.1–1.3 s), and a policy job's work_mem is not settable per job in 2.28 (alter_job(config) carries only the policy's own keys) — reported, not pulled.

What ships

1. L1 without the group indexes (the (A) lever the rig licensed). CreateQueryStoreStatsIntervalHourlySql is now WITH (timescaledb.continuous, timescaledb.create_group_indexes = false); that option speaks only at CREATE, so TimescaleSupport.EnsureIntervalDedupMaterializationIndexesAsync (#3620's converge shape) brings an existing store to the same state at startup: resolves L1's materialization from the catalog by view name, selects its indexes by shape (USING btree (<one column>, bucket DESC) — the bucket index cannot match), drops each in its own transaction under SET LOCAL lock_timeout = '10s', logs each drop with the bytes it released across the hypertable and its chunks, and stops at the first lock timeout so a refresh in flight is yielded to rather than convoyed (a queued AccessExclusiveLock blocks every later shared request — the #3012 convoy). Idempotent under the catalog: a settled store issues nothing. Wired in DarlingWorker after EnsureContinuousAggregatesAsync and before EnsureAggregateCompressionAsync, so #3611's nightly pass compresses a relation already without them. Verified on the rig through the product code: 11/11 dropped (109.3 MiB at rig scale), chunk copies gone, second start a no-op; with a ROW EXCLUSIVE lock held on the materialization the drop yielded at 10.0 s with the Warning and succeeded on the following start; the refresh, all three child refreshes, compress_chunk and decompress_chunk on an L1 chunk all ran unchanged with the bucket index alone. Scoped to L1 only: the composer-grain rollups ARE read through their group indexes (server_id, query_hash), and the day-grain L2 refreshes once a day and was not measured — the scope pin says so.

2. The capture-down read (lever 3). DarlingSelfAlertEvaluator.ReadMissingCaptureSessionsAsync shipped as ROW_NUMBER() OVER (PARTITION BY collector_name ORDER BY log_id DESC) over every collection_log row the server had for deadlocks and blocked_process_report — the shape #3496 named and deliberately left ("a window function cannot early-stop by ordering swap alone"). With 60 days of one server's log across 61 chunks (59 compressed), realistic cadences, its plan on the rig: every chunk decompressed, 103,682 rows Merge-Appended and numbered to keep two, 9,573 buffers, 76 ms + 62 ms planning — two to three orders of magnitude above every other per-server read on the pass, per alert pass, every 30 s, per server. It is now one chunk-orderable LIMIT 1 per collector (ORDER BY cl.collection_time DESC, two literal arms, UNION ALL): 14 buffers, the newest chunk only, 120 of 122 chunk scans "never executed", horizon-independent. No log_id tiebreak, and the doc says why: each arm is one collector on one server, one row per run stamped DateTime.UtcNow at log time — no tie is possible — and the tiebreak's cost was measured (the (server_id, collection_time) index cannot serve the second key, so each arm top-N-heapsorts the whole newest chunk: 1,158 buffers vs 5). The SQL is hoisted to MissingCaptureSessionsSql so the pins read the text the method executes.

3. Every other alerting-side read, EXPLAINed on the same rig with 60–90 days of compressed history, so the issue's "find out what the read is doing" has an answer per site: latest-CPU 21 buffers / 0.5 ms; volume-free-space 29 buffers / 6.5 ms, 90 of 91 chunks excluded at runtime — but 90 ms of PLANNING (the InitPlan MAX prevents plan-time exclusion, so the planner builds 182 chunk paths); pvs-pressure 99 buffers / 5 ms / 104 ms planning (same shape); tempdb 3 / 0.4 ms; anomalous-jobs 39 / 1.6 ms; database-file-growth 5 / 0.05 ms; force-plan-failures (#3586) 63 / 13 ms Index Only Scan; the five database-states statements ≤ 71 buffers / ≤ 3 ms; collection-health (#3496) 2,313 buffers / 4.7 ms / 70 ms planning; agent reads 1–3 buffers. Volume-free-space and pvs are already at execution the shape the brief asks for — the newest chunk, tens of hot buffers; what they cost is planning over a 91-chunk hypertable, and the only cure is a plan-time-excludable collection_time > $2 bound, which is a freshness decision the engine does not make today: an empty result on a stale snapshot would NotifyResolutionAsync("All volumes back above threshold") — a fabricated recovery, the exact failure the alert layer is designed never to produce. Declined here; reported as a residual with the numbers (43 servers × ~330 ms of planning per pass ≈ half a core, continuously).

Levers declined, with why

• Lever 1 (fix the writer's timestamp). The stamp is deliberate and load-bearing (Query Store phase 2: newest-first backfill worker for the history the live path never takes #2022/BLOCKED: narrowing the hourly CAGG refresh window would reintroduce #1759 — the window is a self-healing repair loop, not just a cost #3100); changing where backfilled history lands is a product decision, not a bug fix. What the rig adds to that decision: the cost of a backfill slice to the refresh is independent of its row count — one row or ten thousand, it is one fleet-wide bucket — so the lever, if ever pulled, is the number of distinct hours a tick's slices land in, not their volume.
• Lever 2 as listed (slower cadence / split by server range). Cost is per dirty bucket, not per window or per run; a 2-hourly cadence halves the contention windows and doubles each one's dirty count for the same total I/O, and a server-range split cannot narrow an invalidation that is by time only. Not licensed by the evidence.
• work_mem for the policy job. Spill measured, effect on the rig not measurable, knob not available per job. Reported.

What this does NOT do

It does not claim to have removed the 417 s. It removes the measured write amplification from every re-materialized bucket and takes the largest read off the alert pass; it leaves the dirty-bucket multiplier — the backfill's designed behaviour — exactly where it was. The post-install job_history series for this refresh is the measurement that says how much of the average was per-bucket cost: the trough (276–286 s, when only the newly-closed bucket is dirty) is the number this should lower. Partial, so no Closes. The diagnostic with the production statements that count k is posted on #3597.

Tests

• IntervalDedupMaterializationIndexesTests — pins create_group_indexes = false on L1 and its ABSENCE on every other registered aggregate (hourly, daily, baseline registries — a new aggregate is covered the day it is registered); the catalog read's by-shape selection on L1 only; the bounded lock timeout; and the gated live arm on a scratch database: a fresh L1 has exactly the bucket index (TimescaleDB honours the option), a planted pre-Alert reads die during the interval-hourly Query Store refresh: 26/26 runs over four minutes, 417 s average, breathing with load — the grid protects writers from locks, nothing protects readers from I/O #3597 group index is found and dropped (1 → 0 on the next call), the bucket index survives by shape, and the refresh plus the child corrected hourly still materialize seeded rows correctly (10 identities deduped to their last snapshot, 300 executions).
• CaptureDownChunkOrderTests — source pins (two arms, each ORDER BY cl.collection_time DESC LIMIT 1; no OVER (, no ROW_NUMBER, no log_id) and the gated plan test modelled on ForcePlanFailuresAccessPathTests: eight seeded days across eight chunks, EXPLAIN of the shipped statement with its real bound parameter → no WindowAgg, ≤ 1 chunk executed per arm, and the NEWEST run decides (a SESSION_MISSING three days ago followed by a success is not reported; a success followed by SESSION_MISSING is).
• CollectionSignalsChunkOrderTests doc updated: the capture-down read it left alone is now pinned by its own class.
• Executed locally via the mactest pattern against the rig (DARLING_TEST_PG → the TimescaleDB 2.28.1 / PG 18.4 container): 651 green across the two new classes plus DarlingSelfAlertTests (its live capture-down arm included), AlertPassCommandTimeoutTests, TimescaleContinuousAggregateTests, TimescaleSupportTests, TimescaleAggregateCompressionTests, ForcePlanFailuresAccessPathTests, QueryStoreTrendRoutingLiveTests, QueryStoreCorrectedRollupLiveTests, RollupBackfillLiveTests and more (the only failures were harness artifacts: out-of-tree RepoFile paths and Windows-only upgrade tests). Full solution builds -c Release -p:EnableWindowsTargeting=true with 0 warnings.

[claude]

Reviewed the diff (Darling-only: DarlingSelfAlertEvaluator.cs, TimescaleSupport.cs, DarlingWorker.cs, plus new/updated tests).

• Capture-down read rewrite (ROW_NUMBER() OVER (...) → two ORDER BY collection_time DESC LIMIT 1 arms, UNION ALL): semantically equivalent to the old window-function query (same collector set, same SESSION_MISSING filter, same friendly-name mapping at DarlingSelfAlertEvaluator.cs:4547), parameterized ($1), no injection surface. The lack of a log_id tiebreak is justified and matches the actual data model (one row per collector run, microsecond-stamped).
• create_group_indexes = false + EnsureIntervalDedupMaterializationIndexesAsync: catalog-driven by shape (not by name), scoped correctly to the one materialization, idempotent, per-index transaction isolation with a bounded lock_timeout that correctly breaks the loop on lock contention (leaving remaining drops for the next start) vs. continuing past other per-index failures (permissions, etc.) — matches the documented intent. DROP INDEX IF EXISTS + identifier quoting is safe.
• Ordering in DarlingWorker.cs: placed after EnsureContinuousAggregatesAsync and before EnsureAggregateCompressionAsync as documented, gated behind _timescaleAvailable, runs once at startup — no hot-path perf concern.
• Lite/Darling parity: this touches only the Postgres/TimescaleDB store and Darling's own self-alert evaluator — there's no Lite (DuckDB) or Full Dashboard counterpart for continuous-aggregate materialization indexes or this capture-down read, so no drift here.
• Naive-UTC / schema-qualification conventions from CONTRIBUTING.md are respected throughout (no DateTime.Kind binding issues in the new test seeding code; all identifiers resolved via catalog/constants, nothing hard-coded to an internal Timescale name).

No correctness, security, or parity issues found. Did not run the live-Postgres-gated tests myself (no DARLING_TEST_PG rig here) — relying on the PR's own reported 651-green run against the same TimescaleDB 2.28.1/PG 18.4 pair.

[claude]

LGTM — reviewed the capture-down read rewrite, the create_group_indexes=false change plus its startup converge, and worker ordering; no correctness bugs, injection risk, or Lite/Darling parity drift found (change is Darling/TimescaleDB-only with no Lite counterpart needed).

[claude]

Non-blocking observation: this selects indexes to drop purely by shape (USING btree (<one column>, bucket DESC)), not by any marker that ties them to create_group_indexes. If an operator (or a future migration) ever hand-creates an index on this materialization that happens to match this shape — e.g. (query_hash, bucket DESC) for some ad-hoc diagnostic — EnsureIntervalDedupMaterializationIndexesAsync will silently drop it on the next service start. Given this is TimescaleDB's own internal _materialized_hypertable_N relation that nothing in this codebase indexes by hand today, the risk is low, but it's worth a one-line acknowledgment in the doc comment (which already documents several other edge cases) so a future maintainer adding a legitimate index here knows to steer clear of this shape or update the selector.

[claude]

Non-blocking: dropped++ and freed += bytes fire whenever DROP INDEX IF EXISTS completes without error, including the (very unlikely) race where the index was already gone by the time this runs — IF EXISTS makes that a silent no-op rather than an error. The summary log line would then over-report what this start actually freed. Harmless given this only runs once per service start against a store this service itself owns, but since the surrounding code is otherwise careful to make every logged number accurate (the byte-size comment above even calls out where the read side undercounts), it might be worth the same rigor here — e.g. check the command's affected/notice output, or just note in the log message that the count is "attempted" rather than "dropped."

[claude]

LGTM — reviewed the Darling-only TimescaleDB change (query_store_stats_interval_hourly's group indexes, the interval-dedup index converge on startup, and the capture-down alert read's rewrite from a windowed scan to a chunk-orderable LIMIT 1 per collector). Checked: SQL injection surface (catalog-derived identifiers are quoted via QuoteIdentifier, view names are compile-time constants), lock-timeout/transaction handling in EnsureIntervalDedupMaterializationIndexesAsync (per-index transactions, bounded lock_timeout, correct break-on-first-lock-timeout semantics), the rewritten capture-down query's semantic equivalence to the old ROW_NUMBER() version (verified against live-gated tests and index support), startup ordering (drop-before-compress, after EnsureContinuousAggregatesAsync), and Lite/Darling parity — this is TimescaleDB/PostgreSQL-specific plumbing with no Lite (DuckDB) or deprecated-Dashboard counterpart, so no parity gap. No T-SQL touched. Left two non-blocking inline notes (shape-based index selection risk, a cosmetic drop-counter edge case) — neither is a functional bug.