Top-N-by-CPU routes to the hourly rollup once raw's floor ages past the window (#4231)

[erikdarlingdata]

Refs #4231.

Why

get_top_queries_by_cpu (and its Storage/Viewer counterparts) only ever read raw query_stats. Once retention drops raw's chunks for an older window, the read returns nothing for that window even though the hourly continuous aggregate still holds a summed answer — the same class of silent-empty-window bug #1937/#1759 fixed for other readers. This wires the top-N-by-CPU family into the standing RetentionTierRouter + RollupCoverage gate so it degrades to the hourly rollup instead of going empty, and discloses the precision loss (tier_used, precision_note) rather than hiding it.

What changes

• DarlingDataReader.TopQueriesHourlySql (new const): the hourly-tier twin of TopQueriesSql, groups by (database_name, query_hash) only (the rollup carries no host_object_name), ranks by SUM(worker_time_sum) DESC. FROM clause is a $FROM$ placeholder, substituted only via RollupCoverage.StitchedRelationSql — never a literal relation name (the standing gate).
• GetTopQueriesByCpuAsync now delegates to GetTopQueriesByCpuRoutedAsync, which resolves the tier via RetentionTierRouter.Resolve over RollupCoverage.For(QueryStatsHourlyView, QueryStatsDailyView). A Daily verdict is clamped to Hourly (daily top-N is out of scope here).
• TopQueriesHourlyTextLookupSql: one follow-up query per ranked hourly row, resolving a representative query_text from v_query_stats (the rollup has none).
• DarlingMcpDataTools.GetTopQueriesByCpu adds tier_used ("raw"/"hourly") and precision_note (set only when hourly) to the MCP payload; skips the raw-floor probe entirely on the hourly path (an hourly-routed read never touched query_stats).
• ViewerDataService.GetTopQueriesByCpuTierAsync mirrors the same routing for the WPF Viewer; ViewerServerTab.Queries.cs appends an "— aggregated hourly, per-caller detail unavailable" note to the truncation banner when the tier is hourly.
• TopQueriesHourlyRoutingTests.cs (source-level, no live store): the SQL shape and the never-names-the-relation-directly source pin.
• TopQueriesHourlyRoutingLiveTests.cs (new) — the live equality/routing pin described below.

Test plan

• TopQueriesHourlyRoutingTests.cs (source-level): both green in-process on this rig.
• TopQueriesHourlyRoutingLiveTests.cs (live, own scratch database via ScratchPostgres/#1776 own-store):
  • Ran the real xUnit tests in-process on macOS (dotnet Darling.Tests.dll -class Darling.Tests.TopQueriesHourlyRoutingLiveTests, WindowsDesktop entry stripped, own pmw-4231-3a-debug TimescaleDB 2.30.1 container, removed after). Both tests in the file now pass: Total: 2, Errors: 0, Failed: 0.
  • Root cause of an earlier in-process failure: a stale ComposeStoreAvailability cache, not the router itself. RetentionTierRouter.Resolve is correct as designed — the test's premise (a fixed calendar WindowStart, real DateTime.UtcNow, delete the window's raw rows so it ages past RawMaxAge, 3 days) is sound. The actual cause: ComposeStoreAvailability's answer cache is keyed per NpgsqlDataSource instance, with an unconditional 5-minute ReprobeInterval TTL. Both tests reused ONE NpgsqlDataSource for both the pre-purge (raw) and post-purge (hourly) routed calls. Call 1's probe ran and cached TierCoverage with a null hourly floor — measured before RefreshAsync materialized the successor over the window — and that cached answer is still fresh (well inside 5 minutes) when call 2 runs milliseconds later on the SAME data source. The router's coverage-degrade path (RetentionTierRouter.Resolve(... TierCoverage)) sees the age-picked Hourly tier fail TierCoverage.Covers against the stale null floor and falls back toward Raw. Fix: both tests now open a FRESH NpgsqlDataSource for the post-refresh/post-purge (hourly) call, so its coverage probe measures the store's true post-refresh state. This is a genuine cache-freshness gap in ComposeStoreAvailability — a live production store's routed reads could see the same up-to-5-minutes-stale answer right after a backfill or CAGG refresh finishes. Not fixed in the composer itself here — recorded as a finding; the fix belongs in ComposeStoreAvailability's own single-flight/TTL design, not in a live test.
  • Second, independent seed bug found and fixed in the same pass: the DOP/rollup pin (GetTopQueriesByCpu_HourlyRouted_WithDopOrRollUp_CarriesPrecisionNote) still failed after the cache fix, on Assert.True(rawDoc.RootElement.TryGetProperty("tier_used", ...)). Cause: PlantAsync's seed rows never set max_dop (left NULL); the raw call's min_dop=2 filter is HAVING COALESCE(MAX(max_dop), 0) >= $6, so a NULL max_dop fails the floor, the raw call's population filters to zero rows, and the tool returns the "empty" status payload — which carries no tier_used/precision_note field at all, hence the TryGetProperty failure. Fixed by giving PlantAsync an optional maxDop parameter and seeding this test's two rows with maxDop: 2, matching the pin's own stated intent (an hourly-routed payload "with min_dop set"). Both tests in the file are green after this fix.
  • RED on dev, re-verified directly (real test file, not a harness): git worktree add --detach /tmp/pmw-4231-3a-debug-red bb04415cd (dev's tip at merge-base with this branch), copied TopQueriesHourlyRoutingLiveTests.cs in, built Darling.Tests.csproj there — compile failure: CS0117: 'DarlingDataReader' does not contain a definition for 'GetTopQueriesByCpuRoutedAsync' at both call sites — the routed method doesn't exist on dev. Worktree removed after.
  • Windows suite (unchecked, CI decides): the xUnit test targets net10.0-windows and cannot run natively on macOS; the in-process run above uses the WindowsDesktop-stripped runtimeconfig workaround, which is not CI's real execution path. CI runs it for real on Windows.
• Darling.Tests builds 0 errors / 0 warnings on this branch (-p:EnableWindowsTargeting=true).
• Not measured: re-measurement at the seed's scale after the cache fix — the existing small-seed measurement table below is unchanged from an earlier version of this branch.

Known limits

• DOP / rollUpByHostObject on the hourly path: an hourly-routed read silently ignores minMaxDop and rollUpByHostObject — the rollup has no per-group DOP column and no host_object_name to roll up by. Disclosed via precision_note, not erroring.
• Viewer's hourly SQL is a private inline string, not a shared const — matches the pre-existing split between Service and Viewer SQL text.
• The raw-floor probe (GetQueryStatsWindowFloorAsync) only runs for the raw tier — an hourly-routed read never touches query_stats, so window_truncated=false / effective_start=requestedStart is reported unconditionally there rather than re-probing a table the read never opened.
• Zero-interval-row difference (confirmed real, not corrected here): raw's TopQueriesSql counts a sample_interval_seconds = 0 first-collection row; the hourly rollup's CREATE bakes in IntervalHonestSourceFilter (sample_interval_seconds IS DISTINCT FROM 0) and excludes it. On the live pin's seed this is a 900,000 CPU-us / 1-execution difference for that one query_hash — invisible unless that hash happens to rank in the top-N, since every OTHER group's raw and hourly totals agree exactly. Raw's own read is unchanged here; whether raw's read should apply the same filter is a separate design question.

Measurement

Own seeded rig: 20 servers × 7 days × 96 fifteen-minute cycles × 5 query groups (67,200 raw rows), one bulk INSERT ... generate_series, on a fresh TimescaleDB 2.30.1 container (own pmw-4231-3a-pins container, removed after the run). Reads are for one sample server's full 7-day window, top 20.

	BEFORE (raw, dev's unchanged read)	AFTER (this branch, routed to hourly, incl. text lookup)
Wall-clock, cold	70 ms	56 ms
Wall-clock, 3 warm runs	8 / 8 / 7 ms	6 / 8 / 7 ms
Wall-clock, warm median	8 ms	7 ms
EXPLAIN (ANALYZE, BUFFERS) — ranking query, Execution Time	8.396 ms	0.607 ms
EXPLAIN (ANALYZE, BUFFERS) — largest single Buffers line	shared hit=8375	shared hit=285

At this seed's scale (well short of the issue's field population of 4,078 ms / 162.9k buffers at 7 days) both tiers already answer in single-digit milliseconds on a local container, so the wall-clock gap is not yet the dominant signal — the buffer-count gap is: the hourly rollup's ranking query touches roughly 30x fewer shared buffers than raw's equivalent scan (285 vs 8375 largest line), which is the shape that would separate BEFORE and AFTER by seconds at the field's row counts. The N text lookups (one per ranked row, 5 rows here) added no measurable wall time at this scale — well under a 10% threshold — so the N+1 collapse described below was not triggered; a store with a much wider top-N or a slower text-dimension table could still see it matter, and is left as-is (collapse it only if it costs more than about 10%).

CHANGELOG entry

SECTION: Changed
ENTRY:

• Top-N-by-CPU queries now route to the hourly rollup once raw's retention has dropped the window, instead of returning empty ([Top-N-by-CPU routes to the hourly rollup once raw's floor ages past the window (#4231) #4396]) - get_top_queries_by_cpu (MCP tool, Storage reader, and Viewer) degrades to the hourly continuous aggregate for a window older than raw's floor and discloses the precision loss (tier_used, precision_note) rather than silently returning nothing.
  REF:
  [Top-N-by-CPU routes to the hourly rollup once raw's floor ages past the window (#4231) #4396]: Top-N-by-CPU routes to the hourly rollup once raw's floor ages past the window (#4231) #4396