The daily summary's month read probes each day once and hashes its query count, and its coverage gate is cached, so the server page's Overview stops waiting on it (#3905)

[erikdarlingdata]

Closes #3905.

Why

On the largest production store the web server page's two daily-summary reads were 26 of the Overview tab's 28 seconds, and one of them ran into the viewer role's 15 s timeout. The issue named two causes. Measurement confirmed one and corrected the other.

• The per-call coverage probe: confirmed. DarlingHealthReader.GetDailySummaryRangeAsync probed rollup availability and coverage on every call, uncached, and the page calls it twice per load. Production measured the probe at ≥ 1.7 s. On DARLING01 it is 1–2 ms of execution plus 12 ms of planning, because each min() touches only the oldest chunk, which is small there.

• "Route the other eight CTEs to a rollup tier": no such tier exists. The aggregates over those sources are the anomaly baseline's per-collection supplies and the fleet card's collection_health_hourly, and none of them answers its CTE exactly:

  • the wait supply sums every wait type into one figure, so there is no top wait;
  • the blocked-process supply carries no wait time, so there is no peak block;
  • the deadlock supply keeps 35 days, less than an operator-lengthened raw retention;
  • collection_health_hourly keeps 8 of the month's 30 days;
  • CPU, memory pressure and the DMV snapshots have no aggregate at all.

  The issue's premise that "the daily rollups already exist for several of these sources" is wrong: daily rollups exist only for the query families.

• What actually made the month read slow was the queries CTE, which is already routed. There were two causes:

  1. The not-carried probes (Brains-review campaign: deferred structural residue (from #3538 / #3539 / #3540 / #3541) #3653 A6). The member that names a day the rollup never materialized was written as a bare EXISTS and NOT EXISTS, and PostgreSQL pulls both up into a semi-join and an anti-join. Neither join can push its per-day range into the scan, so the planner materialized the server's entire history in the probed relation (four days of raw on the hourly tier's source probe, the hourly's 90 days on the daily tier's) and rescanned it once per day of the window. Cost grows with days × retained rows, and past work_mem it spills and rereads a temp file per day. On DARLING01's busiest server this member was 629 of 873 ms. On the rig, seeded with one busy server, it was 4.5 s of a 6.1 s month read, with 32,678 temp blocks read. The comment above it promised "one index range per day"; the plan did not deliver that.
  2. COUNT(DISTINCT query_hash). PostgreSQL evaluates a DISTINCT aggregate by sorting every input row of its group. For a busy server's month of hourly rollup rows that sort was 1.1 s of the rig's remaining time.

What changes

• DailySummarySql, results identical on every tier:

  • Each not-carried probe carries OFFSET 0. PostgreSQL's simplify_EXISTS_query refuses to pull up a sub-select with an OFFSET, so each probe stays a SubPlan that runs once per candidate day: an index range with run-time chunk exclusion.
  • The candidate days are fenced the same way, so the source is probed only on the days the rollup probe found empty. On a covered window that is none.
  • Every tier counts the distinct (day, query_hash) pairs instead of using COUNT(DISTINCT). COUNT skips a NULL hash exactly as the DISTINCT aggregate does, and the planner can hash a SELECT DISTINCT.
  • The doc records why no other CTE is routed.

• DarlingHealthReader reads coverage through ComposeStoreAvailability (via GetMeasuredRollupsAsync, below) instead of probing on every call. That is the gate the compose panels already read, and the one Erik's comment pointed at: cached per data source for ReprobeInterval (5 min). The viewer's calendar has cached the same gate for the same interval all along.

  • Why a cached floor cannot produce a wrong number:
    • A floor moves back only on a backfill. A stale floor then under-claims, so a window stays where it was or degrades.
    • A floor moves forward only on a retention drop. RetentionTierRouter's one-day RouteMargin keeps every age-routed window clear of that.
    • The ceiling is read live inside the statement.
    • A day a tier never carried is named NULL either way.
  • A cached failure is different, so the reader never routes on it. A failed shared probe is cached as RollupAvailability.None, which routes every window to raw. That is the compose panels' accepted fallback. But on a store whose raw purges are armed, it would print unique_queries = 0 for every purged day of the calendar: not NULL, and not in days_missing. Before this change, the per-call probe failed the read instead.
    • The cache now marks a failed answer.
    • The reader goes through GetMeasuredRollupsAsync, which probes again on its own token rather than route on a cached failure, so a second failure surfaces as the tool's error, as before.
    • A healthy store's answer is always the cached one.

• ComposeStoreAvailability is now single-flight. Agents race their daily reads, and the compose panels, trend tools and custom alerts read the same gate. It used to be "benignly racy": every stale caller probed. Now one probe runs per data source at a time, with the DarlingFleetReader.CollectionHealthMemo shape (get_fleet_overview's collection-health rollup times out at :35 on the largest store — phase-grid signature, discriminate compression-band contention vs plan before fixing #3735):

  • the shared probe runs on CancellationToken.None, so a caller's token releases only that caller's wait;
  • a failure is still the cached raw fallback, as before.

  That gives "at most once per cache period" literally, and it also covers the compose panels, custom alerts and trend tools that read the same gate.

• Web server page, Overview tab: today's tile and the Daily Health Calendar now render from one fetch of get_daily_summary_range. The tile is the row whose date is the read's own to_date, and there is no separate get_daily_summary read.

  • Behavior change, stated: the tile's Unique queries for today is now the range's count. When the month routes to a rollup, that count comes from the rollup's materialized hours, where the old tile counted raw, so it can trail raw by the hourly refresh lag (one to two hours of newly seen queries).
  • The grid's last row always showed this number, so the tile and the row below it now agree instead of possibly differing on one page.
  • Every other tile figure is unchanged, because both reads computed today's signals from raw.
  • Each panel renders through its own guard, the way fanout() guards its specs, so a render fault in one cannot leave both on their loading strips.
  • For an empty range the tile says nothing has been collected today, instead of the range's MCP-parameter advice.
  • The tile now waits on the month read. A month-read failure shows in both panels.

• MCP tools: no payload, description or parameter change. get_daily_summary and get_daily_summary_range get the faster statement and the cached gate through the reader.

• WPF viewer: it reads the same DailySummarySql, so its calendar gets the statement fix. Its gate was already cached.

• Lite parity: nothing to change. Lite's calendar reads raw DuckDB: no rollup tier, no not-carried member, no coverage probe. DuckDB hashes COUNT(DISTINCT) natively, and Lite's calendar already reads one range per month.

• No store migration.

Test plan

• DARLING01 (9 servers, 17 days; EXPLAIN (ANALYZE, TIMING OFF), median of 3; old and new statements interleaved in one session):

  server	read	before	after
  SQL2025 (busiest query_stats)	month, hourly tier	729 ms	253 ms
  SQL2025	120 days, daily tier	610 ms	324 ms
  SQL2025	today, raw	79 ms	80 ms
  SQL2022	month	431 ms	252 ms
  SQL2022	120 days	434 ms	265 ms
  SQL2016	month	362 ms	200 ms
  SQL2016	120 days	272 ms	223 ms

  The not-carried member alone on SQL2025's month read went from 629 ms to 1.4 ms. What remains, about 200 ms, is the raw wait, collection-log and CPU CTEs, which have no tier (see Why).

• Rig at production scale for one server (5.3M raw rows seeded over 45 days, 571K retained over 4 days, 881K hourly rollup rows; work_mem 16 MB; median of 3):

  read	before	after
  month, hourly tier	3,278 ms (6,233 cold)	186 ms
  120 days, daily tier	12,359 ms (15,927 max)	161 ms
  today, raw	146 ms	45 ms

• MCP, DARLING01, the deployed dev build (before), median of 5: get_daily_summary_range 639 / 372 / 369 ms and get_daily_summary 137 / 79 / 80 ms for SQL2025 / SQL2022 / SQL2016. The "after" needs the deploy.

• New tests:

  • DailySummaryReadShapeSqlTests: no tier uses a DISTINCT aggregate; the reader routes on the cached gate; the Overview tab draws both panels from one fetch.
  • ComposeStoreAvailabilitySingleFlightTests:
    • five racing callers share one probe, which runs on no caller's token;
    • a cancelling caller releases only itself;
    • a failed probe answers the compose accessor with the raw fallback and is cached;
    • the measured accessor re-probes on its caller's token instead of routing on that fallback: it uses a success without writing it back, and surfaces a second failure;
    • a measured answer is served from the cache.
  • DailySummaryReadShapeLiveTests (rig):
    • oracle: the pre-Web viewer server page: the two daily-summary reads are ~26 of the Overview tab's ~28 s on a busy production primary, one at the viewer role's 15 s timeout — raw CTEs over the whole window plus a per-call rollup-coverage probe #3905 statement against the new one, row for row, on raw, legacy hourly, interval-honest hourly and daily. The seed covers a skipped day, an empty day, a NULL hash beside real ones, a NULL-hash-only day (still emitted, count 0), a restart-only day and raw tails.
    • plan: no Materialize and no semi/anti join in the queries CTE. The rollup probe runs once per candidate day (27) and the source probe once per rollup-empty day (23).
    • seam: six raced and two sequential calls through the MCP tools cost one coverage probe.

• Red on the old shape: with DailySummarySql.cs and DarlingHealthReader.cs reverted, the plan test, the probe-count seam test and both SQL pins fail. On the tiny seed the plan test sees the old semi-join plus Materialize with loops=27, and the seam test counts 0 probes.

• Existing pins updated deliberately (text, not intent): DailySummaryNotCarriedTests (probe texts, plus assertions that the fences are present and in the right place), RetentionTierRouterTests, IntervalHonestHourlyRollupTests, ViewerDailySummarySqlTests, ServerPageTabsTests (composite map), DarlingMcpHealthToolsTests (comment).

• Overview tab in a node DOM shim, six response states:

  • one /api/read/get_daily_summary_range fetch;
  • the tile renders the to_date row;
  • a range without today says nothing has been collected yet;
  • empty gives the tile its own sentence;
  • unavailable and an error reach both panels;
  • an empty body renders both panels' empty states rather than hanging.

• Independent code review (subagent) before opening: one should-fix, the cached-failure fallback, fixed as above; four nits, fixed (stale comments, per-panel render guards, the tile's empty sentence, the NULL-hash-only seed day).

• Full Darling.Tests against the rig with DARLING_TEST_PG: 12,535 total, 0 failed, 22 skipped (the env-gated managed-runtime, PostgreSQL-target and symlink tests), on the branch rebased onto dev at 9f476174

• After deploy: rerun the MCP timing on DARLING01, and the production Overview tab.

Filed from this: #3933. The start-up materialization-hole scan has the same pulled-up-EXISTS shape, and it runs at every service start.

🤖 Generated with Claude Code

https://claude.ai/code/session_018Z4T51J4E6L8fjriPHpsif