Repository navigation
Web viewer server page: the two daily-summary reads are ~26 of the Overview tab's ~28 s on a busy production primary, one at the viewer role's 15 s timeout — raw CTEs over the whole window plus a per-call rollup-coverage probe #3905
Description
Summary
On the largest production SQL store, the daily-summary pair dominates the web viewer's server page. On the Overview tab, get_daily_summary + get_daily_summary_range were 26.2 s of 28.5 s of the tab's nine reads on a busy primary (19.4 of 20.8 s on a mid-size one). One of them measured 15.8 s against the viewer role's 15 s statement_timeout, so a panel on the landing tab of every server page sits at the edge of erroring.
The other seven Overview reads (summary, properties, CPU, memory, blocking, deadlock trends, analysis findings) total ~1.6–2.3 s. Nothing already filed covers these two: #3895 is the fleet overview and #3896/#3902 are analysis.
Measured (largest SQL store, 43 servers, nightly 458, 2026-09-22)
Through the web viewer's own HTTP surface (GET /api/read/<tool>, authenticated session, from the store host), one tab's reads issued one at a time:
| read | busy primary | mid-size primary |
|---|---|---|
| get_server_summary | 104 ms | 66 ms |
| get_server_properties | 77 ms | 28 ms |
| get_cpu_utilization (24 h) | 185 ms | 174 ms |
| get_memory_trend (24 h) | 257 ms | 237 ms |
| get_blocking_trend (24 h) | 83 ms | 51 ms |
| get_deadlock_trend (24 h) | 16 ms | 15 ms |
| get_analysis_findings | 1,647 ms | 853 ms |
| get_daily_summary | 14,082 ms | 3,611 ms |
| get_daily_summary_range | 12,070 ms | 15,767 ms |
Panels render independently, so the tab completes when its slowest panel does. The run above landed inside the store's hourly refresh window (:18), which inflates it. The same two reads on the busy primary on a quiet minute, over MCP: 3,075 ms and 6,839 ms. So these two reads are slow at the best of times, and at the viewer's timeout at the worst.
Attribution
pg_stat_activity sampled every ~0.3 s from the store host during the quiet-minute pair:
| statement | samples | longest execution seen |
|---|---|---|
DailySummarySql.RangeSql (WITH wait_per_type AS …, Darling/PerformanceMonitor.Darling.Storage/DailySummarySql.cs:33) |
15 | ≥ 4.9 s, waiting on I/O |
the rollup-coverage probe SELECT (SELECT min(bucket) FROM collect.query_stats_hourly), (SELECT min(bucket) FROM collect.query_stats_daily), … (TimescaleSupport.DetectRollupCoverageAsync) |
8 | ≥ 1.7 s |
Two findings from the source and the store:
- Only the
queriesCTE is tier-routed (Darling viewer: built-in tabs silently lose history past 4 days - CAGG read-routing only covers Custom Views #1661,RangeSqlFor). The other eight CTEs (v_wait_stats,v_deadlocks,v_blocked_process_reports,v_dmv_blocking_snapshots,v_cpu_utilization_stats,v_collection_log,v_memory_pressure_events,config_alert_log) always read raw tables for the whole requested window. On the calendar's month span that means every retained raw chunk of eight tables, most of them compressed. For today alone the eight raw sources plus a rawCOUNT(DISTINCT query_hash)total ~0.55 s on the busy primary (measured per source). So the single-day read's multi-second cost is not its own window; it is the second finding. - The coverage probe runs on every call, twice per tab (both daily reads call it), taking
min(bucket)through continuous-aggregate views. Its answer changes at most once a day, as rollups materialize and retention moves. At ≥ 1.7 s on a quiet minute it is most of the single-day read, and it is paid again by the range read on the same tab.
Shape options (owner's call)
- Cache coverage process-wide with a short TTL (the get_fleet_overview: one collection-health scan per minute per host, however many overview calls race (#3735) #3738/get_collection_health per server is an unmemoized 7-day collection_log scan — #3738's single-flight memo covers only the fleet rollup; share the scan or memo the per-server read #3856 memo pattern). Coverage is a slow-moving fact, and both reads plus the calendar can share one answer.
- Route the raw CTEs by tier too. The daily rollups already exist for several of these sources, and a calendar month does not need raw rows for days a rollup has settled. Where no daily rollup exists (collection log, memory pressure), bound those CTEs to their own retention horizon explicitly rather than to the whole window.
- The single-day
get_daily_summaryon the Overview tab could read the range read's newest row instead of issuing its own coverage-plus-aggregate. The tab already fetches both (the source comment atserver-tabs.jsexplains why they are two reads). That's a UI-side saving, independent of the store-side fixes.
Acceptance
- Server Overview tab on a busy production primary completes in under ~3 s on a quiet minute, with no read within 2× of the
viewerrole'sstatement_timeout. - The rollup-coverage probe runs at most once per cache period, not once per daily read.
Activity
erikdarlingdata commentedon Sep 22, 2026
Verified at source (
d21bbc9a) and measured on DARLING01, 9 servers. This is the first attribution done withget_store_query_statsfrom #3899 / PR #3904, deployed there as a dev build.Both claims hold.
queriesCTE is routed (RangeSqlFor). The other eight CTEs read the rawv_*views for the whole window.DarlingHealthReader.GetDailySummaryRangeAsyncprobes coverage on every call, by design. Its comment reads "Probed per call, uncached: get_daily_health runs at human/model cadence and these are two small lookups", and production's ≥1.7 s falsifies the second half of that.Only this reader is uncached; both of its siblings already cache.
ComposeStoreAvailability.GetRollupsAsync(compose panels) caches availability and coverage perNpgsqlDataSourcewith a 5-minute TTL. Coverage expires unconditionally, so a--backfill-rollupsrun is still picked up without a restart.ViewerDataService.RollupAvailability(the desktop Viewer's calendar) does the same, with its own 5-minuteRollupReprobeInterval.The cache half of the fix is therefore to route the daily-summary reader through
ComposeStoreAvailability.GetRollupsAsync. That is a reuse, not a new mechanism, and it keeps this tool and the calendars on the same gate.DARLING01, per statement (pg_stat_statements,
role: mcp, six tool calls):RangeSqlWITH wait_per_type …, range read (month)RangeSql, single-day readSELECT (SELECT min(bucket) FROM collect.query_stats_hourly), …Six probe executions for six tool calls confirms it runs every call.
Why the probe is 1 ms here and ≥1.7 s in production.
EXPLAIN (ANALYZE)shows eachmin()optimized toLIMIT 1over an orderedChunkAppend, so only the OLDEST chunk is touched. But when that chunk is compressed, theColumnarScanseq-scans and sorts the chunk's whole batch list (Seq Scan on compress_hyper_…→Sort) before decompressing the one batch it needs. So each subquery costs the size of its relation's oldest compressed chunk. The probe also carries the rolled RAW tables'min(collection_time), and rawquery_store_statshas the store's heaviest chunks. That fits the "waiting on I/O" in your sampling: it scales with per-chunk volume, not with the number of rollups.The daily summary's month read probes each day once and hashes its qu…
erikdarlingdata commentedon Sep 23, 2026
Closed by the watcher: delivered in PR #3950, merged to dev.
erikdarlingdata commentedon Sep 23, 2026
Measured live on DARLING01 (dev c5be29e), MCP
get_daily_summary_rangemedian of 5: SQL2025 259 ms (before 639 ms) and SQL2022 220 ms (before 369 ms). First calls: 1,194 ms and 482 ms.