Skip to content

Web viewer server page: the two daily-summary reads are ~26 of the Overview tab's ~28 s on a busy production primary, one at the viewer role's 15 s timeout — raw CTEs over the whole window plus a per-call rollup-coverage probe #3905

Closed

Description

@erikdarlingdata

Summary

On the largest production SQL store, the daily-summary pair dominates the web viewer's server page. On the Overview tab, get_daily_summary + get_daily_summary_range were 26.2 s of 28.5 s of the tab's nine reads on a busy primary (19.4 of 20.8 s on a mid-size one). One of them measured 15.8 s against the viewer role's 15 s statement_timeout, so a panel on the landing tab of every server page sits at the edge of erroring.

The other seven Overview reads (summary, properties, CPU, memory, blocking, deadlock trends, analysis findings) total ~1.6–2.3 s. Nothing already filed covers these two: #3895 is the fleet overview and #3896/#3902 are analysis.

Measured (largest SQL store, 43 servers, nightly 458, 2026-09-22)

Through the web viewer's own HTTP surface (GET /api/read/<tool>, authenticated session, from the store host), one tab's reads issued one at a time:

read busy primary mid-size primary
get_server_summary 104 ms 66 ms
get_server_properties 77 ms 28 ms
get_cpu_utilization (24 h) 185 ms 174 ms
get_memory_trend (24 h) 257 ms 237 ms
get_blocking_trend (24 h) 83 ms 51 ms
get_deadlock_trend (24 h) 16 ms 15 ms
get_analysis_findings 1,647 ms 853 ms
get_daily_summary 14,082 ms 3,611 ms
get_daily_summary_range 12,070 ms 15,767 ms

Panels render independently, so the tab completes when its slowest panel does. The run above landed inside the store's hourly refresh window (:18), which inflates it. The same two reads on the busy primary on a quiet minute, over MCP: 3,075 ms and 6,839 ms. So these two reads are slow at the best of times, and at the viewer's timeout at the worst.

Attribution

pg_stat_activity sampled every ~0.3 s from the store host during the quiet-minute pair:

statement samples longest execution seen
DailySummarySql.RangeSql (WITH wait_per_type AS …, Darling/PerformanceMonitor.Darling.Storage/DailySummarySql.cs:33) 15 ≥ 4.9 s, waiting on I/O
the rollup-coverage probe SELECT (SELECT min(bucket) FROM collect.query_stats_hourly), (SELECT min(bucket) FROM collect.query_stats_daily), … (TimescaleSupport.DetectRollupCoverageAsync) 8 ≥ 1.7 s

Two findings from the source and the store:

  1. Only the queries CTE is tier-routed (Darling viewer: built-in tabs silently lose history past 4 days - CAGG read-routing only covers Custom Views #1661, RangeSqlFor). The other eight CTEs (v_wait_stats, v_deadlocks, v_blocked_process_reports, v_dmv_blocking_snapshots, v_cpu_utilization_stats, v_collection_log, v_memory_pressure_events, config_alert_log) always read raw tables for the whole requested window. On the calendar's month span that means every retained raw chunk of eight tables, most of them compressed. For today alone the eight raw sources plus a raw COUNT(DISTINCT query_hash) total ~0.55 s on the busy primary (measured per source). So the single-day read's multi-second cost is not its own window; it is the second finding.
  2. The coverage probe runs on every call, twice per tab (both daily reads call it), taking min(bucket) through continuous-aggregate views. Its answer changes at most once a day, as rollups materialize and retention moves. At ≥ 1.7 s on a quiet minute it is most of the single-day read, and it is paid again by the range read on the same tab.

Shape options (owner's call)

Acceptance

  • Server Overview tab on a busy production primary completes in under ~3 s on a quiet minute, with no read within 2× of the viewer role's statement_timeout.
  • The rollup-coverage probe runs at most once per cache period, not once per daily read.

Activity

  1. erikdarlingdata commented on Sep 22, 2026

    @erikdarlingdata
    OwnerAuthor

    Verified at source (d21bbc9a) and measured on DARLING01, 9 servers. This is the first attribution done with get_store_query_stats from #3899 / PR #3904, deployed there as a dev build.

    Both claims hold.

    • Tier routing: only the queries CTE is routed (RangeSqlFor). The other eight CTEs read the raw v_* views for the whole window.
    • Per-call probe: DarlingHealthReader.GetDailySummaryRangeAsync probes coverage on every call, by design. Its comment reads "Probed per call, uncached: get_daily_health runs at human/model cadence and these are two small lookups", and production's ≥1.7 s falsifies the second half of that.

    Only this reader is uncached; both of its siblings already cache.

    • ComposeStoreAvailability.GetRollupsAsync (compose panels) caches availability and coverage per NpgsqlDataSource with a 5-minute TTL. Coverage expires unconditionally, so a --backfill-rollups run is still picked up without a restart.
    • ViewerDataService.RollupAvailability (the desktop Viewer's calendar) does the same, with its own 5-minute RollupReprobeInterval.

    The cache half of the fix is therefore to route the daily-summary reader through ComposeStoreAvailability.GetRollupsAsync. That is a reuse, not a new mechanism, and it keeps this tool and the calendars on the same gate.

    DARLING01, per statement (pg_stat_statements, role: mcp, six tool calls):

    statement calls mean max
    RangeSql WITH wait_per_type …, range read (month) 3 356 ms 424 ms
    RangeSql, single-day read 3 45 ms 69 ms
    coverage probe SELECT (SELECT min(bucket) FROM collect.query_stats_hourly), … 6 1.0 ms 2.0 ms

    Six probe executions for six tool calls confirms it runs every call.

    Why the probe is 1 ms here and ≥1.7 s in production. EXPLAIN (ANALYZE) shows each min() optimized to LIMIT 1 over an ordered ChunkAppend, so only the OLDEST chunk is touched. But when that chunk is compressed, the ColumnarScan seq-scans and sorts the chunk's whole batch list (Seq Scan on compress_hyper_… → Sort) before decompressing the one batch it needs. So each subquery costs the size of its relation's oldest compressed chunk. The probe also carries the rolled RAW tables' min(collection_time), and raw query_store_stats has the store's heaviest chunks. That fits the "waiting on I/O" in your sampling: it scales with per-chunk volume, not with the number of rollups.

  2. erikdarlingdata commented on Sep 23, 2026

    @erikdarlingdata
    OwnerAuthor

    Closed by the watcher: delivered in PR #3950, merged to dev.

  3. erikdarlingdata commented on Sep 23, 2026

    @erikdarlingdata
    OwnerAuthor

    Measured live on DARLING01 (dev c5be29e), MCP get_daily_summary_range median of 5: SQL2025 259 ms (before 639 ms) and SQL2022 220 ms (before 369 ms). First calls: 1,194 ms and 482 ms.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions