Skip to content

get_daily_summary's cold call pays the rollup coverage probe: about 10 s, 537 MB read and 322 MB of temp, two-thirds of it one view's oldest chunk #4957

Closed

Description

@erikdarlingdata

Class: bug (performance). On a 43-server SQL Server store running v3.9.0, a get_daily_summary call that finds the rollup-coverage cache cold costs about 13 s. That happens:

  • on the process's first call;
  • after a failed probe;
  • when a view's floor-cache entry is over an hour old (RollupFloorMaxReuse).

Of the 13 s, about 9.7 s is the store-wide coverage probe (TimescaleSupport.RollupCoverageProbeSql, cold form). The summary statement itself is about 3.7 s for today on the busiest server.

Census (get_read_latency hours=168, about 4.5 h since the install)

surface / route calls mean p95 (bucket bound) max timeouts
mcp / get_daily_summary 1 12.6 s ≤ 15 s 12.6 s 0
web / get_daily_summary_range 3 3.2 s ≤ 10 s 8.9 s 0

The store's pg_stat_statements (mcp role) for the coverage probe: 3 calls, mean 3.9 s, max 9.8 s, 192 k blocks read and 104 k temp blocks written. That fits a few cold probes among warm ones.

Plans (EXPLAIN (ANALYZE, BUFFERS), one run each)

statement time shared read / hit temp written
coverage probe, cold form (every view's min(bucket) plus the raw floors) 9,675 ms 68,753 / 36,819 41,212 blocks (322 MB)
… of which min(bucket) on query_store_stats_interval_daily 6,261 ms 30,950 / 24,223 19,589 blocks (153 MB)
… query_store_stats_corrected_hourly 663 ms 11,787 / 257 10,781 blocks
… query_store_stats_hourly 653 ms 11,852 / 255 10,842 blocks
… query_store_stats_interval_hourly 510 ms — —
DailySummarySql.RangeSql, raw tier, today, busiest server 3,650 ms 28,705 / 6,155 0
its wait CTEs alone, 30 days 2,821 ms 7,787 / 13,703 0
its wait CTEs alone, 366 days (warm, right after the 30-day run) 386 ms 11 / 20,655 0

The probe's plan output was cut by the transport's size limit after the first views. The total is the plan's top node, and "—" means not captured.

What the probe's plans show:

  • Each min(bucket) is a DeferredChunkAppend (Order: bucket, Chunks Visited: 1) over the view's oldest chunk.
  • For a compressed chunk, that oldest chunk is decompressed and sorted to find one value.
  • query_store_stats_interval_daily's oldest chunk spans ten days (09-14 to 09-24); the rest are one day each. It alone is two-thirds of the probe.

RangeSql for today: the coll CTE (collection_log, 1.67 s), queries (raw query_stats, 202 k rows, 1.43 s) and waits (0.50 s) are the main costs.

What earlier work changed

Suspected cause

  1. The cold probe's cost is one min() per view over a compressed oldest chunk (measured). The code comments already name this cost: "seq-scans and sorts that chunk's batch list".
  2. How often it runs (inferred, from the code, not measured live): the floor cache is per NpgsqlDataSource. DetectRollupCoverageAsync runs on a timer and on demand, so each data source (the service's worker, the MCP host, the web host) re-pays the full sort at least hourly, plus on every process start.
  3. A cold MCP call pays it inline, ahead of the summary itself, so the first answer takes about 13 s.

Options (inferred, for whoever designs the fix)

  • Read each view's floor from chunk metadata, for example the compressed chunk's _ts_meta_min columns or range_start plus one bounded probe, instead of a decompress-and-sort.
  • Share one floor cache across the process's data sources.
  • Warm the probe in the background after start, so no caller pays it inline.

Code:

  • DarlingHealthReader.GetDailySummaryRangeAsync → GetMeasuredRollupsAsync → ComposeStoreAvailability.ProbeAsync → TimescaleSupport.DetectRollupCoverageAsync / RollupCoverageProbeSql;
  • DailySummarySql.RangeSql.

Activity

  1. added
    bugSomething isn't working
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Oct 2, 2026
  2. erikdarlingdata commented on Oct 2, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed by #4968, merged into dev as ea91586.

  3. removed
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions