Skip to content

Store: re-measure a rollup's coverage floor only when its oldest chunk changes (#4539) - #4553

Merged
erikdarlingdata merged 4 commits into
devfrom
fix/4539-rollup-floor-cache
Sep 28, 2026
Merged

erikdarlingdata merged 4 commits into
devfrom
fix/4539-rollup-floor-cache

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 28, 2026 •

Copy link
Copy Markdown
Owner

Fixes #4539

Why

The rollup coverage probe re-runs min(bucket) against every present rollup's materialization on every cycle. On a compressed materialization chunk that sort touches every one of that chunk's compressed batches before it can decompress the first one — measured on a production store at roughly 170 MB of temp every five minutes, because the oldest Query Store rollup chunks hold tens of thousands of batches. The floor only actually moves when the rollup's oldest chunk changes (retention drops it, or an older backfill creates an earlier one), so re-sorting it on every cycle regardless is wasted work.

What changes

  • A per-data-source cache (ConditionalWeakTable<NpgsqlDataSource, ...>) keyed on each rollup's oldest materialization chunk identity. A cheap catalog read (RollupOldestChunkSql) decides which rollups' chunk identity changed since the last cycle; only those get a fresh min(bucket), and the rest reuse their last-measured floor.
  • RollupCoverageProbeSql gained an overload that takes the set of rollups to actually measure this cycle; a rollup not in that set contributes NULL::timestamp and is never named in the query, the same "don't mention an absent relation" trick already used for a rollup that doesn't exist at all. The existing one-argument overload (used by RetentionArmSafetySql and everything else) is unchanged and stays byte-identical.
  • When a cached floor is used, and when it is not:
    • The floor merge is a pure function, MergeRollupFloors. A rollup measured this cycle always uses the measured value: if it comes back NULL (empty, or re-created with no data), it has no floor and its cache entry is dropped. Only rollups not measured this cycle reuse a cached floor.
    • The cache key is the oldest chunk's name AND the materialization hypertable's identity, so dropping and re-creating a rollup always forces a re-measure.
    • A cached floor is reused for at most one hour (RollupFloorMaxReuse). Nothing in this code deletes rows from a rollup's oldest chunk without dropping it, but a manual refresh over raw data that is already gone, or a manual DELETE, could move the true floor later without changing the chunk. The one-hour limit bounds how long such a change can go unseen, and still cuts the probe from every five minutes to at most once an hour.
    • A rollup that no longer exists has its cache entry dropped.
  • A failed catalog read (a plain PostgreSQL store, or any other failure) falls back to measuring every present rollup, the same behavior as before this cache existed.

Test plan

Live end-to-end pin (RollupFloorCacheLiveTests) against a real TimescaleDB with a compressed materialization:

  • (a) the first call's measured floor matches a fresh min(bucket).
  • (b) a second call on the same data source returns the same floor. Counted through Npgsql's own "Executing command" debug log line for the min(bucket) probe against the materialization view: across both calls combined, exactly one such statement executes on this branch — RED on dev, run: Assert.Equal() Failure: Values differ. Expected: 1. Actual: 2. pg_stat_user_tables' seq_scan/idx_scan counters were tried first and measured unreliable on a compressed chunk (they don't change either way), and this rig doesn't preload pg_stat_statements.
  • (c) after dropping the oldest materialization chunk, the next call re-measures and the floor moves later, tracking the true min(bucket).

Pure pins (RollupFloorCacheTests): a measured NULL with a cached entry evicts it; a rollup not measured this cycle reuses its entry; a rollup that no longer exists drops its entry; the same chunk name on a different materialization hypertable is measured; an entry older than the one-hour limit is measured, and one well under it is reused (an injectable clock, no sleep). A mutation that disables the measured-NULL eviction fails two of them.

IntervalHonestHourlyRollupLiveTests (drop and re-create a rollup with no data on the same data source; the coverage floor must read NULL) failed against the first version of this change and passes now.

A mutation of the pure planner (RollupFloorsToMeasure, dropping the chunk-identity comparison so a rollup is measured only once ever, regardless of a later chunk change) turned pin (c) RED (Expected: 2026-09-26 ... Actual: 2026-09-25 ...); reverting it returns to GREEN.

Ran on this Mac against a scratch TimescaleDB rig: RollupFloorCacheTests, RollupFloorCacheLiveTests, and every other class touching DetectRollupCoverageAsync/RollupCoverageProbeSql plus DocCommentHygieneTests. Total: 255, Failed: 0, Skipped: 3 (the 3 skips are pre-existing environment-gated live tests unrelated to this change). LivePostgresCollectionHygieneTests needs a Windows runtime and runs in CI.

Darling.Tests and Lite.Tests build clean (0 errors, 0 warnings) with -p:EnableWindowsTargeting=true; both target net10.0-windows and can't execute on macOS, so CI decides them for real.

Field effect (expected): on the production store that showed the spill, the coverage probe's two spilling min(bucket) reads stop running on every cycle. They run again only when a rollup's oldest materialization chunk changes (a retention drop), after an hour, or after a restart.

CHANGELOG

SECTION: Fixed
ENTRY: - The rollup coverage check no longer writes about 170 MB of temp files every five minutes on a large store ([#4553]) - Finding a rollup's oldest bucket sorts every compressed batch of its oldest chunk. The check now re-measures a rollup only when its oldest chunk changes, and reuses the last measurement otherwise.
REF: [#4553]: #4553

…k changes (#4539)

TimescaleSupport.DetectRollupCoverageAsync ran min(bucket) against every
present rollup on every call. On a store whose oldest materialization
chunk holds tens of thousands of compressed batches, that min sorts the
whole chunk's batch metadata before it can decompress the first row,
which measured at roughly 170 MB of temp every five minutes on a large
production store.

A rollup's floor only moves when its oldest materialization chunk
changes (retention drops it, or an older backfill lands an earlier one).
This adds a cheap catalog read of each present rollup's oldest chunk
(RollupOldestChunkSql), a pure planner (RollupFloorsToMeasure) that
decides which rollups need min(bucket) re-run this cycle, and a
two-argument RollupCoverageProbeSql overload that only names the
measured rollups (a non-measured view still contributes a NULL
placeholder, so column ordinals never shift). DetectRollupCoverageAsync
caches the last-measured floor per NpgsqlDataSource and reuses it for
every rollup whose oldest chunk is unchanged. The one-argument overload
is unchanged and stays byte-identical to today's output, since
RetentionArmSafetySql and existing callers depend on it.

A reused floor can only be later than the true one, never earlier, so
routing never claims coverage a rollup lacks.

Adds RollupFloorCacheTests (the pure planner cases, the SQL overloads'
selectivity and column count, and a literal-output pin on the
unchanged one-argument form).
… cycle (#4553)

DetectRollupCoverageAsync's cache could serve a floor that no longer
matched reality in two ways:

- a view that was measured this cycle and came back NULL (a fresh,
  empty, or freshly recreated materialization) fell through to the
  same reuse branch as a view that was skipped, so it kept its old
  cached floor instead of losing it;
- the cache key was only the oldest chunk's generated name, which a
  drop and re-create of the continuous aggregate can coincidentally
  reuse for its new materialization hypertable's first chunk.

The floor-merge logic now runs through a pure function
(MergeRollupFloors) that treats 'measured this cycle' and 'reused
from cache' as mutually exclusive: a measured-NULL view has no floor
and its cache entry is dropped, and a view whose rollup disappeared
entirely is evicted rather than carried forward. The cache key adds
the materialization hypertable's own schema-qualified identity
alongside the oldest chunk name. A one-hour TTL (RollupFloorMaxReuse)
forces a re-measure even when nothing in the catalog changed, because
a retention DELETE against the oldest materialization chunk can move
the true floor later without changing that chunk's identity at all.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant