Rollups serve only what they materialized: route by coverage, and back fill behind a disk preflight

[erikdarlingdata]

Closes #1759 (Phases 1 and 2). Phase 0 — the held-paused observability WARN lines — already shipped in #1762.

One PR, two commits, and why

Phase 2 hard-depends on Phase 1's coverage probe: the backfill's resume point, its convergence check and the router's fallback all read the same measured floor, and splitting them would have meant a stacked PR against a protected branch on the night before a release cut. The two commits are cleanly separated and reviewable in order — Phase 1 is read-side only and could ship alone; Phase 2 is an operator verb nothing invokes by accident.

Phase 1 — coverage-aware routing (read-side, zero disk risk)

RetentionTierRouter routed by AGE alone, and nothing fell back to raw. On a store whose rollups were created WITH NO DATA over pre-existing history, every window older than the rollup's materialized floor read empty while raw still held every row — and raw still held them precisely because the #1680 arming gate had held its purge closed for the same reason.

RollupCoverage (a companion to #1665's RollupAvailability) probes each rollup's min(bucket) plus each rolled raw table's min(collection_time) in one round trip, and the router degrades a window to whichever tier is measured to reach furthest back.

The hard half is not falling back when raw would return LESS. On a healthy store with armed purges, raw keeps ~4 days against the rollups' weeks, so a window older than every floor is the normal case there — a naive "window predates the floor → use raw" rule would send every long window to a 4-day table. The rule is therefore comparative: a tier is abandoned only on a positive measurement that a lower tier is deeper, which is exactly the held-purge signature #1759 describes and is silent everywhere else. Half the routing truth table exists to hold that line, and there is a live-PG test for it specifically.

Unknown coverage is inert by construction — a failed probe, a plain-PostgreSQL store and a partially-built one all produce nulls, and nulls never move a window, so the pre-#1759 behaviour is reproduced exactly.

Other things this needed:

• All six production routing call sites gated, enumerated: ComposeSourceRouter.cs:178 (one site, routing dynamically across all three catalog tables), Mcp/DarlingHealthReader.cs:210, ViewerDataService.DailySummary.cs:56, and three in ViewerDataService.FinOps.Workload.cs (:205 on the db-grain pair, :363 and :429 on the query-grain pair). (An earlier draft of this body said seven — a miscount, corrected. Six is the verified number; the guard's rot detector now pins that floor.) The MCP get_daily_health reader is not incidental: it answers the same question as the viewer's Performance Calendar off the same shared SQL, so gating one and not the other would have had them report different query counts for the same day on exactly the affected stores, with no way to tell which was right.

  The source-parsing guard that keeps a future reader from being added un-gated is deliberately two-sided — a positive match on a real coverage lookup (.For() and an explicit rejection of TierCoverage.Unknown. Its first cut was one-sided ("does the argument list mention coverage?") and was holed: "overage" is a substring of the type name TierCoverage, so a reader passing TierCoverage.Unknown satisfied it. That form compiles and reads as deliberate, which makes it the most probable way Rollup CAGGs serve only materialized buckets: old windows read empty and raw purges stay held #1759 returns — not someone dropping an argument, which a compile-shaped review also catches, but someone reaching for the inert value because it satisfied the signature. Both forms are now pinned as mutations (table below). Tightening it also exposed that the scan was matching <see cref="…Resolve(…)"/> inside doc comments and running that phantom match forward into real code; the one-sided predicate had hidden it, because a cref contains the type name. Comments are stripped before scanning now.

• Coverage expires even where availability does not. Availability is cached permanently once complete, because a created aggregate is never dropped — true of existence, false of coverage, which moves backwards on a backfill and forwards on a retention drop. Keeping the AllPresent shortcut would have pinned a pre-backfill floor for the life of the process, so an operator who had just backfilled would keep getting raw fallbacks until a restart. The existing 5-minute reprobe interval now applies unconditionally, in both the composer's per-datasource cache and the viewer's.

• The partial-window notice now prefers the MEASURED floor to the retention span. On these stores the purges are held, so raw holds months rather than its nominal ~4 days; left assuming, the notice would have stamped "older points are not included" on precisely the coverage-fallback panels that are in fact complete — a false alarm introduced by the fix. It falls back to the span when nothing is measured, which reproduces the old text.

The false premise is retired. TimescaleSupport's claim that "real-time aggregation is opted into… correct to query for any window immediately" was wrong twice over — 2.13+ defaults materialized_only to TRUE, and even ON the watermark is a hard partition, so un-materialized history is served by neither branch. The materialized_only test pin stays, now load-bearing and for the true reason: RollupCoverageProbeSql and RetentionArmSafetySql both read min(bucket) to mean "the oldest bucket MATERIALIZED", and unioning the raw branch in would make an empty materialization report raw's own oldest row as the rollup's floor — the router would believe coverage it does not have, and the arming gate would arm a purge over history nothing else holds. The cold-start caveat prescribing a manual backfill no store ever received is retired too, replaced by a pointer to the verb.

Phase 2 — --backfill-rollups, an operator verb with a disk preflight

The rejected option ("just open the gate") is not resurrected. The arming gate is all-or-nothing, so a store with a year of raw must materialize the whole history before the first purge arms and reclaims anything: peak disk comes before any relief. At service start, on the worst-affected stores, that is a plausible disk-exhaustion event.

• Preflight that refuses with numbers. The estimate is calibrated from what the rollup has already materialized (bytes per bucket × buckets to add) — every affected store has that sample, since its refresh policy has been materializing a trailing 3-day window all along. With no sample it bounds from raw's own size and says so, deliberately erring high: refusing a backfill that would have fit is recoverable, filling a production volume is not. Requires the estimate + 25% headroom + a 10 GB reserve. Free space is measured on the volume the store reports (current_setting('data_directory')), so a store on another host refuses rather than measuring this machine's disk. The refusal names the shortfall and both real options — including that waiting is safe, since nothing is being lost while the purges are held.
• Sliced one source chunk at a time, NEWEST first (see the BLOCKING-2 fix above — the direction is a data-safety property). The issue's own API research is right that a manual refresh already batches internally, so this is supervision rather than re-implementation: progress on a multi-hour run, a resume point, and a lock window short enough not to sit across a compression job (the Compression waits for the next fixed tick, so a just-closed chunk can sit uncompressed most of a day (81 GB observed); deadlock correlation to watch #1778 watch).
• Convergence read from DATA. A batch-cap stop is logged server-side and completely silent to the client, so the calls returning is no evidence. Only a measured shortfall escalates to the forced form — the one thing that repairs a hole left by an interrupted pass, whose invalidation records a plain refresh now skips straight over.
• Idempotent and resumable — every pass re-plans from the measured floor; a completed one converges to a no-op.
• Arms nothing. Coverage is the whole job. The arming gate already self-heals at startup, and duplicating that decision in a second place is how the one thing protecting this data stops being the one thing.
• --dry-run prints the plan, the estimate and a duration budget at the ~16 MB/s measured on this host class, and stops.

Concurrency / locking. refresh_continuous_aggregate takes no lock that blocks writers on the source hypertable, so collection keeps running; it cannot run inside a transaction, so each slice is its own statement and partial progress survives an abort. What it does contend with is the compression policy on the same chunks, which is why slices are one chunk wide.

The operator contract, verbatim

Every line the verb tells an operator is reproduced here rather than paraphrased, because these are the contract and a reviewer should be able to read them without opening the source. Both are pinned by BackfillRollupsVerb_DryRunsThenBackfills_AndReportsCoverageItActuallyReached, so editing them breaks a test rather than silently voiding the contract.

On completion — the next step, the confirmation to look for, and the one interaction the operator has to know about: the hourly rollups carry their own 21-day retention policy, already armed on these stores, which will trim the coverage a run just built when it next fires (measured cadence: schedule_interval = 1 day, see #1790):

NEXT: restart the PerformanceMonitor Darling service. The arming gate checks coverage at
startup and releases the held retention policies by itself — there is no arming step here and
nothing to run by hand. The startup log line reading
  'N/N retention policies in place, N armed, 0 held paused pending backfill'
is the confirmation; the first purge then reclaims the raw tables in one pass.

Do not delay the restart. The hourly rollups carry their OWN 21-day retention policy, already
armed on these stores, which will trim the coverage this run just built when it next fires
(roughly daily). Restarting now is what lets the raw policies arm off that coverage first. If
the trim wins the race nothing is lost — raw is still held — and re-running this verb rebuilds it.

And the sibling line on the nothing-to-do path (DarlingCliCommands.cs:2187), which carries the same self-heal promise for the store that is already covered — including a re-run after a successful pass, which is the most likely way an operator sees this verb a second time:

Every rollup already covers its raw table. Any retention policy still held will arm itself on the next service start.

Two BLOCKING review defects, both proven by execution, both fixed

1. The preflight under-estimated by rows-per-bucket. The probe counted ROWS (count(*)) and the arithmetic divided that into bytes to get a per-BUCKET figure, then multiplied by a BUCKET count. A rollup holds one row per (server, database, query_hash, sql_handle, bucket), so the units differ by the distinct queries seen per hour — 205x under on a 200-row/hour store, worse on a real fleet box. That is the one direction this preflight exists to prevent, and the code's own comment said so. Fixed with count(DISTINCT bucket).

The suite could not have caught it, which mattered as much as the bug: both live fixtures seeded one row per bucket, making rows and buckets the same number and the error factor exactly 1. That is not a small fixture, it is a fixture of a shape the product never meets. They now seed 12 distinct queries per bucket, and a live test asserts the probe returns a bucket count by ratio rather than by string — so the semantics are pinned, not the spelling.

2. An interrupted backfill left a hole, reported DONE, and the arming gate then armed a purge over it. min(bucket) was the resume point, the completion verdict, and what the #1680 gate reads before letting the 4-day raw purge drop chunks. Slices ran oldest-first, so the first slice drove that single number to its final value and every later slice was invisible to all three. Proven by execution: killed at slice 32 of 43, 47 of 264 buckets missing, re-run printed "nothing to do — DONE" and exited 0, and the gate then armed over a window whose only surviving copy was the raw rows about to be dropped. The slice-failure path reached the same state, and the "Safe to interrupt" line was false.

Fixed by reversing the slice order, not by adding per-slice bookkeeping. Descending makes floor <= raw_oldest imply completeness by construction: the floor can only reach the bottom once the last, oldest slice has run, in any interleaving. An interrupted run leaves it visibly short — a truthful SHORT instead of a false DONE — and the gate stays closed because it reads the same honest number. Mid-slice kills need no extra state either: a refresh commits per batch newest-first, so the floor lands inside the killed slice and re-planning from it re-covers the remainder. I chose this over per-slice verification because it removes the failure mode rather than detecting it — there is no interleaving left in which a hole can be reported as coverage, so nothing depends on a probe being remembered.

Plus the non-blocking sanity clamp. One stray epoch-era row produced a 739,825-slice plan that burned CPU indefinitely. Plans past a 10-year ceiling now REFUSE and name the offending timestamp — that is corruption someone has to go and look at, not history to back fill. A refusal is deliberately not a skip: [REFUSED] on stderr, DONE blocked, non-zero exit, because burying it among the [OK] lines is exactly how it would be missed.

Three findings from the gated live leg

None were visible by reading; all three are product fixes, not test workarounds.

1. 55P03 concurrent refresh. The verb runs while the service is UP and the aggregate's own refresh policy lands on top of a slice. Reproduced immediately: EnsureContinuousAggregatesAsync attaches the policy, the policy runs at once, the next slice failed. Retried, bounded, transient-only — every other SQLSTATE still fails fast.
2. 22023 "refresh window too small". A day-wide slice's ragged tail is narrower than one bucket for a daily rollup, and the range end is a coverage floor or "now", so it lands mid-bucket most of the time. It aborted the whole daily tier. The planned range now closes on a bucket boundary as well as opening on one.
3. The per-slice convergence check was measuring the wrong thing. "Did the floor reach this slice's start?" fired on the first slice of every run, because a slice's range can legitimately hold no source rows — raw's oldest row lands partway into the first slice, and a collection gap does the same mid-run. A global min(bucket) cannot answer a question about one slice. Convergence is judged once, at the end, against raw's oldest row.

Testing

Full solution rebuild: 0 warnings, 0 errors. Darling suite: 3723 passed, 0 failed (7 skipped — gated on a live SQL Server and the PG-runtime fixture, neither present here).

Gated live legs ran against a real PostgreSQL 18.4 + TimescaleDB 2.28.1 store stood up from the repo's pg-runtime.zip, on a store built into the exact broken shape: pre-existing history, rollup created WITH NO DATA, floor well after raw's.

• The defect measured, not assumed: a window below the floor returns 0 rows from the rollup while raw holds rows for the same window.
• Phase 1 routes that window to raw; after the backfill it routes back to the rollup.
• The healthy shape (raw shallower than the rollup) does not fall back — with the row counts showing why.
• The verb driven end to end through a bring-your-own darling.json, not a re-implementation of its loop: dry run changes nothing, the real run's DONE claim is re-verified against the store per rollup, and a second run reports nothing to do.
• The interrupted-backfill acceptance — the reviewer's data-loss proof inverted. Kill at two thirds of the slices, then assert the partial run does not look complete, the re-plan still has work, the real arming gate refuses (read from the job catalog, not by re-evaluating its predicate), then resume and assert zero missing buckets measured against raw itself rather than against our own arithmetic — and only then does the gate arm.
• The verb's refusal branch end to end, which previously had only its pure arithmetic covered: a single poisoned timestamp, [REFUSED] on stderr naming it, no DONE, exit 1.
• The arming gate then arms query_stats' held retention policy by itself, through EnsureRetentionPoliciesAsync — the real seam, not its predicate.

Mutation table, each verified RED:

Mutation	Caught by
Floor check dropped	7 routing tests; the held-store divergence pin goes Raw → Daily
"Raw is deeper" guard removed (naive floor check)	8 tests, incl. the healthy-store no-regression legs
(A) A routing reader passing TierCoverage.Unknown — compiles, reads as deliberate, and is the likely way back	Guard: ViewerDataService.DailySummary.cs:57 (passes TierCoverage.Unknown — routes on NO evidence)
(B) A routing reader with the coverage argument removed entirely	Guard: ViewerDataService.DailySummary.cs:57 (passes no coverage lookup)
(C) Row-count estimator restored (count(*)), against the R>1 fixture	Unit pin and the live ratio guard — the old 1-row-per-bucket fixture caught neither
(D) Ascending slice order restored	The live acceptance reproduces the defect verbatim: "an interrupted backfill reports coverage … at or below raw's oldest — it looks COMPLETE", plus two unit pins
Preflight bypassed	3 preflight theory rows
Convergence made call-based over a silently under-covering plan	Verb reports INCOMPLETE + names the exact shortfall; live test red
Hierarchical order inverted	The order pin and the live verb (dailies genuinely short)

Known limits of the coverage model, recorded not assumed away

The floor is a depth measure, not a completeness guarantee, and RetentionTierRouter's docs now say so explicitly. Two shapes slip through a one-dimensional floor: a mid-window materialization hole is invisible to it (a service down longer than the refresh policy's 3-day start_offset resumes at now-3d and never backfills the skipped interval, while min(bucket) goes on reporting the original deep floor), and a window straddling the floor is served partially with no signal to the caller (returning the tier is still the right choice — raw would return less on a healthy store — but the part below the floor is missing).

Both are accepted here because both are strictly better than the age-only routing they replace, which served the entire window as EMPTY in exactly these cases. The docs paragraph exists to block one specific inference: "the coverage gate passed, therefore the result is complete." That belief-shape is how #1759 survived as long as it did — the previous comment asserted the view was "correct to query for any window immediately", it read as reasonable, it was pinned by a test, and nobody re-derived it for months.

Tracked as #1791 with both candidate closures scoped (two-dimensional coverage with gap detection; a partial-coverage notice through the existing CombineNotices machinery). Neither is cut-night material — the first has a real cost question on a 5-minute probe cadence, the second is a compose-contract change.

Follow-ups filed

• Backfilled rollup coverage races the hourly tier's own 21-day purge, so the raw policies can miss their arming window #1790 — the arming-window race: the hourly tier's own armed 21-day purge trims freshly-built coverage, so the operator must restart promptly. Shipped surfaced (text above); both immediate fixes rejected with recorded hazards.
• Rollup coverage is a one-dimensional depth measure: mid-window holes are invisible and straddling windows are served partially without a signal #1791 — the coverage-model limits described above.

Interaction with the other lanes in flight

Verified rather than assumed: #1772 (delta-seed time bound) and #1768/#1767 (payload dimensions) touch neither the routing ladder nor the rollups' materialization — the CAGGs never carried the payload columns, so the dimension work is invisible here, and the delta seeder writes raw rows the coverage probe only ever reads a min() from. Branched off origin/dev at 50012266, which already contains both. Full-suite green above is over the merged tree.

🤖 Generated with Claude Code