Rollups serve only what they materialized: route by coverage, and back fill behind a disk preflight

CHANGELOG.md

@@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- **The retention rollups only ever served what they had materialized, so old windows read empty while raw still held every row** ([#1788], closes [#1759]) - not a data-loss bug: the [#1680] arming gate held the raw purges closed on exactly the affected stores, which is why nothing was lost. The damage was that tier-routed reads returned **empty** for old windows, and that raw then grew without bound because the purges never armed. The premise the rollup tier was built on was wrong twice over. `TimescaleSupport` claimed real-time aggregation was opted into and the view was therefore "correct to query for any window immediately, just un-accelerated" - but TimescaleDB has defaulted `materialized_only` to TRUE since 2.13 (the runtime is pinned at 2.28.1), so naming no option means real-time aggregation is **OFF**; and turning it on would not have helped, because the watermark is a **hard partition** rather than a fallback - `build_union_query` emits materialized-below `UNION ALL` raw-at-or-above with no contiguity guarantee, so history below the watermark that was never materialized is served by **neither** branch. Every rollup's refresh policy starts 3 days back, so on a store that existed before its rollups the materialized span began at roughly creation-minus-3-days and never reached further back on its own. The false premise was not merely a stale comment - it was **pinned by a test** asserting the absence of `materialized_only` for the wrong reason.
+
+  **Reads are fixed with no materialization at all**, because the held purges mean raw still holds every row. A new `RollupCoverage` probe (the companion to [#1665]'s `RollupAvailability`) reads each rollup's `min(bucket)` and each rolled raw table's `min(collection_time)` in one round trip, and `RetentionTierRouter` degrades a window to whichever tier is MEASURED to reach furthest back. **The hard half is not falling back when raw would return LESS**: on a healthy store with armed purges raw keeps ~4 days against the rollups' weeks, so a window older than every floor is the *normal* case there, and a naive "window predates the floor -> use raw" rule would have sent every long window to a 4-day table - a regression introduced by the fix, on every store that was working correctly. The rule is therefore comparative: a tier is abandoned only on a positive measurement that a lower tier is deeper, which is precisely the held-purge signature and is silent everywhere else. Unknown coverage is **inert by construction** - a failed probe, a plain-PostgreSQL store and a partially-built one all produce nulls, and nulls never move a window - so the previous behaviour is reproduced exactly. All seven production routing readers are gated, including the MCP `get_daily_health` reader, which is not incidental: it answers the same question as the viewer's Performance Calendar off the same shared SQL, so gating one and not the other would have had them report **different query counts for the same day** on exactly the affected stores, with no way to tell which was right. A source-parsing guard fails the build if a future reader is added un-gated, naming the file and line. Coverage also **expires where availability does not**: availability is cached permanently once complete because a created aggregate is never dropped, which is true of existence and false of coverage - it moves backwards on a backfill and forwards on a retention drop - so keeping the `AllPresent` shortcut would have pinned a pre-backfill floor for the life of the process. And the partial-window notice now prefers the **measured** floor to the retention span, because on these stores the purges are held and raw holds months rather than its nominal 4 days; left assuming, it would have stamped "older points are not included" on precisely the fallback panels that are complete.
+
+  **Materializing the history is a new operator verb, `--backfill-rollups`, deliberately not a startup step.** The arming gate is all-or-nothing, so a store with a year of raw must materialize the WHOLE history before the first purge arms and reclaims anything: **peak disk comes BEFORE any relief**, and doing that automatically at service start, on the worst-affected stores (one already down to roughly 150 GB free), is a plausible disk-exhaustion event. So it preflights and refuses with numbers. The size estimate is CALIBRATED from what the rollup has already materialized (bytes per bucket x buckets to add) - every affected store has that sample, since its refresh policy has been materializing a trailing 3-day window all along - and with no sample it bounds from raw's own size and says so, erring deliberately high because refusing a backfill that would have fit is recoverable and filling a production volume is not. It requires the estimate plus 25% headroom plus a 10 GB reserve, and measures free space on the volume the STORE reports (`current_setting('data_directory')`), so a store on another host refuses rather than measuring the wrong disk; the refusal names the shortfall and both real options, including that **waiting is safe** - nothing is being lost while the purges are held. The run is sliced one source chunk at a time, oldest first (supervision rather than re-implementation: a manual refresh already batches internally, so the slices buy progress on a multi-hour run, a resume point, and a lock window short enough not to sit across a compression job), **judges convergence from DATA** because a batch-cap stop is logged server-side and completely silent to the client, escalates to the forced refresh only on a measured shortfall - the one thing that repairs a hole left by an interrupted pass, whose invalidation records a plain refresh skips straight over - and is idempotent and resumable, re-planning from the measured floor every time. It **arms nothing**: coverage is the whole job, and the arming gate already releases the held policies by itself on the next start. The verb's output says so, and also warns that the hourly rollups' own 21-day policy will trim the coverage a run just built when it next fires, so the restart should not wait.
+
+  **Three findings came out of the gated live leg** against a real PostgreSQL 18.4 + TimescaleDB 2.28.1 store, none of them visible by reading, all three fixed in the product rather than worked around in the test. **`55P03` concurrent refresh**: the verb runs while the service is UP and the aggregate's own refresh policy lands on top of a slice - reproduced immediately, since the ensure sweep attaches the policy and the policy runs at once. Retried, bounded, transient-only, so every other SQLSTATE still fails fast. **`22023` "refresh window too small"**: a day-wide slice's ragged tail is narrower than one bucket for a DAILY rollup, and the range end is a coverage floor or "now" so it lands mid-bucket most of the time - it aborted the entire daily tier; the planned range now closes on a bucket boundary as well as opening on one. **The per-slice convergence check was measuring the wrong thing**: "did the floor reach this slice's start?" fired on the first slice of *every* run, because a slice's range can legitimately hold no source rows (raw's oldest row lands partway into the first slice, and a collection gap does the same mid-run) - a global `min(bucket)` cannot answer a question about one slice, so convergence is judged once at the end against raw's oldest row. The live legs prove the defect by measurement (a window below the floor returns zero rows from the rollup while raw holds rows for it), prove Phase 1 routes it to raw and back to the rollup after the backfill, prove the healthy shape does NOT fall back, drive the **verb itself** end to end through a bring-your-own `darling.json` rather than re-implementing its loop - dry run changes nothing, the `DONE` claim is re-verified against the store per rollup, a second run reports nothing to do - and finish by watching the arming gate arm the held `query_stats` policy by itself through `EnsureRetentionPoliciesAsync`, the real seam rather than its predicate. Every guard verified RED by mutation: floor check dropped, the raw-is-deeper guard removed, a routing reader left un-gated, the preflight bypassed, convergence made call-based over a silently under-covering plan, and the hierarchical order inverted (caught by the pin AND by the live verb reporting the dailies genuinely short). The `materialized_only` pin **stays**, now load-bearing and for the true reason: the coverage probe and the arming gate both read `min(bucket)` to mean "the oldest bucket MATERIALIZED", and unioning the raw branch in would make an empty materialization report raw's own oldest row as the rollup's floor - the router would believe coverage it does not have, and the arming gate would arm a purge over history nothing else holds
+
 ### Added
 
 - **Two guards that turn structural properties into checked ones** ([#1796]) - review closers on [#1789] / [#1793], tests only. **The raw-tier retention policies are now pinned as DERIVED from `TimescaleSupport.RawTierCoverage`**, not merely equal to it. That map is what makes the tiered retention POLICY and the catalog SWEEP unable to judge the same drop differently, and until now the derivation was structural only: re-hardcoding the three rows beside the map produces an identical policy set and passes every behavioural test, right up until the map gains a fourth tier or one of the three changes its covering aggregate — at which point the two paths silently disagree about the exact thing the map exists to keep aligned. No behavioural test can see that, because a duplicated list behaves identically; only reading the construction can. Watched red by re-hardcoding the three rows exactly as a future edit would.
@@ -1847,6 +1857,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 [#1785]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1785
 [#1777]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/1777
 [#1780]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1780
+[#1665]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/1665
+[#1788]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1788
 [#1781]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1781
 [#1782]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/1782
 [#1783]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1783

Darling/Darling.Tests/ComposeSourceRouterTests.cs

@@ -45,7 +45,7 @@ private static PanelPlan Plan(
     public void RecentWindow_RoutesRaw()
     {
         /* oldest point 2 days old — inside the 3-day raw route max → raw. */
-        var route = ComposeSourceRouter.Resolve(Plan("query_worker_us"), Now, Now.AddDays(-2), RollupAvailability.All);
+        var route = ComposeSourceRouter.Resolve(Plan("query_worker_us"), Now, Now.AddDays(-2), RollupAvailability.All, RollupCoverage.Unknown);
         Assert.Equal(ComposeSourceTier.Raw, route.Tier);
         Assert.False(route.IsCagg);
         Assert.Null(route.CaggRelation);
@@ -54,15 +54,15 @@ public void RecentWindow_RoutesRaw()
     [Fact]
     public void MidWindow_RoutesHourlyCagg()
     {
-        var route = ComposeSourceRouter.Resolve(Plan("query_worker_us"), Now, Now.AddDays(-10), RollupAvailability.All);
+        var route = ComposeSourceRouter.Resolve(Plan("query_worker_us"), Now, Now.AddDays(-10), RollupAvailability.All, RollupCoverage.Unknown);
         Assert.Equal(ComposeSourceTier.Hourly, route.Tier);
         Assert.Equal("query_stats_hourly", route.CaggRelation);
     }
 
     [Fact]
     public void OldWindow_RoutesDailyCagg()
     {
-        var route = ComposeSourceRouter.Resolve(Plan("query_worker_us"), Now, Now.AddDays(-40), RollupAvailability.All);
+        var route = ComposeSourceRouter.Resolve(Plan("query_worker_us"), Now, Now.AddDays(-40), RollupAvailability.All, RollupCoverage.Unknown);
         Assert.Equal(ComposeSourceTier.Daily, route.Tier);
         Assert.Equal("query_stats_daily", route.CaggRelation);
     }
@@ -72,7 +72,7 @@ public void HistoricalWindow_RoutesByAge_NotWindowSpan()
     {
         /* A 5-day-SPAN window that is 30→25 days OLD must route by age (30d → daily), NOT by span (5d → hourly):
            the hourly chunks for that range were already dropped, so span-based routing would return empty. */
-        var route = ComposeSourceRouter.Resolve(Plan("query_worker_us"), Now, Now.AddDays(-30), RollupAvailability.All);
+        var route = ComposeSourceRouter.Resolve(Plan("query_worker_us"), Now, Now.AddDays(-30), RollupAvailability.All, RollupCoverage.Unknown);
         Assert.Equal(ComposeSourceTier.Daily, route.Tier);
     }
 
@@ -81,15 +81,15 @@ public void RankedMode_RoutesByAge_WithNoDisplayGrain()
     {
         /* Ranked panels resolve no display grain at all — the v1-killer case. Age-based routing still works:
            a 40-day "top N" reaches the daily CAGG instead of truncating at raw's 4 days. */
-        var route = ComposeSourceRouter.Resolve(Plan("query_worker_us", PanelMode.Ranked), Now, Now.AddDays(-40), RollupAvailability.All);
+        var route = ComposeSourceRouter.Resolve(Plan("query_worker_us", PanelMode.Ranked), Now, Now.AddDays(-40), RollupAvailability.All, RollupCoverage.Unknown);
         Assert.Equal(ComposeSourceTier.Daily, route.Tier);
     }
 
     [Fact]
     public void NoCaggTable_AlwaysRaw()
     {
         /* wait_stats has no CAGG → raw even for a 40-day window (routing is a no-op for the ~30 non-CAGG tables). */
-        var route = ComposeSourceRouter.Resolve(Plan("wait_time_ms"), Now, Now.AddDays(-40), RollupAvailability.All);
+        var route = ComposeSourceRouter.Resolve(Plan("wait_time_ms"), Now, Now.AddDays(-40), RollupAvailability.All, RollupCoverage.Unknown);
         Assert.Equal(ComposeSourceTier.Raw, route.Tier);
     }
 
@@ -101,7 +101,7 @@ public void ObjectNameDimension_NowRoutes_ViaModuleMap()
         var objectName = MeasureCatalog.Dimension("query_stats", "object_name")!;
         var plan = Plan("query_worker_us", groupBy: new[] { objectName });
         Assert.True(plan.UsesModuleJoin);
-        var route = ComposeSourceRouter.Resolve(plan, Now, Now.AddDays(-40), RollupAvailability.All);
+        var route = ComposeSourceRouter.Resolve(plan, Now, Now.AddDays(-40), RollupAvailability.All, RollupCoverage.Unknown);
         Assert.Equal(ComposeSourceTier.Daily, route.Tier);
         Assert.Equal("query_stats_daily", route.CaggRelation);
     }
@@ -110,15 +110,15 @@ public void ObjectNameDimension_NowRoutes_ViaModuleMap()
     public void CoveredDimension_QueryHash_Routes()
     {
         var queryHash = MeasureCatalog.Dimension("query_stats", "query_hash")!;
-        var route = ComposeSourceRouter.Resolve(Plan("query_worker_us", groupBy: new[] { queryHash }), Now, Now.AddDays(-10), RollupAvailability.All);
+        var route = ComposeSourceRouter.Resolve(Plan("query_worker_us", groupBy: new[] { queryHash }), Now, Now.AddDays(-10), RollupAvailability.All, RollupCoverage.Unknown);
         Assert.Equal(ComposeSourceTier.Hourly, route.Tier);
     }
 
     [Fact]
     public void ServerDimension_IsUniversallyCovered()
     {
         var server = MeasureCatalog.ServerDimension("query_stats");
-        var route = ComposeSourceRouter.Resolve(Plan("query_worker_us", groupBy: new[] { server }), Now, Now.AddDays(-10), RollupAvailability.All);
+        var route = ComposeSourceRouter.Resolve(Plan("query_worker_us", groupBy: new[] { server }), Now, Now.AddDays(-10), RollupAvailability.All, RollupCoverage.Unknown);
         Assert.Equal(ComposeSourceTier.Hourly, route.Tier);
     }
 
@@ -127,7 +127,7 @@ public void ProcedureStats_SchemaName_NowRoutes()
     {
         /* schema_name was added to the procedure_stats CAGG in the reshape (#1624) → it now routes. */
         var schemaName = MeasureCatalog.Dimension("procedure_stats", "schema_name")!;
-        var route = ComposeSourceRouter.Resolve(Plan("proc_worker_us", groupBy: new[] { schemaName }), Now, Now.AddDays(-10), RollupAvailability.All);
+        var route = ComposeSourceRouter.Resolve(Plan("proc_worker_us", groupBy: new[] { schemaName }), Now, Now.AddDays(-10), RollupAvailability.All, RollupCoverage.Unknown);
         Assert.Equal(ComposeSourceTier.Hourly, route.Tier);
         Assert.Equal("procedure_stats_hourly", route.CaggRelation);
     }
@@ -136,7 +136,7 @@ public void ProcedureStats_SchemaName_NowRoutes()
     public void QueryStore_RoutesByComposerDims()
     {
         /* query_store_stats routes by module_name/query_hash (the reshaped composer dims). */
-        var route = ComposeSourceRouter.Resolve(Plan("qs_executions"), Now, Now.AddDays(-10), RollupAvailability.All);
+        var route = ComposeSourceRouter.Resolve(Plan("qs_executions"), Now, Now.AddDays(-10), RollupAvailability.All, RollupCoverage.Unknown);
         Assert.Equal(ComposeSourceTier.Hourly, route.Tier);
         Assert.Equal("query_store_stats_hourly", route.CaggRelation);
     }
@@ -145,7 +145,7 @@ public void QueryStore_RoutesByComposerDims()
     public void QueryStore_OldWindow_RoutesToDailyCagg()
     {
         /* query_store_stats_daily now exists → a 40-day window routes to it, not the 21d-capped hourly. */
-        var route = ComposeSourceRouter.Resolve(Plan("qs_executions"), Now, Now.AddDays(-40), RollupAvailability.All);
+        var route = ComposeSourceRouter.Resolve(Plan("qs_executions"), Now, Now.AddDays(-40), RollupAvailability.All, RollupCoverage.Unknown);
         Assert.Equal(ComposeSourceTier.Daily, route.Tier);
         Assert.Equal("query_store_stats_daily", route.CaggRelation);
     }
@@ -173,7 +173,7 @@ public void RouteThresholds_StayBelowRetentionHorizons()
     public void OldWindow_NoRollupsInStore_RoutesRaw(int ageDays)
     {
         var route = ComposeSourceRouter.Resolve(
-            Plan("query_worker_us"), Now, Now.AddDays(-ageDays), RollupAvailability.None);
+            Plan("query_worker_us"), Now, Now.AddDays(-ageDays), RollupAvailability.None, RollupCoverage.Unknown);
         Assert.Equal(ComposeSourceTier.Raw, route.Tier);
     }
 
@@ -187,15 +187,15 @@ public void HourlyMissing_DegradesOnlyTheTableThatLostIt()
     {
         var partial = RollupAvailability.All with { QueryGrainHourly = false };
 
-        var queryRoute = ComposeSourceRouter.Resolve(Plan("query_worker_us"), Now, Now.AddDays(-10), partial);
+        var queryRoute = ComposeSourceRouter.Resolve(Plan("query_worker_us"), Now, Now.AddDays(-10), partial, RollupCoverage.Unknown);
         Assert.Equal(ComposeSourceTier.Raw, queryRoute.Tier);
 
-        var procedureRoute = ComposeSourceRouter.Resolve(Plan("proc_elapsed_us"), Now, Now.AddDays(-10), partial);
+        var procedureRoute = ComposeSourceRouter.Resolve(Plan("proc_elapsed_us"), Now, Now.AddDays(-10), partial, RollupCoverage.Unknown);
         Assert.Equal(ComposeSourceTier.Hourly, procedureRoute.Tier);
         Assert.Equal("procedure_stats_hourly", procedureRoute.CaggRelation);
 
         var qsPartial = RollupAvailability.All with { QueryStoreGrainHourly = false };
-        var qsRoute = ComposeSourceRouter.Resolve(Plan("qs_executions"), Now, Now.AddDays(-10), qsPartial);
+        var qsRoute = ComposeSourceRouter.Resolve(Plan("qs_executions"), Now, Now.AddDays(-10), qsPartial, RollupCoverage.Unknown);
         Assert.Equal(ComposeSourceTier.Raw, qsRoute.Tier);
     }
 
@@ -205,7 +205,7 @@ public void HourlyMissing_DegradesOnlyTheTableThatLostIt()
     public void DailyAgeWindow_DailyMissing_FallsToHourly()
     {
         var partial = RollupAvailability.All with { QueryGrainDaily = false };
-        var route = ComposeSourceRouter.Resolve(Plan("query_worker_us"), Now, Now.AddDays(-40), partial);
+        var route = ComposeSourceRouter.Resolve(Plan("query_worker_us"), Now, Now.AddDays(-40), partial, RollupCoverage.Unknown);
         Assert.Equal(ComposeSourceTier.Hourly, route.Tier);
         Assert.Equal("query_stats_hourly", route.CaggRelation);
     }