Daily rollups: days an earlier hourly repair left short are rebuilt once after the upgrade (#4716)

[erikdarlingdata]

Fixes #4716.

Why

Before the fix that stops it (#4739), an hourly hole repair left the day above it partial in every daily rollup that already held a bucket for that day. A day older than the daily policy's 3-day window is never refreshed again, so it stays short, and it becomes the only copy once the hourly rows age out. #4739 stops new cases. Days already left short stay short, so this adds a one-time check that finds and rebuilds them.

What changes

The heal runs inside the start-time hole repair, after the repair's completion stamp, on the same connection and while the repair's running flag is still set. A failure in it never changes the stamp, the repair's summary line, or the service start. Plain-PostgreSQL mode never reaches it.

• TimescaleSupport.MaterializationHoles.cs (next to A repaired hourly hole refreshes the days its dependent dailies already hold, so no partial day stands (#4716) #4739's code):
  • PartialDailyHealOrder() lists the non-frozen dailies with their source, derived from DailyAggregates and RollupViews. A daily built on another daily comes after it (query_store_stats_interval_daily before query_store_stats_daygrain_daily).
  • PartialDailyHealDays(sourceOldest, utcNow) is the day window: each complete day from the day after the source's oldest materialized bucket, up to but not including AlignDown(utcNow, 1 day) minus the daily policy's 3 days.
  • PartialDailyDaySamplesSql reads sum(sample_count) for one day on the source's materialization hypertable and on the daily's materialization hypertable, never the views, in one statement per day, behind the same OFFSET 0 fence DailyRowExistsAsync uses. The daily side is read first and the source side only when the daily has a row for that day.
  • PartialDayNeedsRefresh(sourceSamples, dailySamples) is the pure decision.
  • HealPartialDailyAsync walks one daily. Each partial day is refreshed on its own, one at a time, with RollupBackfill.RepairAsync (SQLSTATE 42883 falls back to RunSliceAsync, as RefreshDependentDailiesAsync does). Each refresh logs at Information with its duration in seconds and both sample totals. The healed days then chain to the daily's own dependents with RefreshDependentDailiesAsync. A failure logs one Warning (the daily, the day, the SQLSTATE), stops that daily and returns an incomplete outcome. Cancellation propagates.
• PartialDailyHeal.cs (new, shaped like PlanForceActionDetailScrub): orders the dailies, keeps the marker, adds up the run. Marker: collect.collector_state under DarlingObservability.FleetServerId, collector partial_daily_heal, state_key the daily's view name, value 4716, written for a daily only when its whole walk and every refresh succeeded. No migration.
• DarlingWorker.cs: RunPartialDailyHealAsync calls it after the stamp and writes one Information line per run, including a run with nothing to do: dailies checked, already done, days compared, partial, refreshed, days chained on, failures, seconds. A cancelled run says so; any other failure is one Warning.

Two changes to #4739's own path, both in TimescaleSupport.MaterializationHoles.cs:

• Per-refresh logging. RefreshDependentDailiesAsync now logs each forced daily refresh at Information with the daily, the day and its duration in seconds, the same shape as the heal's per-day line. The existing per-daily summary line stays. The function now returns the (daily, day) pairs it refreshed (the dailies chained behind them included), each added once its refresh succeeds, and its callers count them. A daily that fails partway therefore counts the days it did refresh; before, it counted none of that daily's days.
• Each day refreshes once per closed range and is counted once. When a range closes in the seam loop, RefreshDependentDailiesAsync refreshes the dependent dailies' complete days, and the older ChainedDailyRange chase then refreshed the same days again, so DailyBucketsChained counted them twice. RepairRangeAsync now returns the pairs its refresh made (empty unless the range closed), and ChainDailyAsync passes them to the new pure function ChaseRangesAfterRefresh, which returns the chase range minus the days already refreshed for the successor daily, as contiguous whole-day ranges. The chase runs once per remaining range and counts only those days. It still materializes the days the daily holds no bucket for yet, which is what it exists for; its hourly-hole re-scan now runs per remaining range instead of over the whole chase range. If a later range throws, the days already chased stay counted. The catch-up chase (a seam that was already closed) does not go through this and is unchanged.

Two calls where the design left a choice:

• The decision refreshes only when both sides hold rows and the daily holds FEWER samples than its source. "Totals differ" would also refresh a daily that holds MORE than its source, which is the daily outliving rows its source lost and may be the only copy; a refresh from that source would shrink it. A partial day is exactly missing samples, so the narrower rule covers the defect and never discards data. A source with rows and no daily row is left to the hole scan, a daily row with no source rows is left alone, and equal totals need nothing.
• "Days compared" counts days where both sides hold rows. A day with no daily row is not compared, and the statement does not read the source for it.

The live test seeds collect.query_store_stats (other live tests in this suite already insert its raw rows), through query_store_stats_hourly and query_store_stats_daily.

Refresh safety

The forced daily refreshes (#4739's and the heal's) run one at a time on one connection, only on the maintenance path, and never in parallel with each other or with the heal. How each was checked, by reading the code and git grep, not by a concurrency test:

• One at a time on one connection: every refresh is an awaited call inside a sequential foreach (the per-daily and per-day loops of RefreshDependentDailiesAsync and the per-day loop of HealPartialDailyAsync), on the repair's own connection. There is no Task.WhenAll or parallel loop in either. The seam chase, which now runs once per remaining range, awaits each range in a foreach in ChainDailyAsync on the same connection.
• Maintenance path only: the only caller of RepairMaterializationHolesAsync is RunMaterializationHoleRepairAsync (the start-time repair, and the Periodic pass's relaunch when the repair epoch is stale), and the only caller of RepairMaterializationSeamsAsync is the retention re-evaluation pass. Neither is a collection cycle.
• Not in parallel: _materializationHoleRepairRunning is set at the top of RunMaterializationHoleRepairAsync and cleared in its finally, and the heal runs before that finally. The retention re-evaluation pass (the seam pass) returns early while the flag is set, and the Periodic relaunch checks the flag, so no other repair refreshes an aggregate while the heal runs. A daily's own refresh policy job can still land on the same aggregate; RollupBackfill already retries the 55P03 that produces.

Cost

Measured on a production store before this change: one statement summing sample_count over 84 days of the query_store_stats hourly took 265 s cold and 19 s warm. The heal reads one day per statement, so each statement takes a few seconds. It runs one statement at a time on the repair's own connection, never on a collection connection, and each daily is walked once: its marker stops the next start from walking it again.

Test plan

All live runs are on a throwaway PostgreSQL 18.6 with TimescaleDB 2.30.1.

• Darling.Tests builds with 0 warnings and 0 errors, after merging the newest dev (the last merge was clean and brought in two unrelated commits).
• PartialDailyHealPureTests: 25 of 25 pass. The 20 from before cover CompleteDaysTouched, DependentDailiesOf, the heal's order, the five decision cases, both edges of the day window and the statement's shape. Five are new, for ChaseRangesAfterRefresh: days the refresh already refreshed are not chased again and each day is counted once (a four-day seam, the chase capped to three days, the daily holding three of them: 3 refreshed + 1 chased = 4 distinct days, where the old chase ran all 3 of its days again), every day refreshed leaves nothing to chase, nothing refreshed leaves the whole range, another daily's pairs and days outside the chase change nothing, and an edge day shrinks the range from that end. RED first: with the function returning the whole chase range (the old behavior), 3 of the 5 failed (the main case, every day refreshed, an edge day); the other 2 are identity cases and passed on that stub too.
• PartialDailyHealLiveTests: 1 of 1 passes. Five days are seeded; the day whose daily is short is rebuilt (48 samples become 50, equal to its hourly); a day that already matches keeps a marker value a refresh would erase; a day with no daily row still has none; a day whose daily holds one more sample than its hourly keeps it; the oldest day is not compared. The summary says 1 refreshed, the marker is set for all 7 dailies, the second run finds every daily already done and compares and refreshes nothing, and a cancelled run writes no marker.
• DailyAfterHoleRepairLiveTests: 1 of 1 passes.
• FrozenRollupLiveTests: 30 of 30 pass (the 29 existing ones, which include every seam-chase pin on DailyBucketsChained, plus the new one below).
• New live test Outage_SeamCapSplit_TheDayTheDailyAlreadyHolds_IsRefreshedOnceByTheClosedRangeAndCountedOnce passes. It uses the 30-hour cap-split seam: pass 1 repairs 24 buckets; between the passes the successor daily is refreshed over the seam's first day, leaving a partial day (18 of 24 hours, worker time 16200); pass 2 closes the last 6 buckets, DailyBucketsChained is 1, and the day is whole (21600, equal to its hourly total). With the chase changed back to ignore the refreshed days it fails: expected 1, actual 2.
• Fail-first proof for the heal's live test: with PartialDayNeedsRefresh forced to false, PartialDailyHealLiveTests FAILS (expected 1 refreshed, actual 0); restored, it passes. The plant is not committed.
• EXPLAIN (ANALYZE) of the per-day statement (the shape of PartialDailyDaySamplesSql, prepared with its two timestamp parameters, on two hypertables with the two columns it reads, 60 days of hourly rows in the source, one day missing from the daily): for that day with no daily row the source side (the scalar sub-select, shown as InitPlan 2, with its aggregate and both scans) reads never executed and the daily side reads 1 buffer; for a day with a daily row the source side runs (72 rows, 4 buffers).
• Census tests that read this change's sources: StartupCommandTimeoutTests, StorageCommandTimeoutTests, DocCommentHygiene, LivePostgresCollectionHygieneTests, LiveCleanupConversionRatchetTests, McpPayloadContractCensusTests, AlertReadFailureSurfaceTests: 232 of 232 pass.
• Full Darling.Tests suite with no PostgreSQL connection set (live tests skip), on the final tree after merging dev: 17186 total, 0 failed, 1123 skipped, 1 not run. The first full run had one failure, TsqlConventionGuardTests.TheMemberScan_ReadsEveryDeclarationWhole: its member scan stopped short of PartialDayNeedsRefresh because of the is { } property patterns in that expression body. The member now compares the nullable totals with HasValue and Value (same decision; the pure decision cases still pass) and the guard passes with KnownTruncatedRanges unchanged.
• git diff origin/dev --stat shows six files, 1245 insertions and 20 deletions (the deletions are edited lines of RefreshDependentDailiesAsync, RepairRangeAsync, ChainDailyAsync and their comments).
• CI's own run of the PostgreSQL job on this PR (not run here).

CHANGELOG

SECTION: Fixed
ENTRY:

• Daily rollup days an earlier hourly repair left short are rebuilt once after the upgrade ([Darling store: a repaired hourly hole leaves an existing daily bucket partial, and it becomes the only copy after 90 days #4716]) - After the upgrade, the service checks each daily rollup against the hourly data under it, once, and rebuilds any day an earlier hourly repair left short, one day at a time. The repair logs each daily day it rebuilds with the seconds it took, and a day rebuilt after a repaired seam range is no longer rebuilt twice and counted twice in the repair's chained-days count.
  REF:
  [Darling store: a repaired hourly hole leaves an existing daily bucket partial, and it becomes the only copy after 90 days #4716]: Darling store: a repaired hourly hole leaves an existing daily bucket partial, and it becomes the only copy after 90 days #4716