Repository navigation
Hold the raw purge on any hole in the frozen legacy's span, and fill the successor down to raw's floor (#4301) - #4401
Merged
Conversation
Adds an optional third window kind to MaterializationHoleScanWindows, anchored inside a frozen legacy's own materialized span (below its last bucket), for a pre-freeze outage hole H2 identified as invisible to both the existing seam and ordinary windows. Purely additive: absent callers get the unchanged two-window shape. Pin A only (pure-function unit tests). The SQL probe (RetentionArmSafetySql), the RepairMaterializationHolesAsync branch, and the UNKNOWN-vs-provably-unrepairable classification are NOT in this commit -- see the handoff note.
Adds LegacySuccessorHoleExistsSql beside MaterializationHoleScanSql: one OFFSET-0-fenced EXISTS expression a caller can drop into any query, true when raw admits a row in a bucket that neither the legacy nor the successor has materialized. Intended to be shared byte-identical by RetentionArmSafetySql (the gate) and the repair walk's interior/seam branch so the two can never disagree about what a hole is. WIP: nothing calls this yet. RetentionArmSafetySql is NOT yet rewired to be bucket-level over the legacy's interior and the seam (the brief's actual ask) and no pins exist. See lane-4301-1c.md for the handoff.
…#4301) Replaces the row-level seam-only probe in RetentionArmSafetySql's legacy-stitch branch with the shared bucket-level hole definition (LegacySuccessorHoleExistsSql, added on this branch's prior commit) so the gate and the repair walk share one hole definition text- identical. The probe now also reaches the legacy's own interior, not only the seam above it, and is bounded on the successor's first bucket above the legacy's last (not min(bucket)), so an interior repair moving the successor's floor down cannot make the probe miss the seam. Updates the doc comments on RetentionArmSafetySql to describe the new bucket-level probe and its bound, and updates/adds unit pins in TimescaleContinuousAggregateTests.cs for the new SQL shape and for filter parity between the gate and the walk. Does not touch the repair walk's own interior/seam branch (next lane) or the non-legacy coverage path.
Four live tests in FrozenRollupLiveTests.cs against IsRawTierDropSafeAsync, modelled on the existing Outage_Seam* tests' scaffolding: - InteriorHole_BelowLegacysLastBucket_ReportsRawPurgeNotSafe (A): a gap inside the legacy's own span reports Short. RED on e970834 (the old probe never looks inside the legacy's span). - InteriorRepairMovesSuccessorFloorBelowSeam_TrapDoesNotHideTheHole_ ReportsRawPurgeNotSafe (B): a successor bucket below the legacy's last bucket (as an interior repair leaves it) must not let a seam hole above it hide. RED on e970834 (the old s.mn-bounded probe collapses to an empty range and misses the seam). - NoGapAnywhere_ReportsRawPurgeSafe (C): passes on both. - GapBelowRawsFilteredFloor_IsInvisibleToTheProbe_ReportsRawPurgeSafe (D): passes on both. All four GREEN on this branch; A and B verified RED on e970834 by building and running the same file in a detached worktree of that commit.
…ot the bare constant RetentionArmSafetySql's LegacySuccessorHoleExistsSql call now passes MaterializationHoleSourceFilterFor(successor's own CREATE) instead of the bare IntervalHonestSourceFilter constant -- the same filter the repair walk reads off the same successor's CREATE. query_stats_db_interval_hourly's own filter (delta_worker_time IS NOT NULL AND sample_interval_seconds IS DISTINCT FROM 0) is strictly wider than the bare constant, so the two disagreed about which raw rows count for that successor. Also applies the per-successor filter to that slot's fromExpr floor (the successor's own filtered raw floor), matching the walk's own floor computation. source_oldest's own sourceWhere is left as the pre-existing, conservative bare-constant shape -- untouched per brief. Rewrote LegacySuccessorHoleProbe_UsesSameFilterAsTheRepairWalk_ForEverySupersededSuccessor to exercise the actual generated gate SQL (RetentionArmSafetySql) rather than comparing two constants -- the prior form could never fail regardless of what the gate's code did, since it never called the gate. Pin now passes.
…IST) Refactored LegacySuccessorHoleExistsSql into a thin EXISTS(...) wrapper around a new private LegacySuccessorHoleBodySql, and added LegacySuccessorHoleScanSql -- the LIST form the repair walk needs (SELECT hb.bucket ... ORDER BY hb.bucket), wrapping the SAME body. TEXT-IDENTICAL body, so the gate's EXISTS probe and the walk's bucket list can never disagree about what a hole is. Added a unit pin (LegacySuccessorHoleExistsSql_AndLegacySuccessorHoleScanSql_ShareTheIdenticalBody) asserting both wrappers' generated SQL contains the identical buckets clause verbatim.
…floor The walk's #4186 seam-fix block lowered seamFloor to the frozen legacy's last bucket + one bucket width. Per the ruling (#4301 comment 5844202704), the walk now lowers it to raw's own filtered floor (AlignDown'd), the same bound RetentionArmSafetySql's gate probes from, whenever that reaches further back than the successor's own floor. Everything downstream (the successor-only scan, the newest-first cap and walk, the shared cap with the ordinary window) is unchanged and is itself the contiguous downward fill: RollupCoverage.StitchedRelationSql splits its read at the successor's floor, so every row below it must land as a successor bucket for the stitch to read each row exactly once. Removed the now-dead legacyInteriorFrom/legacyInteriorTo third-window parameters from MaterializationHoleScanWindows and the two pins that exercised them (ScanWindows_LegacyInterior_*, ScanWindows_NoLegacyInterior_*): under this ruling a legacy-interior hole is repaired by the same newest-first seam descent as everything else below the successor's floor, so the separate oldest-first interior branch never gets built. Removed LegacySuccessorHoleScanSql (the walk's list-form twin of the gate's LegacySuccessorHoleExistsSql) since the walk no longer shares that hole definition; RetentionArmSafetySql keeps LegacySuccessorHoleExistsSql for its own probe.
Three pre-existing FrozenRollupLiveTests seam tests expected the old seam-tail-only fill (2-bucket seam above the legacy boundary, or 11 of 35 seam buckets on the wide-seam pass). The #4301 fix lowers the walk's seam floor to raw's own filtered floor, so the walk now fills contiguously down to raw's floor: 7 buckets (not 2) on the two 6-hour tail tests, and a 36-bucket seam (not 35) split 24/12 (not 24/11) on the wide-seam test. Updated the expected counts, floors, and comments to match; no assertions were loosened.
Five new live tests in FrozenRollupLiveTests.cs against RepairMaterializationHolesAsync / IsRawTierDropSafeAsync / the StitchedRelationSql read path, modelled on lane 1d's pins A-D: - ARepairedShape_SuccessorFloorAtOrBelowRawsFloor_RepairsNothing (1): an A6-backfilled shape where the walk has nothing left to do. - InteriorHole_RepairedByTheWalk_ThenReportsCovered (2): an interior hole reads Short, converges to Covered after enough walk passes. - StitchedReadThroughTheWalk_KeepsEveryRow_AndSuccessorFloorReachesRawsFloor (3): a stitched sum read (RollupCoverage.StitchedRelationSql) over [raw floor, now) is identical before and after the walk, and the successor's floor reaches raw's floor afterward. RED on abc8ce3, verified in a detached worktree: the old walk leaves the floor at the seam tail (b+6h) instead of raw's floor (b). - CapAcrossPasses_LeavesNoGapBetweenPasses (4): a 31-bucket range wider than the 24-bucket cap fills 24 then 6 with no gap between passes. - ProbeFailure_UnknownSourceState_ReportsRawPurgeNotSafe (5): renaming the successor CAGG mid-run makes the gate's probe fail; the Unknown verdict answers false, same as a measured Short. All 19 tests in the class (14 existing + 5 new) green in-process on a local TimescaleDB 2.30.1-pg18 rig. Also re-ran TimescaleContinuousAggregateTests, MaterializationHoleRepairTests, MaterializationHoleRepairLiveTests, RollupBackfillLiveTests, RetentionReevaluationLiveTests: 73 passed, 3 skipped, 0 failed -- unaffected by the #4301 change.
erikdarlingdata
marked this pull request as ready for review
September 26, 2026 10:59
This was referenced Sep 26, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #4301.
Why
A known limit of #4186. The gate (
RetentionArmSafetySql) trusted the frozenlegacy's whole span as coverage; a hole INSIDE the legacy's own span, or a
seam whose lower bound sat below the legacy's max, could read Covered
forever with no self-release, and the repair walk that was supposed to fix
it stopped at the legacy's boundary instead of reaching raw's own floor.
What changes
RetentionArmSafetySql/LegacySuccessorHoleExistsSql)is bucket-level over the legacy's interior AND the seam, bounded by the
successor's first bucket strictly ABOVE the legacy's last bucket — never
by
s.mn, which an interior repair can move below the legacy's boundaryand hide the seam behind an inverted range.
applied per-consumer inside the probe, not just once at the top.
RepairMaterializationHolesAsync's seam block) lowersits
seamFloortoAlignDown(raw's own filtered floor)whenever that'sbelow the successor's own floor. The walk now fills the successor
CONTIGUOUSLY downward from its own floor, newest-first, capped per pass,
until it reaches raw's floor — not just the tail above the legacy's
boundary.
This replaces the issue body's "a matching repair window refreshes the successor at those hours" (see the discussion on #4301).
RollupCoverage.StitchedRelationSql) split at the successor's own first bucket. They read the frozen legacy below it and the successor from it upward.Behaviour changes
Gaps left by earlier versions' purges below the raw floor cannot be
detected or repaired; from this version the purge holds until every
detectable hole is repaired. On a store that is not A6-backfilled, the walk
now fills the successor down to raw's floor over several passes (bounded by
MaterializationHoleRepairCapBucketsper pass), rather than stopping at theold seam-tail-only fill.
Test plan
All live tests run in-process on a local TimescaleDB 2.30.1-pg18 container
(
CREATE ROLE darling LOGIN SUPERUSER; CREATE DATABASE darling OWNER darling;,Darling.Tests.dllwith theMicrosoft.WindowsDesktop.Appframework entry stripped from its runtimeconfig).
FrozenRollupLiveTests(19 total, 14 pre-existing + 5 new pins), all GREEN:InteriorHole_BelowLegacysLastBucket_...,InteriorRepairMovesSuccessorFloorBelowSeam_...,NoGapAnywhere_...,GapBelowRawsFilteredFloor_...) — unaffected, still pass.Outage_Seam*tests needed their expected counts andfloors updated for the new contiguous-downward fill (7 buckets instead of
2 on the two 6-hour-tail tests; 36-bucket seam split 24/12 instead of
35/24/11 on the wide-seam test) — no assertion was loosened, only the
numbers the new fill actually produces.
ARepairedShape_SuccessorFloorAtOrBelowRawsFloor_RepairsNothing:GREEN. No-op on an A6-backfilled shape.
InteriorHole_RepairedByTheWalk_ThenReportsCovered: GREEN.Short → repeated walk → Covered.
StitchedReadThroughTheWalk_KeepsEveryRow_AndSuccessorFloorReachesRawsFloor:GREEN on this branch. RED on
abc8ce3c4, verified in a detachedworktree (
git worktree add --detach /tmp/pmw-4301-2-red abc8ce3c4): theold walk leaves the successor's floor at the seam tail (
b+6h) instead ofraw's floor (
b) —Assert.Equal(2026-03-24T00:00:00, newFloor)failedwith actual
2026-03-24T06:00:00.CapAcrossPasses_LeavesNoGapBetweenPasses: GREEN. 31 buckets,cap 24: pass 1 repairs 24, pass 2 repairs the remaining 6, floors adjacent
with no gap.
ProbeFailure_UnknownSourceState_ReportsRawPurgeNotSafe: GREEN.Renaming the successor CAGG mid-run (
ALTER MATERIALIZED VIEW ... RENAME TO) makes the probe's query fail (undefined relation), caught as Unknown,which answers
false— fails closed, same as a measured Short.Also re-ran (no regressions, unaffected by this change):
TimescaleContinuousAggregateTests,MaterializationHoleRepairTests,MaterializationHoleRepairLiveTests,RollupBackfillLiveTests,RetentionReevaluationLiveTests— 73 passed, 3 skipped, 0 failed.Windows suite (Darling.Tests, Lite.Tests): unchecked here — macOS build is
0 errors / 0 warnings for both projects; CI decides the Windows-only run.
Measurement
EXPLAIN (ANALYZE, BUFFERS)ofRetentionArmSafetySqlforprocedure_statson the PIN 2 seed (interior hole at H2, legacy through H5, successor floor
at H6): Planning Time 1.677 ms (Planning Buffers: shared hit=276),
Execution Time 1.226 ms. Small, chunk-pruned index scans throughout
(
Index Only Scan,Index ScanwithChunks excluded during runtime: 0on the relevant ChunkAppend nodes) — no sequential scan of any hypertable's
full span.
CHANGELOG entry
SECTION: Fixed
ENTRY:
walk that stopped short of filling it, on a rollup upgraded from an
earlier release with an outage spanning the upgrade ([Hold the raw purge on any hole in the frozen legacy's span, and fill the successor down to raw's floor (#4301) #4401]) - the gate
now checks the frozen legacy rollup's own interior for gaps, not just the
seam above it, and the repair walk now fills the successor rollup
contiguously down to the raw table's own floor instead of stopping at the
legacy's last materialized bucket.
REF:
[Hold the raw purge on any hole in the frozen legacy's span, and fill the successor down to raw's floor (#4301) #4401]: Hold the raw purge on any hole in the frozen legacy's span, and fill the successor down to raw's floor (#4301) #4401