Skip to content

The raw purge gate judges each rollup against the rows that rollup can hold (#4300) - #4432

Merged
erikdarlingdata merged 3 commits into
devfrom
fix/4300-per-slot-source-floor
Sep 26, 2026
Merged

erikdarlingdata merged 3 commits into
devfrom
fix/4300-per-slot-source-floor

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

Refs #4300 (item 4).

Why

Since #4423, the collector writes query_stats rows with a real interval but no attributable worker time (CPU-unknown rows). The raw purge gate used to judge every consumer of query_stats against ONE shared, filtered floor. When raw's oldest hour holds only such rows, one consumer's own row filter can reject that hour forever while another consumer's filter admits it — and the shared floor pinned every slot to whichever filter was used to compute it, holding the whole gate short even once the OTHER consumer had fully caught up. This PR gives each coverage slot its own filtered source floor, built from that slot's own materialized view definition, so a slot is only held to the rows its own rollup could ever hold.

What changes

RetentionArmSafetySql now emits one unfiltered source_oldest column (unchanged, used only for the empty-store check), then per coverage slot a filtered source_oldest_i next to coverage_oldest_i. Each slot's filter comes from that consumer's own materialized view definition; a slot with no filter of its own falls back to the same interval-honest filter used before this change.

MeasureRetentionCoverageAsync reads column 0 for the empty-store check as before, then walks pairs (source_oldest_i, coverage_oldest_i) starting at column 1 instead of comparing every slot to one shared column. A slot whose own filter admits no rows reads Covered outright (nothing for that consumer to be short against); otherwise the same NULL-coverage-is-Short / coverage-later-than-source-is-Short rule applies, now against that slot's own floor.

This lets query_stats's two consumers — the query-grain rollup and the database-grain rollup added by #4423 — each hold their own floor: a CPU-unknown hour that one rollup's filter admits and the other rejects can no longer hold the whole gate short forever.

Test plan

New live class RawGatePerSlotSourceFloorLiveTests (own scratch database, TimescaleSupport.IsRawTierDropSafeAsync — the product's own read path).

  • The floor case: raw's oldest hour holds only CPU-unknown rows (no worker time, a real interval, real execution count); the next three hours hold normal rows; both successors refreshed over the whole range. Expect Covered — the database-grain successor's own filter never admits the CPU-unknown hour, so its own floor starts at the first normal hour. RED on the pre-fix commit (dev before this change): FloorCase_DbSlotOwnFloorSkipsCpuUnknownHour_Covered fails its Assert.True(safe, ...) — the shared floor pinned the gate to the CPU-unknown hour, which the database-grain successor can never materialize, so the pre-fix code reads Short forever.

  • The query-grain case: same seed, but the query-grain successor is deliberately left with a hole at that same hour (refreshed from the next hour onward only). Expect Short — the query-grain successor's own filter DOES admit CPU-unknown rows, so its own floor is that hour, and it is missing. Refreshing that one hour flips the verdict to Covered. This proves the fix narrows each slot to its own floor rather than loosening the gate generally: QueryGrainCase_HoleAtOwnFloor_ShortThenCovered fails its second assertion (Assert.True(coveredSafe, ...)) on the pre-fix commit for the same shared-floor reason.

  • Mutation: a one-line change forcing every slot back onto the shared, unfiltered-per-relation floor turned the floor-case pin RED (Assert.True(safe, ...) failed) and the query-grain pin RED as well; reverting restored both to green, confirming the pin actually exercises the per-slot logic.

  • Class totals (in-process, own container, removed after): RawGatePerSlotSourceFloorLiveTests 2/2; TimescaleContinuousAggregateTests + RollupBackfillTests + FrozenRollupLiveTests + RawPurgeTriggerLiveTests + RetentionHoldAndCadenceRawRedirectLiveTests + DocCommentHygieneTests (required) run together: 185/185, all green.

  • Darling.Tests.csproj and Lite.Tests.csproj both build 0 errors / 0 warnings with EnableWindowsTargeting=true. Both target net10.0-windows and cannot run in-process here as WPF classes; Darling.Tests.dll ran the tests above in-process per the repo's proven recipe, Lite.Tests stays build-only.

  • Other coverage-gated relations keep no source filter. The interval-honest filter applies only to query_stats and procedure_stats, whose successors bake it in. query_store_stats, and every layered rollup whose own retention waits on its consumer, keep no WHERE, as before. Applying the filter there referenced columns those relations don't carry. The coverage query then failed, and held policies never armed. QueryStoreCorrectedRollupLiveTests 8/8, TimescaleSupportTests 70/70, RollupBackfillLiveTests (6 passed, 3 skipped), and the held-policy re-arm test (no Warning) pass on a local rig with this rule.

CHANGELOG entry

SECTION: Fixed
ENTRY:

Pins the fix already on this branch: the floor case (both successors
refreshed, expect Covered) and the query-grain case (a hole at the
CPU-unknown floor hour, expect Short then Covered once refreshed).
Both fail on the pre-fix code and on a mutation that reverts to the
shared source floor.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 26, 2026 16:16
@erikdarlingdata
erikdarlingdata merged commit 8caf5a0 into dev Sep 26, 2026
17 of 18 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/4300-per-slot-source-floor branch September 26, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant