Repository navigation
The raw purge gate judges each rollup against the rows that rollup can hold (#4300) - #4432
Merged
Merged
Conversation
Pins the fix already on this branch: the floor case (both successors refreshed, expect Covered) and the query-grain case (a hole at the CPU-unknown floor hour, expect Short then Covered once refreshed). Both fail on the pre-fix code and on a mutation that reverts to the shared source floor.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #4300 (item 4).
Why
Since #4423, the collector writes
query_statsrows with a real interval but no attributable worker time (CPU-unknown rows). The raw purge gate used to judge every consumer ofquery_statsagainst ONE shared, filtered floor. When raw's oldest hour holds only such rows, one consumer's own row filter can reject that hour forever while another consumer's filter admits it — and the shared floor pinned every slot to whichever filter was used to compute it, holding the whole gate short even once the OTHER consumer had fully caught up. This PR gives each coverage slot its own filtered source floor, built from that slot's own materialized view definition, so a slot is only held to the rows its own rollup could ever hold.What changes
RetentionArmSafetySqlnow emits one unfilteredsource_oldestcolumn (unchanged, used only for the empty-store check), then per coverage slot a filteredsource_oldest_inext tocoverage_oldest_i. Each slot's filter comes from that consumer's own materialized view definition; a slot with no filter of its own falls back to the same interval-honest filter used before this change.MeasureRetentionCoverageAsyncreads column 0 for the empty-store check as before, then walks pairs(source_oldest_i, coverage_oldest_i)starting at column 1 instead of comparing every slot to one shared column. A slot whose own filter admits no rows reads Covered outright (nothing for that consumer to be short against); otherwise the same NULL-coverage-is-Short / coverage-later-than-source-is-Short rule applies, now against that slot's own floor.This lets
query_stats's two consumers — the query-grain rollup and the database-grain rollup added by #4423 — each hold their own floor: a CPU-unknown hour that one rollup's filter admits and the other rejects can no longer hold the whole gate short forever.Test plan
New live class
RawGatePerSlotSourceFloorLiveTests(own scratch database,TimescaleSupport.IsRawTierDropSafeAsync— the product's own read path).The floor case: raw's oldest hour holds only CPU-unknown rows (no worker time, a real interval, real execution count); the next three hours hold normal rows; both successors refreshed over the whole range. Expect Covered — the database-grain successor's own filter never admits the CPU-unknown hour, so its own floor starts at the first normal hour. RED on the pre-fix commit (dev before this change):
FloorCase_DbSlotOwnFloorSkipsCpuUnknownHour_Coveredfails itsAssert.True(safe, ...)— the shared floor pinned the gate to the CPU-unknown hour, which the database-grain successor can never materialize, so the pre-fix code reads Short forever.The query-grain case: same seed, but the query-grain successor is deliberately left with a hole at that same hour (refreshed from the next hour onward only). Expect Short — the query-grain successor's own filter DOES admit CPU-unknown rows, so its own floor is that hour, and it is missing. Refreshing that one hour flips the verdict to Covered. This proves the fix narrows each slot to its own floor rather than loosening the gate generally:
QueryGrainCase_HoleAtOwnFloor_ShortThenCoveredfails its second assertion (Assert.True(coveredSafe, ...)) on the pre-fix commit for the same shared-floor reason.Mutation: a one-line change forcing every slot back onto the shared, unfiltered-per-relation floor turned the floor-case pin RED (
Assert.True(safe, ...)failed) and the query-grain pin RED as well; reverting restored both to green, confirming the pin actually exercises the per-slot logic.Class totals (in-process, own container, removed after):
RawGatePerSlotSourceFloorLiveTests2/2;TimescaleContinuousAggregateTests+RollupBackfillTests+FrozenRollupLiveTests+RawPurgeTriggerLiveTests+RetentionHoldAndCadenceRawRedirectLiveTests+DocCommentHygieneTests(required) run together: 185/185, all green.Darling.Tests.csprojandLite.Tests.csprojboth build 0 errors / 0 warnings withEnableWindowsTargeting=true. Both targetnet10.0-windowsand cannot run in-process here as WPF classes;Darling.Tests.dllran the tests above in-process per the repo's proven recipe,Lite.Testsstays build-only.Other coverage-gated relations keep no source filter. The interval-honest filter applies only to
query_statsandprocedure_stats, whose successors bake it in.query_store_stats, and every layered rollup whose own retention waits on its consumer, keep no WHERE, as before. Applying the filter there referenced columns those relations don't carry. The coverage query then failed, and held policies never armed.QueryStoreCorrectedRollupLiveTests8/8,TimescaleSupportTests70/70,RollupBackfillLiveTests(6 passed, 3 skipped), and the held-policy re-arm test (no Warning) pass on a local rig with this rule.CHANGELOG entry
SECTION: Fixed
ENTRY:
REF:
[The raw purge gate judges each rollup against the rows that rollup can hold (#4300) #4432]: The raw purge gate judges each rollup against the rows that rollup can hold (#4300) #4432