Make the catalog sweep honor the #1680 coverage gate (#1784)

[erikdarlingdata]

Closes #1784. Stacked on #1789 — see "Stacking" below.

Two purges drop the same raw chunks, and only one honored the #1680 coverage gate.

1. The tiered policy is created paused and armed only once the hourly aggregate covers everything the table holds.
2. DarlingRetention.PurgeAsync's catalog sweep then dropped those very chunks at the per-collector 30-day horizon, with no coverage check at all.

So on a store where the gate is deliberately holding the policy — because the rollup does not reach back over raw's history — the sweep destroyed exactly the uncovered history the gate exists to protect. The gate was not bypassed by a bug in the gate; it was bypassed by an older path that never learned the invariant.

Field fuse on a production instance: raw from ~Jul 9, coverage from ~Jul 24, so the uncovered Jul 9–24 slice begins dying ~Aug 8 unless a backfill extends coverage first — and the product must not depend on an operator running a verb in time.

Both purges now consult the SAME predicate, reached through one shared map of raw tier → covering aggregate (TimescaleSupport.RawTierCoverage), so they cannot judge the same drop differently. The policy setup was refactored onto that map rather than keeping its own copy.

The ruled clamp shape was superseded — accepted, and why

The originally-ruled fix was "clamp the tiered-table cutoff to min(catalog horizon, rollup coverage floor)". That ruling has been reviewed and withdrawn in favour of the binary gate shipped here; this section records the argument so the decision is auditable rather than folklore.

The choice is binary by structure, not by preference. drop_chunks removes the OLDEST chunks first. When rollup coverage lags, the uncovered prefix IS the oldest data. So there is no cutoff value that drops the covered tail while sparing the uncovered head — the operation is all-or-nothing, and any clamp on the cutoff is expressing a choice the mechanism cannot make.

That is why min(horizon, floor) fails in BOTH directions, which is the tell that the shape was wrong rather than the value:

The ruled fix was "clamp the tiered-table cutoff to min(catalog horizon, rollup coverage floor)". I implemented a binary gate instead, because the clamp loses data in one direction and disables the backstop in the other. Worked against the field numbers:

Case	min(catalog horizon, coverage floor)	Outcome
Coverage lagging, Aug 20	min(Jul 21, Jul 24) = Jul 21	drops Jul 9–21 — every row uncovered
Fully covered (healthy)	min(now−30d, coverage_oldest) = coverage_oldest, which is ≤ the oldest row	drops nothing, ever — the backstop stops backstopping

The binary rule is the one the arming gate already uses (coverage_oldest <= source_oldest), and it satisfies every requirement that was set: no-op while coverage lags, self-healing the moment a backfill lands, tiered tables never permanently excluded. Reusing the identical predicate through the ONE shared RawTierCoverage map is also what makes the two purge paths constitutionally unable to judge the same drop differently — a clamp would have been a second, differently-shaped rule for the same decision, free to drift from the gate it is supposed to honour.

Not an outright exclusion of tiered tables, deliberately: a never-armed policy would then mean genuinely unbounded growth. The moment coverage reaches back, the normal 30-day horizon applies again.

Verification — both halves, both watched red

A guard that never drops anything would satisfy the "uncovered survives" half on its own, so the backstop is pinned too. Against a live PostgreSQL 18.4 + TimescaleDB store: a 45-day-old row with the aggregate refreshed only over the recent window (the field state — a rollup that starts later than raw), then the same row with the aggregate force-refreshed back over it.

Mutation	Result
Coverage guard removed	RED — Assert.Equal() Failure: Expected: 1, Actual: 0. The uncovered 45-day row was dropped: the #1784 data loss, reproduced.
Guard forced always-skip	RED — Assert.Equal() Failure: Expected: 0, Actual: 1. The row survived when coverage was sufficient: the backstop stopped backstopping.

Data assertions come before log assertions in both halves, so a mutation lands on the data rather than on missing log text.

Field signature

While the sweep defers on an uncovered tiered table, per cycle, per affected table, at Warning:

Retention purge SKIPPED for query_stats: its rollup does not yet cover the oldest rows, so dropping would delete history no aggregate holds. Resumes by itself once a backfill extends coverage.

Because a skipped table's rows persist past their horizon, this also trips #1789's guard in the same sweep, so the box shows both lines together:

dimension GC deferred: a dim-feeding table kept rows past its horizon this cycle; dimension content is retained until its purge completes

That second line names the observable STATE rather than any cause, precisely because of this PR: it now has two trigger classes — a failed purge statement, and this PR's deliberate coverage SKIP — and naming either would be a lie in the other case. The per-table line directly above it carries which table and which of the two.

After a backfill extends coverage past raw's oldest row, the operator-visible change is a DISAPPEARANCE, not an announcement — worth stating plainly in the client prompt so their agent does not hunt for a success line that does not exist. On the next cycle: both warnings above stop; Retention purge: {Tables} table(s) purged, …, 0 failed shows a higher table count; and the startup Retention policy for query_stats created but HELD PAUSED … line stops for that relation as the policy arms itself with no manual step.

Coupling to #1789, wired here rather than deferred

A coverage-skipped table's rows persist past their horizon — which is exactly the dimension-GC hazard #1789 guards, where the GC's cutoff assumes they are gone. The skip therefore also sets dimFeedingPurgeFailed, so the dimension GC defers in the same cycle. That interaction is why this is stacked rather than branched from dev: on dev the flag does not exist yet, and shipping the skip without it would open the dangling-digest window #1789 exists to close.

Stacking: branched from feature/1782-dim-gc-held-purge-guard. Review git diff origin/feature/1782-dim-gc-held-purge-guard..HEAD — four files, the same set listed below. If the gauntlet would rather have these independent, I can rebase onto dev and move the one-line coupling into whichever merges second; I chose correctness-on-both-branches over review independence and am flagging the choice rather than assuming it.

Two defects found in my own work by the repo's own guard

DocCommentHygieneTests.NoMemberCarriesTwoStackedSummaryBlocks (the #1751 pin) went red twice on this branch. Both times my insertion pushed an existing <summary> block away from the member it documented — IsSafeToArmRetentionAsync in TimescaleSupport.cs, and PurgeOneAsync in DarlingRetention.cs — leaving each member undocumented and its doc stranded above my new one. Per that test's own failure text, both were fixed by moving the block back to its member, not by deleting it. Worth noting because it is the exact class the pin was written for, caught on the first run after the edit.

Known interaction — ruled SHIP-WITH-ISSUE, tracked as #1795

query_stats and procedure_stats are both coverage-gated tiers (#1784) and dim-feeding tables (#1782), so those two facts compose: the clamp's skip sets the deferral flag, and on a coverage-lagging store — the field box today — the dimension GC defers on every sweep until a backfill lands. Proven by execution, not inference: a 400-day-old orphan dimension row survives the sweep with nothing failed anywhere.

Shipping it is the right call. The deferral is logically correct — the clamp deliberately preserves those fact rows, so pruning their content would dangle the digests and serve NULL payload for data that is present; the GC's premise is genuinely false while the clamp holds. The cost is recoverable disk that self-corrects at backfill (#1788 extends coverage, the clamp opens, the GC resumes with no operator step), dimensions grow by distinct content only so the accretion is small, and the alternative was the ~Aug 8 data loss this PR exists to prevent. Data loss beats disk growth.

The proper closure — pruning against the oldest SURVIVING digest-carrying fact rather than an assumed horizon, which stays bounded even while the clamp holds — is filed as #1795, non-release-gating.

Build and test

• -t:Rebuild, 0 Warning(s) 0 Error(s): PerformanceMonitor.Darling.Storage, PerformanceMonitor.Darling.Service, Darling.Tests.
• Darling.Tests against live PostgreSQL 18.4 + TimescaleDB 2.28.1 on a fresh database: 3,650 passed / 0 failed / 10 skipped.

Unarmed pending the gauntlet.

Generated with Claude Code