Defer the payload-dimension GC when a dim-feeding purge failed (#1782)

[erikdarlingdata]

Closes #1782.

The payload-dimension GC prunes query_text_dim / query_plan_dim rows whose last_seen has aged past the fact tables' retention plus a margin, on the reasoning that any fact which could still reference that content is gone. The fact purge immediately above it in the same sweep is failure-isolated: a table whose drop_chunks fails, and whose DELETE fallback then also fails, is warned and skipped while the sweep continues straight into the GC. Those rows are still there carrying digests, and their dimension rows get pruned out from under them — the resolving view then serves NULL payload for real collected data, with no error raised anywhere, which reads as a collection outage rather than a retention bug.

A dim-feeding table that failed its purge now defers the GC for that cycle, with one fixed log line:

dimension GC deferred: a dim-feeding table kept rows past its horizon this cycle; dimension content is retained until its purge completes

Self-ending on the next sweep that purges cleanly; no operator step.

Field signature — corrected twice pre-arm, and the reason is the keeper. The line first carried the wording of the originally-ruled guard, "raw purges held", while the shipped trigger was something else entirely. An operator grepping it would have gone after the arming gate, coverage and #1759 territory when the cause was a failed purge statement with a completely different remedy.

The fix is not better words for the cause — it is not naming a cause at all. This guard has TWO trigger classes once the stack lands: a purge statement that failed, and a purge deliberately SKIPPED because its rollup does not cover the table (#1784). Any wording naming one is a lie in the other. So the line names the observable STATE the guard actually detected — a dim-feeding table still holding rows past its horizon, which is the fact that invalidates the GC's cutoff — and the cause stays one line above in the per-table warning, owned by the code that knows which of the two it was.

One fixed line rather than two per-cause lines: the string only earns its keep by being greppable across builds and stores, and two would force an operator to know both to find every deferral while duplicating cause information already on the preceding line.

The test now pins the signature in full rather than by prefix — a prefix pin is exactly what let the wording drift away from the trigger while every assertion still passed. Watched red by restoring the old wording.

The issue's premise was disproved first, and the fix targets a different trigger

#1782 proposed that a HELD tiered retention policy could let facts outlive the cutoff. It cannot. Verified against origin/dev @ 50012266:

There are TWO purges on these tables, and only one is coverage-gated.

1. The tiered policy — RawRetentionInterval = "4 days" (TimescaleSupport.cs:1223), created paused, armed by the Retention policies run their first check immediately — no external window to pause them, caused permanent data loss #1680 gate. This is the one that is HELD on the field box.
2. Darling's own catalog sweep — DarlingRetention.PurgeAsync iterates CollectorCatalog.All and drops chunks at the per-collector horizon (DarlingRetention.cs:143-175). query_stats and procedure_stats are both new(1, 30) (CollectorScheduleDefaults.cs:42-43) → 30 days. No coverage check, no held check; the only conditional is per-table failure isolation.

The dimension cutoff is widest(30) + ChunkIntervalDays(1) + 1 = 32 days (DarlingRetention.cs:247).

So a held policy changes whether facts live 4 days or 30 — both inside the 32-day cutoff. Facts go at 30 by a purge that is not held; dimensions are pruned at 32. Dimensions outlive their facts by two days by construction, and the ordering survives any retention override because both sides resolve through the same retentionDaysFor (raise query_stats to 90 and the cutoff becomes 92).

Guarding on the held state would have been actively harmful. The field box's policies are held right now, so that guard would have deferred the GC indefinitely until #1759 Phase 2 arms them — unbounded dimension growth, which is the exact failure #1768 exists to prevent — to prevent something unreachable by that route. The guard is keyed to the reachable trigger instead: a purge that actually failed.

The flag is tracked by the sweep rather than inferred afterward because the sweep is the only thing that knows — a failed purge is warned and skipped, leaving no state a later query could read back.

Verification

Mutation, watched red on a live PostgreSQL 18.4 + TimescaleDB store:

Mutation	Result
Guard removed (if (false && dimFeedingPurgeFailed))	RED — Assert.Equal() Failure: Values differ / Expected: 1 / Actual: 0: the dimension row was pruned while the fact purge had failed

The failure is injected genuinely, not simulated: the fact table is renamed for the duration, so both the drop_chunks and the DELETE fallback fail against a relation that does not exist. The test then restores it and asserts the same row IS pruned once the purge is healthy, so it cannot pass merely because the GC never ran.

The assertion order is deliberate and worth a reviewer's eye. My first cut asserted the log line before the row count, and the mutation went red on missing log text while the actual data loss went unexamined. Reordered so the substantive property is asserted first; the red above is that reorder's result. A guard that logs and prunes anyway would now fail on the data, which is the failure that matters.

Gc_DeletesDimRowsPastTheHorizon_AndKeepsTheOnesInsideIt stays green — it drives the GC SQL directly and is unaffected.

Spun off: a larger, opposite-signed defect — #1784

Establishing the above surfaced that the ungated catalog sweep bypasses the #1680 coverage gate entirely. On a store where the tiered policy is held because rollup coverage does not reach raw's history, the sweep drops those same chunks at 30 days with no coverage check — destroying exactly the uncovered history the gate is holding the policy to protect. Field fuse: raw from ~Jul 9, coverage from ~Jul 24, so the uncovered slice starts dying ~Aug 8 unless a backfill extends coverage first.

That also inverts #1759's framing — held tiers are not costing disk, they are costing rollup-uncovered history — and I added one comment there pointing at #1784 so the canonical issue's narrative is corrected.

Filed as #1784; fix (clamping the sweep to the gate's own coverage predicate) is a separate PR, deliberately: different trigger, different risk class, and #1784 is data-loss-class with a dated fuse that should not wait on this one's review.

Known interaction — ruled SHIP-WITH-ISSUE, tracked as #1795

query_stats and procedure_stats are both coverage-gated tiers (#1784) and dim-feeding tables (#1782), so those two facts compose: the clamp's skip sets the deferral flag, and on a coverage-lagging store — the field box today — the dimension GC defers on every sweep until a backfill lands. Proven by execution, not inference: a 400-day-old orphan dimension row survives the sweep with nothing failed anywhere.

Shipping it is the right call. The deferral is logically correct — the clamp deliberately preserves those fact rows, so pruning their content would dangle the digests and serve NULL payload for data that is present; the GC's premise is genuinely false while the clamp holds. The cost is recoverable disk that self-corrects at backfill (#1788 extends coverage, the clamp opens, the GC resumes with no operator step), dimensions grow by distinct content only so the accretion is small, and the alternative was the ~Aug 8 data loss this PR exists to prevent. Data loss beats disk growth.

The proper closure — pruning against the oldest SURVIVING digest-carrying fact rather than an assumed horizon, which stays bounded even while the clamp holds — is filed as #1795, non-release-gating.

Build and test

• -t:Rebuild, 0 Warning(s) 0 Error(s): PerformanceMonitor.Darling.Storage, PerformanceMonitor.Darling.Service, Darling.Tests.
• Darling.Tests against live PostgreSQL 18.4 + TimescaleDB 2.28.1 on a fresh database: 3,649 passed / 0 failed / 10 skipped (post-merge with origin/dev).
• origin/dev merged in; the only conflict was the CHANGELOG link-ref tail, resolved keep-both with no re-sorting. Diff versus dev is exactly three files.

Unarmed pending the gauntlet.

Generated with Claude Code