Take raw query-statistics tables out of the catalog sweep's drop path (#4427)

[erikdarlingdata]

Refs #4427.

Why

The daily retention sweep (DarlingRetention.PurgeAsync) gated every collector table's Timescale drop on a floor-only coverage check (IsTieredDropSafeAsync → TimescaleSupport.IsRawTierDropSafeAsync), which compares each rollup's oldest bucket against raw's oldest row. A hole inside the covered span — an outage seam the repair hasn't reached, a range the repair deferred, a failed repair — moves neither floor, so the sweep read Covered and dropped the chunk the service-triggered, fully-gated raw purge is holding for exactly that reason. The gate held precisely when a hole existed; the sweep, on a shorter horizon (operator-controllable down to 1 day), defeated it.

What changes

• On a TimescaleDB store, collect.query_stats, collect.procedure_stats and collect.query_store_stats (membership from TimescaleSupport.RawRelations itself, never a copied list) now leave the sweep's drop path entirely. They are counted neither as purged nor as failed — the sweep logs one Information line naming the owner and moves on. The gated, service-triggered purge (DarlingWorker.TriggerRawPurgeCoreAsync) already owns them with the full gate: a fresh Covered verdict, a current repair epoch, a resolvable successor, and a clean interior-hole scan.

• The skip is decided by whether the TimescaleDB extension is installed in the store (a pg_extension probe on each sweep), not by the service's cached timescaleAvailable flag. A stale flag therefore cannot reopen the drop path. If the probe itself fails, the raw tables are kept that pass and one Warning is logged.

• True plain-PostgreSQL mode (no extension row) is unchanged: there are no rollups there, so the DELETE fallback stays raw's only purge for these tables.

• purge_now now runs the same gated trigger once, on a pooled connection, right after the sweep (when Timescale is available), then reads back each raw relation's outcome and adds a rawTables array to the JSON response: relation, outcome (ran, not_covered, hole, epoch_stale, gate_unknown, gate_error, no_chunks, run_failed) and a plain-language note. When the requested custom retention is shorter than the gated horizon (4 days), every raw entry's note says plainly that the custom value was not applied to it. The gate is never bypassed.

• purge_now reads the database clock before the trigger runs. A stored outcome older than that is reported as gate_unknown ("no outcome was recorded for this pass"), never as this pass's result.

• If the raw step fails (connection or trigger), purge_now still returns the sweep summary, with rawTables carrying gate_error and the message. A shutdown still propagates.

• The report's horizon note comes from TimescaleSupport.RawRetentionSpan, and a custom retention longer than it gets a note that it doesn't apply to raw tables.

Every raw drop path

Site	File:line	Gate
Daily/on-demand catalog sweep (DarlingRetention.PurgeAsync)	DarlingRetention.cs:331 (new skip)	Fixed here — raw tables now skip this path entirely on Timescale
Plain-PostgreSQL DELETE fallback, same sweep	DarlingRetention.cs:~370 (PurgeOneAsync)	Runs only when the store has no TimescaleDB extension (probed each sweep): no rollups exist there, so nothing to defeat
Service-triggered raw purge (Periodic pass)	DarlingWorker.cs:7269 TriggerRawPurgeCoreAsync → CALL run_job(id) (TimescaleSupport.cs:6355)	Full gate: fresh coverage + repair epoch + interior-hole scan (#4299/#4391) — the intended owner
purge_now	DarlingWorker.cs:8576 RunPurgeNowAsync	Fixed here — now runs the same trigger, same full gate, for raw tables
TimescaleDB's own job scheduler for the raw jobs	n/a	Permanently unscheduled (ConvergeRawArmedStateSql) — raw jobs never run on Timescale's own cron; only the service-triggered CALL run_job above fires them
Startup/hourly repair + relaunch check	DarlingWorker.cs:7177 (ReevaluateRetentionPoliciesAsync)	Calls the same gated trigger (TriggerRawPurgeAsync) — no separate drop path
Continuous-aggregate/legacy-hourly retention policies	TimescaleSupport.cs RetentionPolicies	Coverage-gated (#1680/#1905) against their own successors — not one of the three raw tables, out of scope
CLI verbs (DarlingCliCommands.cs)	—	No CLI verb drops raw chunks or rows directly; the backfill/status verbs read state only
Compression policies	—	Compress, not drop — no data loss path; out of scope
Query Store slice repair	QueryStoreSliceRepair.cs:271	A DELETE FROM collect.query_store_stats that removes duplicate rows and reinserts one per slice; not retention, and no rows are lost

No other floor-only raw drop path was found.

Test plan

New file Darling/Darling.Tests/RawTablesLeaveCatalogSweepLiveTests.cs, own-store (ScratchPostgres, not [Collection("live-postgres")]):

• InteriorHole_SurvivesSweep_RawTableLeavesDropPath — seeds query_stats over 10 days, refreshes every successor EXCEPT a 1-day interior window (a real hole well inside the covered span). Confirms the floor-only check (IsRawTierDropSafeAsync) reads true despite the hole, then runs PurgeAsync(timescaleAvailable: true, retentionDaysFor: _ => 1) and asserts the hole's 24 raw rows and its chunk count are unchanged.

  • RED on origin/dev at runtime: Assert.Equal() Failure: Expected: 24, Actual: 0 at RawTablesLeaveCatalogSweepLiveTests.cs:200 — dev's sweep drops the hole's chunk because the floor-only gate is the only check it runs for query_stats.

• PlainPostgres_StillDeletesExpiredRawRows — plain-PostgreSQL mode, confirms the DELETE fallback still purges expired raw rows. GREEN on both dev and this branch (unchanged behavior).

• PurgeNow_RawTableReport_ShowsHoleAndCustomRetentionNote — drives the same hole scenario through TriggerRawPurgeCoreAsync then DarlingWorker.BuildRawTablePurgeNowReportAsync, asserting the raw relation's outcome is hole and the note mentions the custom-retention caveat.

• Mutation: commented the RawRelations.Contains check out (timescaleAvailable && false && ...), rebuilt, reran InteriorHole_SurvivesSweep_RawTableLeavesDropPath — same failure, Assert.Equal() Failure: Expected: 24, Actual: 0 at the same line. Restored the check, rebuilt, reran — GREEN (3/3 in the new class).

• StaleTimescaleAvailableLatch_DoesNotReopenDropPath: the extension is installed but PurgeAsync gets timescaleAvailable: false, with a short retention over an interior hole. The hole's rows survive.

  • Mutation: deciding the skip on the timescaleAvailable argument again (the previous head's logic) fails it with Assert.Equal() Failure: Expected: 24, Actual: 0. Restored, it passes.

• ProbeRawSkipSafe_ConnectionFailure_KeepsRowsAndLogsOnce: a probe that can't connect keeps the raw tables and logs exactly one Warning.

• PlainPostgres_StillDeletesExpiredRawRows now drops the extension from its scratch database first. The TimescaleDB image's template1 carries it, so the test previously never exercised true plain-PostgreSQL mode.

• PurgeNow_RawTableReport_StaleOutcome_ReportsGateUnknown: a real ran record older than the pass start is reported as gate_unknown.

• RawPurgeNowWiringTests: RunPurgeNowAsync calls both the gated trigger and the report builder, and the response carries rawTables.

• PayloadDimensionLiveTests: its two catalog-sweep tests now pin the new contract. Raw rows survive the sweep whether or not the rollup covers them, and dimension content they reference is kept.

Latest run against a local TimescaleDB 2.30.1/PG18 container: RawTablesLeaveCatalogSweepLiveTests 6/6. Together with PayloadDimensionLiveTests, RawPurgeNowWiringTests, DarlingRetentionTests, RawPurgeTriggerLiveTests and DocCommentHygieneTests: 153 passed, 0 failed.

Earlier run, before the probe and report changes:

• RawTablesLeaveCatalogSweepLiveTests: 3/3 pass.
• DarlingRetentionTests: 43/43 pass.
• ScheduledPurgeLaunchTests: 5/5 pass.
• RawPurgeTriggerLiveTests: 8/8 pass.
• AlertReadFailureSurfaceTests: 27/27 pass.
• DocCommentHygieneTests: 77/77 pass.
• DarlingCommandExecutorTests: 32/32 pass.
• Darling.Tests.csproj build (-c Release -p:EnableWindowsTargeting=true): 0 errors, 0 warnings.
• Lite.Tests.csproj build (-p:EnableWindowsTargeting=true): 0 errors, 0 warnings.

Not run locally: the full suite. Darling.Tests/Lite.Tests target net10.0-windows and cannot execute their full run on macOS either way; CI decides the rest.

CHANGELOG entry

SECTION: Fixed
ENTRY:

• The daily retention sweep no longer drops raw query statistics over a hole no rollup holds; the gated service-triggered purge owns those tables, and purge_now reports what it held ([Take raw query-statistics tables out of the catalog sweep's drop path (#4427) #4429])
  REF: [Take raw query-statistics tables out of the catalog sweep's drop path (#4427) #4429]: Take raw query-statistics tables out of the catalog sweep's drop path (#4427) #4429