Raw retention purges run only when the service triggers them, never at PostgreSQL start (#4299)

[erikdarlingdata]

Refs #4299.

Why

An armed raw-retention job could run on TimescaleDB's own schedule when PostgreSQL started, before the service's start sweep could hold it. After a long enough outage, that drops raw history no rollup has materialized yet. The restart probe below shows it: with the old shape, 6 of 10 chunks were gone about 5 seconds after PostgreSQL was ready.

What changes

• The three raw retention jobs are never scheduled on TimescaleDB's runner (query_stats, procedure_stats, query_store_stats).
  • Their armed/held verdict is config->>'darling_armed'. A missing key reads as held.
  • Every retention pass converges it, writing only when a job's scheduled flag or armed state actually differs. A DBA's alter_job(scheduled => true) is reverted and logged at Information. That's a converge, not a re-hold: it never arms the job (Held retention policies only arm on service restart - re-evaluate them on a periodic cadence too #3812).
  • Non-raw retention jobs keep today's scheduled semantics.
• The service runs the purge itself (TimescaleSupport.RunRetentionPurgeJobAsync: CALL run_job(id) in its own transaction, lock_timeout 5 s), and only from the hourly Periodic pass (DarlingWorker.TriggerRawPurgeCoreAsync). It needs, all checked in the same pass:
  • a Covered verdict measured by the trigger itself (IsRawTierDropSafeAsync), never the stored darling_armed, which keeps a prior pass's true through an Unknown probe (Retention gate is arm-only, so a policy whose coverage list GROWS is never re-held — a store upgrading into a new consumer keeps purging its source #1877). Short, Unknown or a probe error does not purge;
  • a hole repair that finished with no failures under the current pg_postmaster_start_time(), stamped on the jobs as darling_repair_epoch (microseconds, IS DISTINCT FROM-guarded). A repair with an isolated per-aggregate failure leaves the stamp unwritten;
  • no hole in the range the run would drop, from a fresh scan (HoleFreeThroughAsync). A successor rollup that can't be resolved (mid-rebuild, renamed, dropped) holds the purge instead of reading as hole-free. The hole scan for raw-sourced rollups now starts at raw's actual floor. Deferred ranges are passed empty: a deferred range is still a hole, so the fresh scan finds it.
  • An error inside the trigger's own checks logs at Warning and records gate_error with its SqlState.
• The Startup pass never purges. A new PostgreSQL start, when no repair is running in the process, launches one repair, and a second service honours the stamp. The Periodic pass relaunches the repair when ANY raw job's epoch is stale, and that repair is awaited at shutdown like the start-path one.
• Readers and alerts follow the verdict:
  • Retention Held reads darling_armed for raw jobs;
  • the raw jobs leave the Query Store aggregate tax scales serially with fleet size — measure, alert, and evaluate before large onboardings #2136 cadence query and take their cadence from the trigger: the last run's time against the hourly pass;
  • a new "Raw Purge Over Horizon" alert fires for a raw table over its horizon whatever the verdict, and says why the purge didn't run: not covered, repair pending, a hole, an unresolvable successor, a trigger error, or failed with its SqlState. Each attempt is recorded in config->>'darling_last_purge' (built server-side with jsonb_build_object);
  • it also fires when that record is more than 2 hours old (twice the hourly pass), meaning the trigger itself has stopped running. A record that can't be read neither fires nor clears the alert; it's counted as a read failure;
  • the stuck re-arm never touches a raw job (pinned).
• Docs: RetentionArmSafetySql and the "ordering is not load-bearing" lines now describe the service-triggered purge and its two exceptions:
  • the first start after the upgrade;
  • a DBA's alter_job/run_job before the next converge.
• No migration.

Test plan

Live pins that FAILED on the pre-fix code (behaviour, not only a compile failure):

• The restart probe (rig): old shape, 10 → 4 chunks about 5 s after ready; this branch, 0 drops and 0 job runs over 2 minutes (12 polls).
• RawRetentionServiceTriggeredLiveTests: 3 of 4 failed on the pre-fix commit (converge not wired). The fourth can't compile there.
• RetentionHoldAndCadenceRawRedirectLiveTests, the darling_armed = true case: read Armed = false on an earlier commit.
• DarlingSelfAlertTests, the raw stuck-job warning text: failed on an earlier commit.
• RawRetentionPurgeJobLiveTests: before the fix, the purge primitive returned SqlState 42601 on every call (Npgsql's extended protocol rejects a multi-statement BEGIN; SET LOCAL ...; CALL ...; COMMIT; in one command), so the service-triggered purge could never run. The lock pin now requires 55P03, and a new success pin requires the chunks to drop.

New members, so compile-RED on the pre-fix code:

• RawPurgeTriggerLiveTests (6: drop only when Covered + current epoch + no hole; Short, stale epoch, a hole and the Startup pass all drop nothing; a second service's purge drops nothing more and records its own pass), each also asserting the recorded outcome;
• RawRepairEpochStampTests/RawRepairEpochTriggerLiveTests;
• RawArmedStateConvergeTests;
• the Raw Purge Over Horizon and raw cadence unit pins in DarlingSelfAlertTests.

Pre-existing tests migrated deliberately from scheduled to darling_armed, with no assertion removed: QueryStoreCorrectedRollupLiveTests, RollupBackfillLiveTests, RetentionReevaluationLiveTests, FrozenRollupLiveTests and TimescaleContinuousAggregateTests. Windows CI was green after the migration.

Added after review, each run in-process on macOS:

• The fresh-measurement pins (RawPurgeTriggerLiveTests): a stale darling_armed = true with an Unknown probe, and an unresolvable successor. Both FAILED at runtime on the pre-fix commit (the chunk count went 10 → 4). Both drop nothing now, recording not_covered / gate_unknown.
• RawPurgeTriggerGateErrorTests: a hole scan made to fail (a held lock + statement_timeout) drops nothing and records gate_error with 57014; the clean-repair stamp rule.
• RawPurgeConvergeLiveTests:
  • scheduled = false after every verdict branch (Covered, Short, Unknown);
  • a second converge on a converged store writes nothing, and the revert line logs once;
  • the three-state record read (never written, unreadable, present).
• DarlingSelfAlertTests: a stale record fires; a recent one doesn't; an unreadable record neither fires nor resolves and is counted; the new reason texts.

Class totals at cac3ede73:

• RawPurgeTriggerLiveTests 8/8;
• RawPurgeTriggerGateErrorTests 2/2;
• RawPurgeConvergeLiveTests 5/5;
• DarlingSelfAlertTests 248/248;
• AlertReadFailureSurfaceTests 27/27;
• RawRetentionPurgeJobLiveTests 2/2;
• RetentionHoldAndCadenceRawRedirectLiveTests 3/3;
• FrozenRollupLiveTests 21/21;
• MaterializationHoleRepairTests 20/20;
• TimescaleContinuousAggregateTests 44/44 (after merging dev's Hold the raw purge on any hole in the frozen legacy's span, and fill the successor down to raw's floor (#4301) #4401/Don't re-hold the raw purge for an empty successor mid-upgrade (#4300) #4406).

The commits after cac3ede73 change comment text, one log level (Debug for the expected raw stuck-job line, now pinned at Debug), three test seed names and one doc-block placement. CI is green on d3d7e252a: every Darling build and PostgreSQL test job, and all Lite jobs.

• The full Windows suite (CI), green on d3d7e252a.

Notes for review

• AlertReadFailureSurfaceTests: the new alert's wrapper catch is added to s_exemptions (it matches its StaleMute/WebTls siblings, which get their evidence as parameters). DarlingSelfAlertEvaluator.cs goes (…, 12, 14) → (…, 12, 15), and the total exempt count 30 → 31. The alert's record read is counted separately, as a read failure.
• RetentionReevaluationLiveTests' hand-arm step writes both darling_armed and scheduled, which is how a real re-hold is exercised. A DBA's bare scheduled => true is a converge (logged), not a re-hold.
• For #4186 gate follow-ups: release needs a second start after an outage, plus three smaller holds #4300's first item: the repair the Periodic pass launches runs the same walk, seam included.
• Release-note facts:
  • on a downgrade, the previous version behaves as it does today;
  • the first start after the upgrade still runs the old armed job once, before the service converges it.

Findings during development

• The third raw job: RawTierCoverage in TimescaleSupport.cs names exactly three raw relations — query_stats, procedure_stats, and query_store_stats.
• Does lock_timeout apply inside run_job? Yes. With a second session holding ACCESS EXCLUSIVE on the oldest chunk for 30s, CALL run_job($1) wrapped in BEGIN; SET LOCAL lock_timeout='5s'; ...; COMMIT; raised ERROR: canceling statement due to lock timeout at 5.07s elapsed, not after the full 30s hold. Chunk count was unchanged after the failed call.
• Can a crash leave a retry behind? Yes, safely, in both a clean backend termination and a SIGKILL crash-restart. Blocking CALL run_job on a held lock, then ending the blocked backend (pg_terminate_backend in one trial, SIGKILL in another, with a confirmed crash-restart log line in the SIGKILL trial), left job_stats.total_runs/last_run_started_at unset and the chunk count unchanged over a 10-minute, 20-poll window in both trials. The next hourly pass is the retry path; nothing needed manual cleanup.
• The purge primitive's original SQL failed on every call. RunRetentionPurgeJobSql sent BEGIN; SET LOCAL lock_timeout = '5s'; CALL run_job($1::integer); COMMIT; as one NpgsqlCommand with a positional parameter. Npgsql's extended (prepared-statement) protocol rejects multiple statements in one command (SqlState 42601). The existing lock-timeout pin only asserted false + elapsed<20s, which the 42601 also satisfies, so nothing had shown the success path. Fixed by using an explicit NpgsqlTransaction with SET LOCAL lock_timeout and CALL run_job($1::integer) as separate commands, then CommitAsync. RunRetentionPurgeJobAsync now returns a RetentionPurgeOutcome(bool Ran, string? SqlState) instead of a bare bool, so a caller can see the SqlState when a run doesn't happen.

CHANGELOG entry

SECTION: Fixed
ENTRY:

• An armed raw-retention purge no longer runs when PostgreSQL starts, before the service can hold it ([Raw retention purges run only when the service triggers them, never at PostgreSQL start (#4299) #4391]) - After a long enough outage, TimescaleDB's own scheduler could run the query_stats, procedure_stats or query_store_stats retention job the moment PostgreSQL came up, dropping raw history that no rollup had materialized yet. These three jobs are now never scheduled on TimescaleDB's runner. The service runs the purge itself, from its hourly evaluation only, and only after it measures coverage itself, confirms the hole repair finished cleanly since the current PostgreSQL start, and finds no hole in the range it would drop. A new Raw Purge Over Horizon alert says why a purge didn't run, or that the purge trigger has stopped. The first start after the upgrade still runs the old armed job once.
  REF:
  [Raw retention purges run only when the service triggers them, never at PostgreSQL start (#4299) #4391]: Raw retention purges run only when the service triggers them, never at PostgreSQL start (#4299) #4391