Held retention policies waited for a restart nobody was going to do: the coverage gate is re-judged on the running service's hourly store-maintenance tick, the sweep counts transitions instead of verdicts and writes one unconditional 'Retention re-evaluation' line per pass, and every operator sentence that made the restart mandatory now makes it optional (#3812)

[erikdarlingdata]

The lie

"Held means held until the coverage gate releases it" — true, with an unstated clause: once someone restarts the service. TimescaleSupport.EnsureRetentionPoliciesAsync is the only thing that arms a held retention policy, and until this PR it had exactly one call site, the service start path. A store that took --backfill-rollups and was then left alone sat held indefinitely after a backfill that had worked, with the Retention Held alert firing every hour and reading like the backfill had failed. Every operator-facing sentence agreed with the code — the runbook's "step 3 is not optional", the alert's "arms at STARTUP … runs on the service start path and nowhere else", the verb's "NEXT: restart" — so the product was consistently telling operators to restart a service to do something the running service should have done itself.

Mechanism

The same sweep now runs on the running service's hourly store-maintenance tick — the _nextCompressionCheckUtc guard in DarlingWorker that already carries the #1581 compression-job self-heal — as its own awaited method (ReevaluateRetentionPoliciesAsync), on its own pooled connection, under one whole-pass budget, with its own catch-all. The start-path call is kept as-is: this is an additional cadence, not a move.

Every step of the sweep was already safe to repeat, and was tested as such before it was scheduled: add_retention_policy(if_not_exists => true) returns -1 for an existing policy and changes nothing (a NOTICE when the arguments match — measured on 2.28.1 — so the hourly pass writes nothing to the store's server log at its default log_min_messages; the WARNING the doc comment recorded is the differing-arguments case the converge fixes on the same pass); the horizon converge is named-config-only and a no-op once converged; the coverage verdict is a three-valued measurement with Unknown touching nothing; arm and hold are idempotent state sets.

What changed inside the sweep is what it counts and what it says. Task<int> → Task<RetentionPolicySweepSummary>. The old int was applied: policies whose create-or-already-exists step succeeded — the count that proves every policy's SQL is valid on the TimescaleDB under test, and nothing about which of them the gate armed or held. It survives as InPlace; the live tests that compared it now read .InPlace. Beside it: Held (a STATE — every policy the gate is holding after this pass), Armed (a TRANSITION — paused → scheduled on this pass), Unchanged (already in the state the verdict asked for, plus every Unknown), ReHeld (the #1877 transition, scheduled → paused, a subset of Held), Indeterminate, Converged, Failed. Every policy lands in exactly one of Held/Armed/Unchanged/Failed. The transition is read off the job's scheduled flag BEFORE the flip through a new read-only statement (RetentionPolicyScheduledSql) rather than by making the flip conditional: the arm/hold pair stays the plain state set RetentionHorizonConvergeCannotArmTests and TimescaleContinuousAggregateTests pin (one statement, one boolean, mirror images), and a pin here holds them to that.

One unconditional INFORMATION line per evaluation, in the #3756 discipline — written on the all-unchanged hour exactly as on the hour that armed something, because a check whose negative outcome is indistinguishable from its non-execution has not reported:

Retention re-evaluation: N policies held, M armed this pass, K unchanged

and at startup the same shape prefixed Retention evaluation at startup:. Per-policy lines are levelled by transition, not silenced (the issue's own objection to an hourly sweep was 24 identical warnings a day per held policy): a hold that was already a hold is Debug on the hourly pass and stays the Warning it was at startup (the line the runbook sends an operator to); a policy ARMED this pass is Information on both passes — before this the release was the one event in the sweep that never got a distinct line; a policy RE-HELD this pass is a Warning on both, because coverage regressing under an armed policy is an event whatever cadence found it. The horizon summary drops to Debug on the hourly pass so that pass has one Information line, not two.

Evidence

The issue's store: two restarts on consecutive days, identical 20/20 retention policies in place, 16 armed, 4 held paused pending backfill — the gate correctly holding four policies whose coverage was genuinely short. That store restarts nightly; a field store on a stable build has no such luck. With this PR the hold on that store is re-judged at :30 past every hour, and the hour after a backfill completes reads Retention re-evaluation: 0 policies held, 4 armed this pass, 16 unchanged with four ARMED lines naming the relations.

Why the hourly tick and not the 30 s loop, and why AFTER the compression read

Retention wants hours: a held policy costs disk by the day, so hourly is far tighter than "next restart" and far looser than the alert loop needs to be, and the compression tick is the same store, the same domain (TimescaleDB background jobs) and the same cadence class. The pass runs after the compression read inside the tick, and the order is #3575's, not taste: the compression read must sample the job catalog at :30 past its minute, half a grid step from the :00 instants the compression policies start on, and twenty coverage probes ahead of it on a large store would slide that sample toward one. The cost of the order is one tick of lag on the Retention Held alert's resolution edge — that alert's read rides inside the compression method and so sees the flags as they stood before this pass's arm — stated in the worker comment, the alert text and the runbook rather than traded for the phase.

The pass is awaited on the serial sweep loop, so it carries #2327's shape: one linked 5-minute budget for the WHOLE pass, not the 300 s bulk-setup deadline each of its statements carries. On a healthy store the pass is seconds (the coverage probe is a chunk-pruned min() per relation, 685 ms cold on the largest store per #2874). A pass cut short leaves its unjudged policies as they were and writes its own WARNING.

The first pass after startup fires within seconds of the start-path ensure (the stamp seeds at MinValue) and is deliberately not skipped: its Retention re-evaluation: line is the proof the hourly path is wired on this store, visible in the same log window an operator reads after a restart.

The tick is named as the home for every "decided at startup, never re-decided" store-side defect: #3812 is its first tenant; #3815, #3816 and #3817 are queued as further tenants and are not built here. A tenant's contract is written on the guard: its own method, its own catch-all, its own await after the compression read, in the order it appears.

Coupling check: does arming move the phase grid?

No — a retention job has no slot on it, so there is nothing to pre-assign or assign at arm time. add_retention_policy here names neither schedule_interval nor initial_start, so the job is created with TimescaleDB's own retention defaults — measured on 2.28.1: schedule_interval = 1 day, fixed_schedule = false, initial_start = NULL, no job_stats row until its first run. The phase grid (CompressionPhaseMinutes, HourlyRefreshPhaseOrder, #3035/#3012) phases compression and hourly-refresh policies, which DO name initial_start. Arming is alter_job(scheduled => true), which names neither next_start nor config. Measured: a never-run policy created paused starts on the scheduler's next pass after the arm (430 ms on the rig) and then runs a day after each FINISH; re-arming an already-armed job leaves next_start unmoved; a hold-then-arm cycle on a job that has run leaves the next_start it had. No other job moves. The log line therefore names no slot, and the code comment on the ARMED line says why.

One consequence worth stating: because the first run lands on the scheduler's next pass, the arm on the hourly tick IS the first purge — the raw tier reclaims in one pass within seconds of the ARMED line, which is what the runbook already promised for the start path.

What it does NOT do

• Does not change the coverage gate's verdict, thresholds, or the PAUSED-at-creation design. Both passes run the identical create / converge / measure / flip sequence; the pass decides what is said, never what is done.
• Does not move the start-path call, and does not make the first hourly pass skip anything.
• Does not add a table, a migration, or a second timer; the retention re-evaluation rides the compression tick's existing stamp.
• Does not make the arm/hold statements conditional — transitions come from a prior read.
• Does not take TimescaleDB availability is probed only at startup - a transient store blip disables the store health checks for the life of the process #3815/Background-job self-heal covers compression jobs only - a dead CAGG refresh or retention job is never detected or re-armed #3816/Store-object convergence runs only at startup - one failed item stays broken until restart #3817 (a coordinator scope fence): the tick is named as their home; their bodies are not built.
• Does not touch the AlertDeliveryChannelTests fixture whose prose quotes the [QUESTION] Since upgrading to 3.7.0, I get an email every hour #3296 alert's own "then RESTART the service" — that is a historical quotation of what the alert said, exercised for channel delivery, not a statement about the product now.

Operator-facing text that moved with the mechanism

The Retention Held alert detail (the restart is now "only if you want it armed immediately"; the hourly Retention re-evaluation: line is named as the confirmation; the one-tick resolution lag is stated; the hand-arm warning gains its hourly consequence), docs/retention-hold-runbook.md (step 3 is a wait, the restart optional; a third confirmation; the trim-race note stays), the --backfill-rollups verb's closing lines ("NEXT: nothing, unless you are in a hurry"), and the Darling README's cadence table.

Tests

Rig: TimescaleDB 2.28.1 / PostgreSQL 18 (Docker), plus the pure pins on the Mac harness.

• Core pin, live (RetentionReevaluationLiveTests): one query_stats row thirty days back holds that policy at creation; pass 1 (Periodic) reads (1 held, 0 armed, 19 unchanged, 0 re-held) and its exact line; a hand-arm parked at next_start => 'infinity' is undone on pass 1b as (1, 0, 19, re-held 1) with the RE-HELD Warning; the coverage is MOVED the way a backfill moves it (refresh_continuous_aggregate on the hourly, then the daily so the hourly tier's own gate stays Covered); pass 2 reads (0 held, 1 armed, 19 unchanged), the ARMED Information line and scheduled = true; pass 3 reads (0, 0, 20 unchanged) with the exact all-unchanged line, no Warning, no ARMED, and exactly one Information line. No restart anywhere in the sequence. Every pass satisfies Held + Armed + Unchanged + Failed == 20.
• Shape pins (RetentionReevaluationTests): the tally arithmetic; both templates exactly and once each, unconditional between the horizon summary and the return (the only if ( is the pass switch, with an else); the prior read is read-only and single-relation; the arm/hold statements carry no flag guard while the converge does (positive control) and the sweep reads → measures → flips in that order; the three-argument overload is the Startup pass and the bare int is gone; per-policy levels by transition and the three "next start" strings gone; the worker's tick order (stamp → compression → retention, same guard, nothing between the awaits), the startup call kept exactly once, the retention pass not inside the compression method; the worker method's connection, pass, budget and three ordered catches with no rethrow; the runbook, verb and README sentences.
• Moved pins: DarlingSelfAlertTests.RetentionHeld_TheDetail_NamesTheHourlyReEvaluation_AndMakesTheRestartOptional (was …NamesTheRestartAsPartOfTheRemedy; now also asserts the ABSENCE of "then RESTART" and "not optional"); RollupBackfillLiveTests asserts the verb's new closing lines and the absence of the old; PayloadDimensionTests anchors the four-argument signature; five TimescaleSupportTests sites read .InPlace.
• Census list run on the harness before push.

Closes #3812