TimescaleDB availability is probed only at startup - a transient store blip disables the store health checks for the life of the process #3815

Summary

_timescaleAvailable is probed exactly once, at startup, and never re-evaluated. A transient store condition during service start silently downgrades the service to plain-PostgreSQL mode for the entire lifetime of the process — including switching off the very self-health checks that would report the degradation.

Companion to #3812 (same class of defect: a decision that can only be revisited by restarting a service designed to run for months).

The latch

TimescaleSupport.TryEnableAsync has one call site: Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs:1364. The field is also force-cleared in the block's catch at DarlingWorker.cs:1415:

_timescaleAvailable = false;
_logger.LogWarning("TimescaleDB setup failed — continuing in plain-PostgreSQL mode: {Message}", ex.Message);

That catch is deliberately broad, and correctly so — the comment above it is right that "a partially-converted store is fine" and that falling back is always safe. The defect is not the fallback. It is that the fallback is permanent, and the store conditions that trigger it are usually transient: the store restarting, a failover, a lock held by something else, a momentary connection failure. The store-migrate step above it retries with a budget (StartupFailureTriage); this one does not, and never reconsiders.

What the flag turns off for the rest of the run

_timescaleAvailable is read at six decision points after startup:

The first row is the bad one. A store that is genuinely on TimescaleDB, running with a spuriously-false flag, has its compression policy jobs un-self-healed AND cannot report that fact, because the alert that reports it lives behind the same gate. That is the #1581 field incident (uncompressed data grows without bound until the disk fills and collection stops for the whole fleet) with its backstop disabled.

It also means a store in this state cannot raise the Retention Held alert — so #3812's symptom would be invisible too.

Suggested fix

Re-probe on a cadence. TryEnableAsync is already idempotent (it is a CREATE EXTENSION IF NOT EXISTS + detect), tolerant, and cheap.

The natural home is the fleet-level maintenance block that already exists in the same loop — but note the ordering problem: the compression check at :2159 is gated on the flag, so the re-probe has to run before it and outside its gate, or a false flag keeps its own correction from running.

Two shapes, either is fine:

1. Re-probe then converge. On a slow cadence (hourly is ample), if the flag is currently false, re-run TryEnableAsync; if it now returns true, run the setup sequence the startup block runs (convert/compression/CAGGs/retention) and flip the flag. This heals fully.
2. Re-probe only. Flip the flag on a successful detect and let the startup sequence wait for the next restart. Cheaper, and it at least restores the health checks and the alerting, which is where the real damage is.

(1) is the better answer and folds naturally into #3812's periodic convergence pass — if that lands, this becomes "run the convergence pass on a cadence, and let it re-probe first."

Also worth logging

Today a service running in falsely-degraded mode says so exactly once, in a warning at startup, and then looks completely normal forever. Whatever cadence lands, a store that is detected as Timescale-capable while the flag is false should log that it is recovering, at Information.