A transient store outage at startup ends collection for the process lifetime — migrate failures are all terminal, with no retry and no triage

[erikdarlingdata]

A single store-unreachable moment at startup ends collection for the life of the process, with no retry and no distinction from a genuinely broken rung

Surfaced by #2894's residual-2 work and deliberately kept out of it: this is a caller-side error-policy question, not a migration-applier one, so putting it in PgMigrations would have been the wrong place.

The shape

Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs:1011:

:1011    var applied = await PgMigrations.MigrateAsync(migrateConnection, _logger, stoppingToken);
:1012    _logger.LogInformation("Postgres store ready (schema v{Version}, {Applied} migration(s) applied)", …);
:1014  }
:1015  catch (Exception ex) when (ex is not OperationCanceledException)
:1016  {
:1017    _logger.LogCritical("Cannot reach or migrate the Postgres store: {Message}", ex.Message);
:1018    return;
:1019  }

One catch, one LogCritical, one return. Every failure mode is terminal and they are all treated identically:

• the store is momentarily unreachable — a restart, a failover, a network blip, the store still coming up alongside the service
• the advisory lock was held by a peer instance longer than the wait allows
• a rung genuinely cannot apply against this store

The first two are transient and would succeed on a retry seconds later. The third will never succeed and should stay terminal. Nothing distinguishes them, and the service does not try again — collection is over until somebody restarts the process.

Why this is a real gap rather than a defensible default

The very next block states the opposite policy explicitly, and says why. Role provisioning at :1021 carries:

a failure degrades (the Viewer cannot connect as admin/viewer until a later start succeeds) but never kills collection

So the codebase already reasons about degrade-vs-kill for store-adjacent startup work, and lands on degrade one block later. The migrate path has no such reasoning attached — the return reads as an unexamined default sitting next to a documented decision.

The blast radius is the whole point of the product. This is the monitoring tool's own collection loop. A transient store hiccup during service start is exactly the moment an operator is least likely to be watching, and the symptom — no data at all, one LogCritical line — looks like a much larger failure than a five-second outage.

What changed recently that makes this expressible

#2894's residual-2 change makes the advisory-lock waiter independent of an unbounded holder, so lock-wait expiry is now a distinguishable outcome rather than an indistinguishable timeout. That is what makes a retry policy writable at all: before, "couldn't get the lock" and "couldn't reach the store" arrived looking alike.

The design question

1. Which failures are retryable? Connect/transport failures and lock-wait expiry are the obvious candidates. A rung that fails to apply is the obvious non-candidate. Where the boundary sits for anything ambiguous is the actual decision.
2. Bounded retry with backoff, or retry for the life of the process? A monitoring service that retries forever and logs at a sane cadence is arguably right for the transient cases; an unbounded retry against a genuinely broken rung is a log flood.
3. Does a retry re-enter the whole migrate, or resume? MigrateAsync is idempotent rung-by-rung, so re-entering is probably safe — worth confirming rather than assuming, because the answer decides whether retry is a loop around the existing call or something more careful.
4. What does the Viewer/Fleet Health show meanwhile? "Store unreachable, retrying" is a different operator story from silence.

Note there is a nearby precedent to check for consistency rather than invent against: a 6×2 s retry already wraps part of the startup path.

Not verified

No live reproduction — the terminal behaviour is read from source at the lines quoted, not observed by taking a store down mid-start. No measurement of how often this actually fires in practice; the case for fixing it rests on the asymmetry between failure modes and on the adjacent block's contrary policy, not on an incident.

[erikdarlingdata]

Closed by PR #2944. Three refinements to the issue text above, all measured rather than argued.

The premise was understated: the service does not merely idle. The return completes the worker task successfully, so the host process stays up, the MCP and web hosts keep serving, and on Windows the service keeps reporting Running — SCM recovery never fires because the process never exits. Every automated signal says healthy. Darling/README.md said "idles" too and is corrected in the PR.

The advisory-lock bullet is the least reachable of the three transient cases. Contention does not fail — it blocks and is waited out for up to MigrationLockWaitTimeoutSeconds, so only the expiry reaches the retry. That also produced the second cap: an attempt is not quick just because a refused connect is, and 25 attempts against a blocking wait would be ~10 hours rather than two minutes.

DarlingManagedPostgres.IsTransientConnectionFault could not be reused, and the reason matters for anyone tempted to consolidate them. Its rule — a PostgresException means the server replied, so never transient — is correct at its own site and wrong here, because two of the three transient cases arrive as exactly that: SIGKILL gives the in-flight session 57P01, and every connect during the crash recovery that follows gets 57P03 (74 consecutive measured over 0.64s). It is also [SupportedOSPlatform("windows")] while this path runs on Linux.

The classifier is an allowlist: anything unrecognised keeps the old behaviour byte for byte. 57014 and 57P04 are deliberately terminal despite sharing class 57 with the three retryable states, so it discriminates state by state and never on the two-character class.

What is deliberately not closed here: the operator-visibility half — every health surface reads the store, so none can report that the store is the problem, and /api/ping answers "ok" regardless. That is #2953.