Darling keeps retrying a slow managed PostgreSQL start instead of never collecting (#4508)

[erikdarlingdata]

Refs #4508

Follows #4509, which gave the store connect/migrate step a sustained retry. This PR gives the managed PostgreSQL start and config load the same retry, and fixes what /api/ping reports while it runs.

Why

The managed PostgreSQL start loop and config load had the same shape #4509 fixed for the store:

• two minutes of fast retries on a failure the classifier calls retryable;
• then a fall-through to the terminal catch that a non-retryable failure takes.

A bundled PostgreSQL that takes longer than two minutes to start (a slow disk, a long crash recovery) therefore left the service running for the rest of its life without collecting.

What changes

• The managed PostgreSQL start (DarlingWorker.cs) gets the same sustained arm as the store:
  • once the fast budget is spent on a still-retryable failure, it logs CRITICAL once, then a WARNING on each retry;
  • it publishes Retrying, never Stopped;
  • it retries every SustainedRetryDelay (60 s), through StartupFailureTriage.NextAction.
• Config load gets the identical arm, only for failures it already classified as retryable. No new failure class becomes retryable, and a non-retryable failure still stops.
• The sustained flag: CollectorRuntimeState.Snapshot gains Sustained, and PublishRetrying takes a sustained flag.
• The status says it's a sustained retry. Before, it reported an attempt number past the fast budget's cap ("attempt 30 of 25"). Now, once the retry is sustained:
  • GET /api/ping drops attempts and adds "sustained": true and "retryEverySeconds" (taken from SustainedRetryDelay, 60 today). It keeps attempt, step, detail and the 503 degraded status.
  • The step detail ends with "retrying every 60s, attempt N (past the 120s fast budget)", built from SustainedRetryDelay and RetryBudget by one helper, so /api/ping, MCP and the Viewer show the same text.
  • A fast retry is unchanged: no sustained or retryEverySeconds key, and attempts still shows the cap.

Test plan

• The code-reading checks (StartupFailureTriageTests) cover all three sites, each anchored on that site's own NextAction(attempt, <site budget>.Elapsed, ex) line:
  • CRITICAL once, then a WARNING per retry;
  • publishes Retrying, never Stopped;
  • the store site's two existing checks now use the same per-site anchor, because all three sites share one catch filter.
• DescribePing_OmitsAttemptsOnceSustained_ButKeepsItWhileFast (CollectorRuntimeStateTests):
  • a sustained snapshot gives 503, degraded, attempt 30, no attempts key, "sustained":true, "retryEverySeconds":60, and the detail phrase;
  • a fast snapshot keeps "attempts":25 and has neither new key nor the phrase;
  • PublishRetrying_Sustained_AppendsRetryPhraseToDetail pins the detail phrase at the publish site;
  • RED at runtime before the ping change: Assert.Null() Failure … Expected: null / Actual: 0.
• Mutations, each reverted:
  • removing the managed PostgreSQL start's sustained arm fails its two new checks;
  • reverting the ping change fails the ping test.
• The run: StartupFailureTriageTests + CollectorRuntimeStateTests + DocCommentHygieneTests: Total: 163, Failed: 0.
• Windows-only: Lite.Tests isn't touched.

CHANGELOG

SECTION: Fixed
ENTRY: - A managed PostgreSQL or a config load that takes more than two minutes to start no longer leaves Darling running without collecting ([#4538]) - The sustained start-up retry from [#4509] now also covers the bundled PostgreSQL's own start and, for failures already treated as transient, loading darling.json. While that retry runs, /api/ping and the status detail say it is a sustained retry, with the interval, instead of reporting an attempt count past the cap.
REF: [#4538]: #4538
REF: [#4509]: #4509