Darling keeps retrying the store at start instead of never collecting after the retry budget (#4508)

[erikdarlingdata]

Fixes #4508

Why

DarlingWorker.RunCollectionLoopAsync's store connect + PgMigrations.MigrateAsync retry loop classifies each failure with StartupFailureTriage.IsRetryable. A failure that classifier accepts is retried every 5 seconds for up to 25 attempts or 120 seconds, whichever comes first. Once that budget runs out, the old code fell through to the loop's bare terminal catch — the same one a non-retryable failure hits — logged one CRITICAL line, published Stopped, and returned. That return completes the worker task successfully, so the web and MCP hosts stay up and the process keeps reporting healthy while collection never starts again for the life of the process. A store that is down for longer than two minutes (a longer failover, a slow crash recovery) and a migration rung that can never apply ended up producing the identical outcome, even though the whole point of the classifier is to tell those two cases apart.

What changes

• StartupFailureTriage gets a new pure decision method, NextAction(attempt, elapsed, exception), returning one of three outcomes (RetryFast, RetrySustained, Stop) plus the delay to use. This is the retry decision, pulled out of the loop into a seam that can be exercised without a running store, a config file, or a migration.
• A new constant, StartupFailureTriage.SustainedRetryDelay (60 seconds), for the slower, unbounded retry once the fast budget is spent.
• The store connect/migrate loop in DarlingWorker.cs gets a third catch arm, between the existing fast-retry arm and the terminal arm: when the failure is still one IsRetryable accepts but the fast budget/attempt cap has been spent, it logs a CRITICAL line once (the first time the fast budget runs out), then a WARNING on every later attempt, keeps publishing CollectorRuntimeState.CollectorPhase.Retrying (never Stopped), and retries forever on SustainedRetryDelay.
• A non-retryable failure is unaffected: it still hits the same terminal catch, at any attempt, with the same CRITICAL line, PublishStopped, and return.
• Cancellation (stoppingToken) still ends the loop promptly — the new arm's filter, like the existing ones, excludes OperationCanceledException.

The other two triaged start-up sites (config load, managed-PostgreSQL bootstrap) have the same fast-budget-then-stop shape. This PR changes only the store connect/migrate loop that #4508 names.

Test plan

Build: dotnet build Darling/Darling.Tests/Darling.Tests.csproj -c Release -p:EnableWindowsTargeting=true — 0 warnings, 0 errors. Same for Lite.Tests.csproj (build-only; targets net10.0-windows, cannot run on macOS — CI decides it; no code in that project touches this change).

Ran Darling.Tests.dll in-process on macOS (Microsoft.WindowsDesktop.App stripped from the runtimeconfig, per the repo's macOS test recipe). No store rig needed — every new pin is pure or a source-census pin.

• Darling.Tests.StartupFailureTriageTests: 66 passed, 0 failed (was 58 before this PR's new pins).
• Darling.Tests.CollectorRuntimeStateTests: 15 passed, 0 failed.
• Darling.Tests.DocCommentHygieneTests: 77 passed, 0 failed (new doc-commented members).
• Darling.Tests.RepoFileResolutionEquivalenceTests, RepoFileAdoptionTests, CommentFilterAdoptionTests: all green, unaffected.

RED on dev (compile-level: the members are new): copied the new test file into a clean detached worktree at dev's tip (origin/dev, no code changes) and built it there. It fails to compile — StartupFailureTriage.NextAction, StartupRetryDecision, and StartupFailureTriage.SustainedRetryDelay don't exist on dev at all (11 CS0117/CS0103 errors). That is expected: the whole fix is a new seam dev doesn't have.

Mutation: with the fix built, changed NextAction's final line from return new NextStartupAction(StartupRetryDecision.RetrySustained, SustainedRetryDelay); to return new NextStartupAction(StartupRetryDecision.Stop, TimeSpan.Zero);, rebuilt, and reran StartupFailureTriageTests: NextAction_BudgetSpentButStillRetryable_IsRetrySustained failed (65 passed, 1 failed), everything else stayed green. Reverted the line, rebuilt, reran: 66/66 green again — full clean-tree build after the revert is also 0 warnings / 0 errors, confirming nothing was left mutated.

New pins added (all in Darling.Tests.StartupFailureTriageTests unless noted):

• (a) NextAction_BudgetSpentButStillRetryable_IsRetrySustained — retryable failure past either cap → RetrySustained at SustainedRetryDelay. RED on dev (compile failure, see above); this is the mutation target.
• (b) NextAction_NonRetryableFailure_IsAlwaysStop — non-retryable failure at attempt 1 and at the caps → always Stop, zero delay.
• (c) NextAction_WithinBudget_IsRetryFast — retryable failure inside the fast budget → RetryFast at RetryDelay.
• TheStoreSustainedRetryArm_LogsCriticalOnceThenWarningPerRetry — source-census pin: the sustained arm's flag is set exactly once and only in the critical branch, both a critical and a warning call site exist, and the arm calls NextAction through the seam.
• TheStoreSustainedRetryArm_PublishesRetryingNotStopped — the sustained arm publishes Retrying, never Stopped, and never returns.
• CollectorRuntimeStateTests.EveryRetryPublish_ReportsTheTriagesOwnAttemptCap updated: now expects 4 PublishRetrying call sites (three startup steps plus the store site's new sustained arm), all still reporting StartupFailureTriage.Attempts.
• TheThreeSitesHaveDistinctBudgetVariables updated: now expects 4 IsRetryable(ex) occurrences (was 3) and still exactly 3 stopwatches — the sustained arm reuses storeRetryBudget rather than starting a fourth.

CHANGELOG

SECTION: Fixed
ENTRY:

• After a long store outage at start, Darling now keeps retrying and starts collecting when the store is back ([Darling keeps retrying the store at start instead of never collecting after the retry budget (#4508) #4509]) - Once the two-minute startup retry budget for a recoverable store connection or migration failure ran out, Darling stood down and never collected again until the service was restarted, even though the service itself kept running. It now keeps retrying, more slowly, until the store is reachable again.
  REF:
  [Darling keeps retrying the store at start instead of never collecting after the retry budget (#4508) #4509]: Darling keeps retrying the store at start instead of never collecting after the retry budget (#4508) #4509