Repository navigation
A retryable store failure that outlasts the 120-second start budget leaves the service running without collecting #4508
Description
Claude posting for Erik Darling
If the store stays unreachable, or a migration keeps hitting a retryable error, for longer than the start-up retry budget (120 seconds), the Darling service keeps running but never collects, until someone restarts it.
Seen
In DarlingWorker.RunCollectionLoopAsync (by reading the code):
- The retry: the store connection and
PgMigrations.MigrateAsyncretry on retryable failures (StartupFailureTriage.IsRetryable: connection errors, lock timeouts55P03,55006, and so on), up toStartupFailureTriage.Attempts(25) withinStartupFailureTriage.RetryBudget(120 s). - When that runs out: the terminal catch logs a critical line ("collection does not start"), publishes the collector state as stopped, and returns.
- What's left: the web and MCP hosts keep running, so the process doesn't exit and no service manager restarts it. Collection doesn't resume until a person restarts the service.
- Things that can outlast 120 s: a store that's restarting or failing over, or a migration waiting on a lock held by a long TimescaleDB continuous-aggregate refresh (the hourly Query Store rollup refresh can run for minutes).
Expected
After the budget, keep retrying with a capped backoff (e.g. every 60 s), logging a critical line once and a warning on each retry, and start collection as soon as the store is reachable and migrated. The published collector state stays truthful while it waits. A non-retryable failure still stops as it does today.
Activity
Darling keeps retrying the store at start instead of never collecting…
erikdarlingdata commentedon Sep 28, 2026
Claude posting for Erik Darling.
Fixed by #4509 (0ed0c47), so this issue is closing.
After the two-minute start retry budget, a store failure that is still retryable no longer stops the service from collecting. It logs a critical line once, then a warning on each retry, reports Retrying rather than Stopped, and tries again every 60 seconds. Collection starts as soon as the store is ready. Non-retryable failures still stop as before.
A follow-up applies the same sustained retry to the managed PostgreSQL bootstrap step (and to config load where its failures are already classed as retryable), and makes the retry counter read as a sustained retry past the fast budget.
Fix the sustained-retry anchor collision and add census pins for all …
Omit attempts from /api/ping once the sustained retry has no cap (#4508)
Ping body says sustained retry, not just drops the attempt cap (#4508)
Darling keeps retrying a slow managed PostgreSQL start instead of nev…
erikdarlingdata commentedon Sep 28, 2026
Claude posting for Erik Darling.
Follow-up merged in #4538 (0524634): the managed PostgreSQL start and config load (for failures already classed as retryable) now keep retrying past the fast budget the same way the store step does. While a sustained retry runs,
/api/pingreports"sustained": trueand"retryEverySeconds", and the status detail ends with "retrying every 60s, attempt N (past the 120s fast budget)".