Skip to content

Darling startup: a store briefly at its connection limit stops collection for good, and a stopped container still looks healthy #4733

Closed

Description

@erikdarlingdata

Two gaps in how Darling handles a failure at startup. Both change a documented choice, so each one states the proposed rule. Line numbers are as of 1248a67.

1. A store at its connection limit when the service starts is treated as a permanent failure

StartupFailureTriage.cs:257-277. IsRetryable is an allowlist without 53300 (too many connections). A bring-your-own or compose store can be briefly full at start. That happens after a restart storm, while the previous instance's connections are still closing, or when another application holds them. The service then stops: the process stays up but never collects until someone restarts it, which is the state #4509 set out to remove. Managed and compose stores rarely reach the limit, but a shared bring-your-own store can.

Treating class 53 as terminal is a deliberate, tested rule (StartupFailureTriage.cs:112-116, pinned at StartupFailureTriageTests.cs:118-121).

Proposed: make 53300 alone retryable, because it clears by itself. Keep 53100, 53200 and 53400 terminal, because they need an operator. Update the pin, and explain the split in the comment. The retry interval stays at the existing 60 seconds.

Test that must fail first: a 53300 at start is retried, and a 53100 still stops.

2. In a container, a terminal startup failure leaves a running container that never collects

Program.cs:503-507 keeps the process up after a terminal verdict, which is the documented choice at StartupFailureTriage.cs:225-229. Only the store compose service has a healthcheck, so a compose deployment shows the darling container as up.

Proposed: a healthcheck on the darling compose service that fails in the terminal stopped state. Exiting with a non-zero code is not the answer: under restart: unless-stopped, a plain config error then restarts the container in a loop.

Test: the health endpoint or probe reports unhealthy after a terminal verdict, and healthy otherwise.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions