Skip to content

Darling worker: no signal when the fleet gate skips slots, and Extended Events sessions left on removed servers #4732

Closed

Description

@erikdarlingdata

Four smaller gaps in DarlingWorker, two of them optional. Line numbers are as of 1248a67. Related: #4710.

1. No signal when the fleet gate cannot keep the cadence

DarlingWorker.cs:483,491,2660,9617 and CollectorCadence.cs:30-31. Missed slots are skipped with one Info log line. There is no skipped-slot count, no queue-wait measure and no gate self-alert, so an operator cannot see it happen. Capacity at hundreds of servers is worked out by arithmetic, not measured.

Expected: a queue-wait or slots-skipped measure, plus a self-alert past a threshold. Measure with a synthetic fleet before changing the gate width. Do not scale the gate with the core count: its ceiling of 16 is tied to the store's 24-connection pool.

Test: a saturated gate records skipped slots and raises the self-alert.

2. Removing a server leaves its Extended Events sessions running on the target

DarlingWorker.cs:4586-4608,3500 and DarlingXeSessions.cs:294,383,480. The cost is bounded: each ring buffer is 4 MB.

Expected: an explicit command-line cleanup verb. The service does not drop the sessions itself when a server is removed. The session names are shared with Lite and with any other Darling service, and an unreachable server cannot be cleaned anyway.

Test: the verb's argument handling and exit codes, using the existing DarlingToolExitCode values.

3. Scheduling uses the wall clock (optional)

DarlingWorker.cs:9545,9609,3465,9291. A backward clock step pauses collection for as long as the step. Automatic clock steps are only seconds long. If this is fixed, clamp a due time that is more than one interval ahead back to now, as RecomputeNextDueAsync does. A monotonic scheduler is too big a change for this.

4. After a server's definition is edited, the run in flight finishes on the old connection string (optional)

DarlingWorker.cs:4622,9529. The cost is one run's rows and one re-baseline. If this is fixed, discard the runtime after the connect when the definitions no longer match.

Activity

  1. added 20 commits that reference this issue on Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions