Darling service robustness: stuck-compression self-heal + single-instance / arg guards (#1581)

[erikdarlingdata]

Darling service robustness — the two durable #1581 product fixes

Closes the durable product fixes from the #1581 field incident (a prod box climbed toward OOM over ~30h with Windows error 487, "could not reserve shared memory region"). The confirmed root cause was three TimescaleDB compression-policy jobs that had silently died (next_start = -infinity, so the scheduler never re-fired them) on the largest hypertables — compression halted, uncompressed data grew without bound, autovacuum churned the swelling TOAST, the disk bled, and worker-spawns hit the 487 wall. A separate self-inflicted event during triage — a stray Service.exe --version that fell through into a real startup and spawned a second service instance fighting the first over the bundled PostgreSQL shared-memory segment — caused a ~7.5h outage. This PR makes both failure classes impossible to hit silently.

Fix A — stuck-compression-job self-heal

• Hourly, TimescaleDB-only, fleet-level check in the worker sweep (reuses the disk-check cadence idiom; gated on _timescaleAvailable).
• Reads every compression-policy job from timescaledb_information.job_stats JOIN jobs and flags one stuck when next_start = -infinity (compared in SQL, so it never depends on Npgsql's infinity conversion — the exact incident) OR it has been Running past max(2× schedule interval, 2h floor). Non-compression job types are never touched.
• Re-arms a stuck job once via a parameterized alter_job($1, next_start => now()) and fires a CRITICAL alert through the same deliverer/history/cooldown the store-disk-pressure self-alert already uses (real email/webhook + audit row). If it re-hangs after the self-heal, or the re-arm itself fails (e.g. the store login doesn't own the job), the service escalates and stops re-arming (looping alter_job on a re-hanging job is a product-bug signal for a human), and writes a resolution row on recovery.
• Failure-isolated at three layers (worker connection-open, evaluator wrapper, the read) so a store hiccup can never abort the collection sweep; a clean no-op on plain-PostgreSQL stores.

Fix B — single-instance guard + argument robustness

• The exe classifies its first argument before building the host: --version/--help print and exit 0; any unknown option prints usage to stderr and exits non-zero without starting the host (the --version that triggered the outage now prints and exits). Recognized one-shot verbs and the no-arg SCM start are unchanged; the known-verb allowlist is the single source Program's dispatch mirrors, so the two can't drift.
• A Global\PerformanceMonitorDarlingService named single-instance mutex is acquired between host build and run (Build doesn't touch Postgres; Run does): a second invocation that reaches startup logs a CRITICAL and exits without starting, so it can never fight the first over PostgreSQL or the MCP/web ports. An abandoned mutex (previous instance crashed) is acquired with a warning; the owner releases it on graceful shutdown so an SCM stop→start re-acquires cleanly. Windows-gated.

Darling service only. No store schema change.

Testing

Independently reviewed line-by-line and re-verified: Release build clean, full Darling.Tests green — 2723 passed / 0 failed / 142 gated-live skipped. Coverage: the pure stuck-detection predicate (all branches), the self-heal state machine (re-arm-once, escalate, re-hang, re-arm-failure, recovery+re-stuck, multi-job, both throwing seams isolated), the argument-classification decision table, and the mutex behaviors (fresh / held / abandoned / SCM restart), plus a gated-live test that forces next_start = -infinity on a real compression job and proves detection + re-arm heals it.

One review fix applied on top of the build (test-only): the abandoned-mutex test polled the production non-blocking WaitOne(0) immediately after Thread.Join, racing the kernel's post-join native-thread teardown that marks the mutex abandoned — a window that exists only for the in-process crash simulation, never in production. The retry now lives in the test (the guard must not retry — a false there means a live second instance holds the lock); confirmed 10/10 green in a determinism loop. Production guard code unchanged.

🤖 Generated with Claude Code