Stuck-compression detector: -infinity while Running is the mid-run marker, not death

[erikdarlingdata]

Root-causes today's repeat CI flake (two hits in ~90 minutes on unrelated PRs #2124/#2128, both at CompressionJobSelfHeal_DetectionQueryValid_AndRearmSucceeds leg 3) — and the field's transient stuck→self-healed alert noise turns out to be the same product bug.

The engine fact, measured live

On TimescaleDB 2.x (pg17, tight-polled through a triggered run):

1|Scheduled|real
25|Running|NEG-INF     ← healthy job, mid-run
40|Scheduled|real

From scheduler pickup to run completion, job_stats.next_start reads -infinity with job_status = 'Running' — the engine computes the real next start only at completion. So -infinity alone does not mean "the scheduler will never run it again".

What that meant

• Product: IsCompressionJobStuck's first arm fired on -infinity unconditionally, so the self-alert check flagged any healthy compression run it happened to sample, alerted it as stuck, and "self-healed" it with a pointless re-arm. The stuck→self-healed alert pairs the box protocol has been excusing as "detection working" were largely this false positive.
• CI: the live test's leg 3 re-arms with next_start => now() (making the job immediately due) and then asserted on a single snapshot — racing the very run the re-arm triggered. Different sibling-fixture jobs appear in each failure's collection because the fixture's policies run their first check immediately on creation.

The fix

• A Running job's -infinity defers to the elapsed-bound arm — StuckRunningBound is what actually distinguishes a hung run from a healthy one. -infinity on a non-running job (the field's dead-scheduler case) still alerts exactly as before, and a hung run carrying the mid-run marker still trips the bound.
• New pure pins: mid-run marker → not stuck; hung-run-with-marker → stuck via the Running arm.
• Live test leg 3 now settles via the existing WaitUntilDetectorReportsHealthyAsync (the same helper leg 1 already uses) instead of racing its own triggered run.

🤖 Generated with Claude Code

[claude]

Review

Went through the diff (CHANGELOG.md, Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs, Darling/Darling.Tests/TimescaleSupportTests.cs) against the PR's own claim: job_stats.next_start reads -infinity while job_status = 'Running' for the duration of a run, not just on scheduler abandonment.

Correctness — IsCompressionJobStuck traced through cleanly:

• Arm 1 (nextStartIsNegativeInfinity && !isRunning) now only fires for the dead-scheduler case; a Running job with the mid-run -infinity marker correctly falls through.
• Arm 2 (unchanged elapsed-bound logic, just reusing the new isRunning local) still catches a hung run that's carrying the marker — verified against the new NegativeInfinityWhileRunning_IsTheMidRunMarker_NotStuck test, which pins both the "3 minutes in, healthy" and "30 hours in, still stuck" cases in one method.
• The existing DateTime.MinValue/never-ran guard on lastRunStartedAtUtc (from Flaky: CompressionJobSelfHeal detection test false-fails on the nightly despite its settle-wait #1760) is untouched, so a Running job whose last_run_started_at is still the NULLIF'd sentinel correctly doesn't spuriously trip the elapsed-bound arm either — no new gap introduced there.
• Live test leg 3 now settles via WaitUntilDetectorReportsHealthyAsync, which polls the actual ReadStuckCompressionJobsAsync predicate (the thing under test) rather than a hand-rolled next_start <> -infinity check that only covered one of the two arms — this closes the exact race described in the PR body (asserting on a single snapshot immediately after alter_job(next_start => now()), while the scheduler's own pickup was still in flight).

Parity — no Lite counterpart exists for this detector (IsCompressionJobStuck/StuckCompressionJobsSql/TimescaleDB support are Darling-only; grepped Lite/ and Lite.Tests/ for the symbol names, no hits), so there's nothing to drift out of sync here.

Security/perf — no new I/O, no string-built SQL (the StuckCompressionJobsSql const is unchanged; the live test's alter_job call is parameterized), and the added check is a single string comparison — no measurable perf impact.

One very minor nit, not blocking: the new CHANGELOG entry is the only "Fixed" bullet in the Unreleased section without a ([#NNNN], ...) issue reference — understandable since this was caught via CI flake investigation rather than a filed report, but flagging in case there's a tracking issue worth linking.

No correctness, parity, or security issues found.