Stuck-compression detector: -infinity while Running is the mid-run marker, not death

CHANGELOG.md

@@ -24,6 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Upgrading a 3.3.0-era Darling store to 3.4.0 no longer fails the migration ladder** ([#2119], field report by ghauan on #2105) - migration rung 51 is assembled at runtime from the live view generator, which since #2069 emits the `query_plan_gz` column that rung 54 adds - so a store at or below V50 replayed rung 51 referencing a column three rungs before it existed, the ladder halted with `42703`, and the service stopped. Nothing was harmed (each rung runs in its own transaction) but the upgrade could not complete. Rung 51 now pre-adds the column with V54's own idempotent ALTERs (rung 54's copy no-ops after it), the retry just works on an already-failed store, and new ladder pins make the generated-rung replay hazard - a later column teaching a generator new SQL that an earlier rung re-emits - fail in CI instead of on an operator's store.
 - **The upgrade path itself is now release-gated** ([#2119] follow-up) - the 3.4.0 ladder failure escaped because every pre-release check ran either a fresh store (full generator, no ladder) or the dogfood box (which walks each rung in the era it ships); no check pointed new binaries at a store a RELEASED build had made, the one path that replays old rungs and the one path users take. Two gates now: a committed fixture of the previous release's ladder exactly as that release resolved it (generators frozen at the tag), which `MigrationUpgradeLadderLiveTests` builds on scratch Postgres and climbs with the current ladder on every CI run - verified two-sided at birth (fails with the exact field 42703 on the pre-fix build, climbs V39->V54 clean on the fixed one, including from a mid-failure retry) - and a release-cut step that boots the previous release's container image and then the candidate on the SAME volume (docs/releasing.md).
 - **Store Disk Pressure no longer re-notifies every cooldown while free space sits at an unchanged level** ([#2101], reported with the diagnosis by gotqn) - the self-alert re-fired on the plain cooldown, so a store volume parked at 7.3% free produced an identical CRITICAL notification every ~15 minutes for as long as the condition stood - one static condition, dozens of notifications. It now runs behind the same worsening gate the target-server volume alert has had since the #754 follow-up: fire once on entry, re-fire only when free% drops at least a full point below the last-alerted level (still cooldown-limited), one resolution row on recovery - which also clears the watermark, so a volume that oscillates around the threshold cannot go permanently silent. Deliberately NOT applied to the state-only self-alerts (Collection Stopped, Agent Not Running, Capture Down): those have no level to worsen, and their per-cooldown "still broken" reminder is wanted. The reporter's second ask - exposing the hardcoded self-alert thresholds as store-backed settings - is real but is its own design decision (control-plane knobs, Viewer rows, MCP settings groups) and stays tracked on the issue.
+- **Compression Job Stuck no longer false-alarms on a job it caught mid-run** - measured live on TimescaleDB 2.x: from the moment the scheduler picks up a due job until its run completes, `job_stats.next_start` reads `-infinity` with `job_status = 'Running'`, and the real next start is only computed at completion. The detector's first arm treated `-infinity` unconditionally as "the scheduler will never run it again", so any healthy compression run the self-alert check happened to sample got flagged as stuck, alerted, and "self-healed" with a pointless re-arm - the transient stuck-then-self-healed alert pairs the field has been shrugging off were this false positive, and the stuck-detector live test's CI flake was the same race (it re-arms with `next_start => now()` and then read a single snapshot while the run it had just triggered was still executing). A RUNNING job's `-infinity` now defers to the elapsed-bound arm, which is what actually distinguishes a hung run from a healthy one - a genuinely dead job (`-infinity`, not running) still alerts exactly as before, and a hung run still trips the bound.
 
 ## [3.4.0] - 2026-08-06
 

Darling/Darling.Tests/TimescaleSupportTests.cs

@@ -152,14 +152,33 @@ managed store compact (#1458). */
     [Fact]
     public void IsCompressionJobStuck_NextStartNegativeInfinity_IsStuck()
     {
-        /* The dominant failure mode: next_start = -infinity, so the scheduler never re-fires it. Stuck
-           regardless of status/last-run — it will never run again. */
+        /* The dominant failure mode: next_start = -infinity on a job that is NOT running — the scheduler
+           abandoned it and never re-fires it. */
         Assert.True(TimescaleSupport.IsCompressionJobStuck(
             nextStartIsNegativeInfinity: true, jobStatus: "Scheduled", lastRunStartedAtUtc: null,
             scheduleInterval: TimeSpan.FromHours(12), nowUtc: s_now, out var reason));
         Assert.Contains("-infinity", reason, StringComparison.Ordinal);
     }
 
+    [Fact]
+    public void IsCompressionJobStuck_NegativeInfinityWhileRunning_IsTheMidRunMarker_NotStuck()
+    {
+        /* Measured live on TimescaleDB 2.x: from scheduler pickup to run completion, next_start reads
+           -infinity WITH job_status = 'Running' — the engine only computes the real next start when the
+           run finishes. An unconditioned -infinity arm flagged every healthy job caught mid-run (the
+           field's transient stuck→self-healed alert noise, and the CI flake where the live test caught
+           its own re-arm-triggered run). Mid-run belongs to the elapsed-bound arm: */
+        Assert.False(TimescaleSupport.IsCompressionJobStuck(
+            nextStartIsNegativeInfinity: true, jobStatus: "Running", lastRunStartedAtUtc: s_now.AddMinutes(-3),
+            scheduleInterval: TimeSpan.FromHours(12), nowUtc: s_now, out _));
+
+        /* ...which still catches a genuinely HUNG run that carries the mid-run marker. */
+        Assert.True(TimescaleSupport.IsCompressionJobStuck(
+            nextStartIsNegativeInfinity: true, jobStatus: "Running", lastRunStartedAtUtc: s_now.AddHours(-30),
+            scheduleInterval: TimeSpan.FromHours(12), nowUtc: s_now, out var reason));
+        Assert.Contains("Running", reason, StringComparison.Ordinal);
+    }
+
     [Fact]
     public void IsCompressionJobStuck_HealthyScheduled_IsNotStuck()
     {
@@ -1553,9 +1572,12 @@ this job and judged it healthy. */
            DETECTION logic is covered by the pure IsCompressionJobStuck unit tests. */
         Assert.True(await TimescaleSupport.TryRearmJobAsync(connection, jobId, null, ct));
 
-        /* (3) After a real re-arm (next_start => now()) the job is scheduled/running within bound, not stuck. */
-        var afterRearm = await TimescaleSupport.ReadStuckCompressionJobsAsync(connection, DateTime.UtcNow, null, ct);
-        Assert.DoesNotContain(afterRearm, s => s.JobId == jobId);
+        /* (3) After a real re-arm (next_start => now()) the job settles healthy. SETTLES, not "reads healthy
+           on one snapshot": next_start => now() makes the job immediately due, the scheduler picks it up, and
+           from pickup to completion job_stats reads next_start = -infinity with status Running — the mid-run
+           marker (measured live; the detector now defers that state to its elapsed-bound arm). A single
+           un-settled read raced the very run the re-arm triggered, which was this test's own flake. */
+        await WaitUntilDetectorReportsHealthyAsync(connection, jobId, ct);
     }
 
     /// <summary>

Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs

@@ -2800,14 +2800,25 @@ public static TimeSpan StuckRunningBound(TimeSpan? scheduleInterval)
     /// <summary>
     /// The pure stuck-compression-job decision (#1581). A compression policy job is STUCK when either:
     /// <list type="bullet">
-    /// <item>its <c>next_start</c> is <c>-infinity</c> — the scheduler will NEVER re-fire it (the dead-job
-    /// bug that let uncompressed data grow without bound until the disk filled), or</item>
+    /// <item>its <c>next_start</c> is <c>-infinity</c> while the job is NOT currently running — the scheduler
+    /// abandoned it and will NEVER re-fire it (the dead-job bug that let uncompressed data grow without bound
+    /// until the disk filled), or</item>
     /// <item>it has been in the <c>Running</c> state since a <c>last_run_started_at</c> older than
     /// <see cref="StuckRunningBound"/> — a run that began long ago and never finished (a hung run).</item>
     /// </list>
     /// A job with neither condition is healthy and is NOT flagged. No I/O, so it pins directly with a
     /// controllable clock. Scoping to compression jobs happens in the query — this decides only "stuck".
     ///
+    /// <para><b><c>-infinity</c> is ALSO the engine's mid-run marker</b>, measured live on TimescaleDB
+    /// 2.x (pg17): from the moment the scheduler picks up a due job until its run completes,
+    /// <c>job_stats.next_start</c> reads <c>-infinity</c> with <c>job_status = 'Running'</c>, and the real
+    /// next start is only computed at completion. So <c>-infinity</c> alone cannot mean "dead" — an
+    /// unconditioned first arm flagged every healthy job the check happened to catch mid-run, alerted it as
+    /// stuck, and "self-healed" it with a pointless re-arm (the field's transient stuck→self-healed noise;
+    /// the CI flake was the live test catching its own re-arm-triggered run). A running job is therefore
+    /// left to the second arm, whose elapsed bound is what actually distinguishes a hung run from a
+    /// healthy one.</para>
+    ///
     /// <para>A <paramref name="lastRunStartedAtUtc"/> of <see cref="DateTime.MinValue"/> counts as NEVER RAN,
     /// not as "started in year 1" (#1760). <see cref="StuckCompressionJobsSql"/> already NULLIFs TimescaleDB's
     /// <c>-infinity</c> never-ran sentinel, so this is the second line of defence: the sentinel maps to
@@ -2822,13 +2833,15 @@ public static bool IsCompressionJobStuck(
         DateTime nowUtc,
         out string reason)
     {
-        if (nextStartIsNegativeInfinity)
+        var isRunning = string.Equals(jobStatus, "Running", StringComparison.OrdinalIgnoreCase);
+
+        if (nextStartIsNegativeInfinity && !isRunning)
         {
             reason = "next_start is -infinity — the scheduler will never run it again";
             return true;
         }
 
-        if (string.Equals(jobStatus, "Running", StringComparison.OrdinalIgnoreCase)
+        if (isRunning
             && lastRunStartedAtUtc is DateTime startedUtc
             && startedUtc != DateTime.MinValue)
         {