Stop the compression detector reading TimescaleDB's never-ran sentinel as year 1 (#1760)

[erikdarlingdata]

Closes #1760.

The flake was a production defect, not a slow runner. ReadStuckCompressionJobsAsync is the compression self-heal's detector (#1585/#1586), and it flagged healthy jobs as stuck for the whole of their first run.

Root cause, measured

Both facts below were measured against a real PostgreSQL + TimescaleDB 2.28.1 cluster, not inferred.

1. last_run_started_at is -infinity, not NULL, for a job that has never run. Freshly added policy has no bgw_job_stat row at all (every column NULL); it is alter_job that materialises the row carrying the sentinel:

 job_id | next_start_neg_inf |          next_start          | job_status | last_run_started_at | lrs_neg_inf | lrs_is_null
--------+--------------------+------------------------------+------------+---------------------+-------------+-------------
   1000 | f                  | 2026-07-28 00:21:41.86261-04 | Scheduled  | -infinity           | t           | f

Npgsql maps -infinity to DateTime.MinValue, so IsCompressionJobStuck computed nowUtc - DateTime.MinValue — roughly 739,000 days — which clears every StuckRunningBound (floor 2h). The job was reported "stuck in the Running state for 388,000,000 minutes".

2. Why that was reachable at all. job_status and last_run_started_at come from independent sources in TimescaleDB's own view definition:

CASE WHEN pgs.state = 'active' THEN 'Running' ... END AS job_status
...
LEFT JOIN pg_stat_activity pgs ON pgs.datname = current_database()
                              AND pgs.application_name = j.application_name

job_status is derived from pg_stat_activity; last_run_started_at is bgw_job_stat.last_start. A job's FIRST run therefore reads Running while its start time is still the sentinel. The window is structural, not hypothetical — and every job on a fresh CI cluster is in exactly that state.

ApplyCompressionPolicyAsync creates a policy for all 26 collector tables at once and the test seeds a 40-day-old wait_stats row, so several jobs are starting real compression work around the moment the detector reads. That is the interleaving the nightly hit.

Why the previous hardening could not have worked

The issue asked to distinguish two candidates. Both are settled by evidence rather than argument.

Candidate 2 (the wait times out and returns quietly) is refuted outright. The wait does not return quietly on timeout — TimescaleSupportTests.cs:645 asserts loudly with a distinctive "never settled to a real next_start" message. The observed nightly failure was Assert.DoesNotContain() Failure, a different assertion entirely.

Candidate 1 is confirmed, and worse than described. The wait polled next_start <> '-infinity' — one of the two arms the detector evaluates — so "settled" and "the assertion will pass" were different statements. But the value it polled was the one the caller's own alter_job(next_start => now() + interval '1 hour') had written one statement earlier. Instrumented (M3 below), the original wait returned after POLLS=1 DELAYS=0. It never waited at all, and the 30s deadline was never approached — which is precisely why no timeout increase could ever have helped.

The fix

Production. StuckCompressionJobsSql becomes a public const (the existing RearmJobSql idiom in the same file) and NULLIFs the sentinel, so both -infinity tests run in SQL rather than through Npgsql's infinity mapping — which is the discipline that comment already claimed for next_start. IsCompressionJobStuck additionally rejects DateTime.MinValue, so a future caller reading the column un-guarded cannot resurrect the false positive.

Test. The settle-wait now polls ReadStuckCompressionJobsAsync itself. That is predicate identity by construction — one definition, not two copies that drift — so settled-according-to-the-wait is settled-according-to-the-assertion. It reports the detector's own Reason on timeout instead of assuming next_start. No retry wrapper, no timeout increase.

Leg (1) also gains the assertion it always claimed to make. ReadStuckCompressionJobsAsync is failure-isolated: a broken query is swallowed and returns an EMPTY list, so DoesNotContain alone passed just as happily against SQL that never compiled — the one thing that leg exists to prove. It now runs the production const directly, where a syntax or column error throws, and requires the job to be present in the result.

Mutation table

Each row was applied, built, and watched red.

#	Mutation	Result	Test that went red
M1	Remove NULLIF from StuckCompressionJobsSql	RED (2)	StuckCompressionJobsSql_GuardsTheNeverRanSentinel, StuckCompressionJobsSql_NeverRunJob_ReadsNullLastRunStartedAt_AgainstDevPostgres
M2	Remove the DateTime.MinValue guard from IsCompressionJobStuck	RED (1)	IsCompressionJobStuck_RunningWithNeverRanSentinel_IsNotStuck
M3	Restore the original next_start-only wait, instrumented	RED, reporting POLLS=1 DELAYS=0	measurement probe, removed before commit

M1 is the load-bearing one: the live test went red, which proves the sentinel really is -infinity in the catalog. Had it been NULL, removing the NULLIF would have changed nothing and the test would have stayed green — so the guard is demonstrably not dead code.

M3 is the flake's mechanism made measurable rather than argued.

Verification

• Darling.Tests full suite against a live PostgreSQL 18 + TimescaleDB 2.28.1 rig: 3651 passed, 0 failed, 7 skipped.
• TimescaleSupportTests alone: 19/19, including the 3 new tests (2 pure, 1 gated live).
• Build: dotnet build Darling/Darling.Tests/Darling.Tests.csproj -c Debug -t:Rebuild — 0 Warning(s), 0 Error(s).

The new live test is deterministic rather than timing-dependent: arming the job an hour out both materialises the stat row carrying the sentinel and guarantees the scheduler cannot run the job out from under the assertion. It asserts the raw column really does carry -infinity (so the guard cannot pass for the wrong reason) and that the production query hands back NULL.

Field relevance

Worth flagging for the release: on any store where a compression job is observed during its first run, the self-heal was re-arming a healthy job via alter_job(next_start => now()). That is not data loss and it is self-limiting — the sentinel clears once the first run finishes — but it means "compression job stuck" alerts on a newly provisioned or freshly upgraded store may have been false, and the recorded reason ("stuck in the Running state for 388,000,000 minutes") is the signature to look for in field logs.

🤖 Generated with Claude Code