CI: distinct-chunk counting for job_history's two-scan floor proof (flake from #4235)

[erikdarlingdata]

Fixes the EventWindowedReadsAreBoundedLivePostgresTests flake blocking every armed PR's required check.

Census

Workflow: gh run list --workflow Build --limit 80, covering 2026-09-25 05:13Z-09:41Z (the full window CI's volume returned). Jobs per run: REST endpoint actions/runs/<id>/jobs. Failure output: gh run view --job <id> --log-failed, filtered to [FAIL] and Total: for every failed PG-tests job.

Excluded as own-PR compile breaks (build also failed). Runs: 36098645834, 36099235605, 36100129482, 36100816723, 36104266862, 36105160075, 36106541517, 36110524470, 36112451005, 36112955353, 36114913004, 36117448144, 36119112979.

Two more excluded (own bug, not flake). 36115560721: fix/4198-collection-health-default, Lite's McpToolsListBudgetTests ceiling broken by that PR's own description change. 36097633466: fix/4226-viewer-timer-reads, failed on ViewerW2aLivePostgresTests, the read that PR was changing.

Cross-cutting failures, where the change did not touch the failing area:

Run	Time (UTC)	Change	Shard	Test	Symptom
36109281530	07:45	dev push, #3953 query-store interval fix	PG tests (2)	PgTargetBlockingTests.ThirtyOneDaysOfLightBlocking...	avg_blocked_sessions InRange(2.8,3.0), actual 2.727
36100667038	05:56	PR fix/4226-viewer-timer-reads (Overview reads)	PG tests (0)	PgTargetAnomalyTests.TheAuroraWaitProfile...	current_ms_per_sec expected 3200, actual 1500
36115696476	08:56	dev push, #4259 get_query_heatmap defaults	PG tests (0)	EventWindowedReadsAreBoundedLivePostgresTests (job_history) + PgTargetAnomalyTests.TheAuroraWaitProfile... (same as above)	job_history: old=8,new=7 (needed reduction 2, got 1)
36117118128	09:12	dev push, #4228 AG reads	PG tests (0)	EventWindowedReadsAreBoundedLivePostgresTests (job_history) only	old=8,new=7, same as above

Baseline confirmed green: 36111242422 (08:07Z) and 36112268301 (08:19Z), both fully passing, both after #4235 merged (06:11Z) and before the two failing runs.

That is two flake families, not one. PgTargetAnomalyTests/PgTargetBlockingTests share a "worst-scoring tile" pattern (#3653 A8 option B), unrelated to chunk counting or job history. I filed #4274 for that one with its own evidence rather than fixing it here. Tracing PgTargetAnomalyDetector's tile selection is real investigation, not a small fix.

Root cause (confirmed with numbers, not the shared-pollution theory)

The working theory going in was shared-store pollution shifting chunk counts based on CI's shard assignment. That is not what is happening here. I reproduced the job_history failure alone, on a freshly created darlingtest, with zero other classes run first:

BuildJobHistorySql(false): expected the floor to exclude at least 2 of the 4 seeded old-day chunks (old=7, new=6)

That is the same reduction (1) the polluted CI runs showed (old=8, new=7 there). The +1 on both sides in CI is a servers-table chunk unrelated to job_history, so it nets zero effect on the difference. The real cause is structural, not pollution:

• ChunkIntervalDays = 1 (TimescaleSupport.cs). Job_history's chunks are calendar-day-aligned.
• EventWindowFloor.For sets the floor to exactly WindowStart - 1 day, which is now - 2 days. That is an exact instant, never a chunk boundary.
• #4229's own GROUP BY fix split BuildJobHistorySql's single scan into two independently floor-bounded scans, a job_stats CTE plus the base join. So the floored plan visits every in-window chunk twice.
• PlanChunkScans.Count counts scan-node lines, not physical chunks. A 48-hour lookback from an arbitrary instant almost never fits in 2 calendar days, so 3 in-window calendar-day chunks get scanned, each twice. new reads as 6 nodes. old's single scan across all 7 existing chunks reads as 7. Reduction by node count: 1. The test's own minReduction: 2 was already discounted from the default 3 for this doubling, but it is still unreachable by construction. No amount of database cleanup changes that math.

Fix

Darling/Darling.Tests/PlanChunkScans.cs: added DistinctChunkCount, counting unique _hyper_N_M_chunk names instead of scan-node occurrences. Count and its existing tests (PlanChunkScansTests.cs) are untouched, so other callers keep today's node-counting behavior.

Darling/Darling.Tests/EventWindowedReadsAreBoundedLivePostgresTests.cs: AssertChunksAsync now compares DistinctChunkCount. A chunk visited by two scan nodes over the same hypertable now counts once, which is what "the floor prunes out-of-window chunks" actually means. The other five AssertChunksAsync calls each scan two different hypertables. Their chunk names never collide, so this change does not affect them.

Measured after the fix, fresh database: old=7, new=3 distinct chunks, reduction 4 (comfortable margin over minReduction: 2, tolerant of a modest pollution shift either side).

Proof it still catches a real regression

Per the lane orders, deleted the floor from one read to prove the assertion still fails without it. Swapped the job_history call's floor parameter for a no-op (1970-01-01, changing nothing else) and reran:

BuildJobHistorySql(false): expected the floor to exclude at least 2 of the 4 seeded old-day chunks (old=7, new=7)

Correctly fails (reduction 0). Reverted before committing, so the pushed diff only touches the two files above.

Test plan

• dotnet build Darling/Darling.Tests/Darling.Tests.csproj -c Debug: 0 Warning(s), 0 Error(s)
• EventWindowedReadsAreBoundedLivePostgresTests alone, fresh darlingtest, 5 consecutive runs: 5/5 pass (Total: 2, Failed: 0 each time)
• Floor-removed proof run: fails as expected, reverted
• Full Darling.Tests suite, once, fresh darlingtest: Total: 13893, Errors: 0, Failed: 1, Skipped: 47, Not Run: 1, Time: 552.377s. The one failure, ServerListAndSummaryPlanShapeTests.TheShippedReads_TouchFarFewerChunks..., did not appear in any CI run in the census. It passed alone on a fresh database: an artifact of running all 13,893 tests serially. A single CI shard runs about a third of the classes, so it accumulates far less collection_log chunk state than an unsharded run. Per the lane orders' pre-existing-failure rule, a failure that clears on a fresh database alone is not filed.
• Rig stopped after the run

Issues filed

• CI flake: PgTarget anomaly/blocking worst-tile assertions fail on first attempts unrelated to the change #4274: PgTargetAnomalyTests/PgTargetBlockingTests worst-tile flake family (2+1 occurrences in this census), root cause not confirmed, needs real tracing into tile selection.

For the coordinator to double-check

• The DistinctChunkCount reasoning holds for job_history (same hypertable, two scans). I did not audit every other live plan-shape test for the same double-scan-over-one-hypertable shape. Any such test has the identical latent bug.
• I did not touch #4198's in-flight PRs or the fix/4198-* build failures. Those are each branch's own compile breaks, unrelated to this flake.