Cache the I/O-latency and CPU baselines by UTC day, not the hour (#4248)

[erikdarlingdata]

Closes #4248.

Why

PgBaselineProvider's IoLatency arm reads the raw file_io_stats hypertable at per-file grain over the full 30-day baseline window. The cache key was the analysis hour. So every server recomputed that 30-day median/MAD scan every hour, even though a 30-day baseline moves by about 1/720 an hour. The issue measured about 50 MB of temp spill per call on that arm alone. That was 13.5 of 22.1 GB of one store's total temp writes since its stats reset.

Confirmed on the rig: Cpu reads its own raw table (cpu_utilization_stats) the same way. The code already documents this split, in the #1743 comment on GetBaselineQuery.

Of the nine RobustTierScaffold arms, only Cpu and IoLatency read a CREATE TABLE hypertable directly. The other seven arms read a _baseline materialized view that holds one row per collection, which is far fewer rows for the same 30 days. They are BatchRequests, WaitStats, SessionCount, QueryDuration, Memory, WaitMsPerSec and BlockingPerMinute. The two event arms (Blocking, Deadlock) bypass the scaffold entirely. So Cpu and IoLatency are the full set of arms this ruling applies to in PgBaselineProvider.cs.

What changes

• RoundedKeyTime(metricName, analysisTime) picks RoundedDay (midnight UTC) for Cpu/IoLatency (IsDailyCacheMetric) and keeps RoundedHour for every other arm. Both the cache key and the compute's window end (ComputeBucketsAsync) read this one function, so Every scheduled analysis pass recomputes every 30-day baseline: a fresh DarlingAnalysisService per pass never reuses (or shares with MCP) the bucket cache #3941's "one key, one set of rows" invariant holds at either grain.
• A successful compute of a daily-cache metric gets CachedBaseline.FreshUntilUtc, set to RealTime plus 24 hours. The existing key-match check (ComputedAt == roundedHour) is what actually stops an entry answering once the UTC day ends. The 24-hour span on FreshUntilUtc is a backstop only, so an entry never dies mid-day from the old 1-hour CacheTtl. A failed compute keeps FreshUntilUtc null, so it still retries within the hour, for every metric.
• BaselineCache (the process-wide shared tier from Every scheduled analysis pass recomputes every 30-day baseline: a fresh DarlingAnalysisService per pass never reuses (or shares with MCP) the bucket cache #3941) and its sweep now call the same PgBaselineProvider.IsFresh. So they honor the same lifetime, instead of always applying CacheTtl.
• Lite's BaselineProvider.cs and BaselineCache.cs get the same change for the same two arms (v_cpu_utilization_stats, v_file_io_stats). So both products flag the same anomalies from the same cache grain.
• The lookup side is unchanged. LookUp still buckets on the analysis instant's own hour-of-day and day-of-week, in both products.

PgTargetBaselineProvider.cs (PostgreSQL targets) is left alone. It uses the same cache, and every one of its arms reads a raw hypertable. Its metric names are not in IsDailyCacheMetric's set, so it does not change here. It has keyed arms (one series per statement), and KeyedBaselineCacheWarnCount's cardinality warning assumes hourly turnover. So a day-long lifetime there needs its own ruling. #4298 tracks it.

Pins added

• Darling, live (DarlingAnomalyBaselineTests.cs): EndToEnd_IoLatencyArm_TwoCallsHoursApartOnOneDay_ShareOneCompute_NextDayRecomputes_AgainstDevPostgres. It seeds file_io_stats, then calls GetBaselineAsync at hour 1 and hour 20 of the same UTC day. It counts 0 new reads, with CommandCapture, the existing Every scheduled analysis pass recomputes every 30-day baseline: a fresh DarlingAnalysisService per pass never reuses (or shares with MCP) the bucket cache #3941 technique. A call the next UTC day counts 1 new read.
• Darling, updated: PgTargetClockTests.cs's literal source-text assertion now expects RoundedKeyTime(metricName, analysisTime), replacing the old RoundedHour(analysisTime).
• Lite: DailyCacheEntry_AnswersForItsWholeUtcDay_NotJustTheTtl_AndStopsAtMidnight pins the day-long liveness and the midnight cutoff directly against BaselineCache, with a synthetic entry and no DB seed needed. ASharedEntry_IsTheFreshAnswer_AndReadsNothing exercises MetricNames.Cpu directly, and its "next analysis hour is another window" case needed a correction: that is no longer true for Cpu under this change. The comment and assertion now say so, instead of passing on a check that had gone silently weak.

Lite's full suite ran clean, and Darling's four failures are in code this PR does not touch (below). So I did not add an automated pg_stat_statements.temp_blks_written live test. Gating it correctly on whether CI's darling-pg job preloads pg_stat_statements needed more checking than this lane had time for. I measured it by hand on the rig instead:

	before (hourly key)	after (daily key)
IoLatency, 3 calls same UTC day, over an hour apart	3 computes, 3 potential spills	1 compute
IoLatency, 1 call the next UTC day	recomputes (expected)	recomputes (expected)

Test plan

• Darling.Tests: 0 Warning(s), 0 Error(s).
• Lite.Tests: 0 Warning(s), 0 Error(s).
• Full Darling.Tests.exe run once, after git merge origin/dev and a dropped and recreated darlingtest: 14004 total, 0 errors, 4 failed, 48 skipped, 1 not run.
  • All 4 failures are EXPLAIN-plan-shape and chunk-pruning tests on collection_log and wait-rate tiles (ForcePlanFailuresAccessPathTests, WaitRateTileReadCountLiveTests, ServerListAndSummaryPlanShapeTests, CaptureDownChunkOrderTests). None relate to PgBaselineProvider or BaselineCache, the only production files this PR touches.
  • I re-ran each alone on the same database. 2 of 4 passed, which points to interference between tests on the shared darlingtest database. The other 2 (ServerListAndSummaryPlanShapeTests, CaptureDownChunkOrderTests) still failed alone on that rig. This diff touches none of the code they test.
• Full Lite.Tests.exe run once: 5373 total, 0 errors, 0 failed, 0 skipped.
• Diagnosis confirmed on the rig with the measured table above, not just asserted from the issue's production numbers.
• GitHub Actions CI on 8a1f0ee9: build, Darling PostgreSQL tests and Lite tests all pass.

CHANGELOG entry

SECTION: Fixed
ENTRY:

• CPU and I/O-latency baselines recompute once a day, not every hour ([Cache the I/O-latency and CPU baselines by UTC day, not the hour (#4248) #4291]) - Darling and Lite recomputed these two anomaly baselines every hour. Each time, they scanned 30 days of raw rows. The I/O-latency scan alone spilled about 50 MB to temp. They now recompute once per UTC day, and their 30-day window ends at midnight UTC. A 30-day baseline barely moves within a day. A failed compute still retries within the hour.
  REF:
  [Cache the I/O-latency and CPU baselines by UTC day, not the hour (#4248) #4291]: Cache the I/O-latency and CPU baselines by UTC day, not the hour (#4248) #4291

🤖 Generated with Claude Code

https://claude.ai/code/session_01FVjn4PBJN71NQXdFo6ZxNQ