Analysis passes share each 30-day baseline once per analysis hour instead of recomputing it every pass (#3941)

[erikdarlingdata]

Closes #3941.

Why

Every scheduled analysis pass was cold. The worker builds a fresh DarlingAnalysisService per pass, and the service built its own PgBaselineProvider / PgTargetBaselineProvider, so the providers' 1-hour bucket cache never outlived a pass. Every pass recomputed every 30-day baseline, and the MCP host's singleton recomputed them again for analyze_server and compare_analysis.

Measured on a rig first (PostgreSQL 18.6 + TimescaleDB 2.30.1; a store built the way the service builds it, with 71 hypertables and 23 continuous aggregates; 6 PostgreSQL targets and 6 SQL Server servers, each with 31 days of one-minute and five-minute collections, about 2.9M rows). The field has 43+ servers. A pass's cost is per server, so the share of the pass holds at that size, and store load scales with servers x passes. Averages over 6 servers, warm buffers:

Pass (two runs on the same store)	Wall	Statements	Baseline reads	Baseline time
PostgreSQL target, fresh service (today)	938 to 1,252 ms	91	12	884 to 1,017 ms (81 to 94 %)
SQL Server, fresh service (today)	167 to 209 ms	67	9	137 to 159 ms (76 to 82 %)
PostgreSQL target, second pass in the hour (this PR)	36 to 301 ms	55	0	0
SQL Server, second pass in the hour (this PR)	26 to 107 ms	47	0	0

The seeded SQL Server servers only had wait and CPU history, so the other SQL Server arms read empty aggregates. On real data the SQL Server share will be higher. The issue's diagnosis holds. The keyed statement arms were only part of the cost: every unkeyed arm ran again on every pass, plus a clock read and supply probes per compute (the 36 and 20 statements that drop out above).

What changes

A process-wide second tier, BaselineCache. The three hosts read the store as three roles (worker = owner, MCP = mcp, web = viewer, #3914). Sharing providers would have run the MCP and web reads as the worker's role, so they share the computed buckets instead. The tier is a DI singleton. The worker hands it to every per-pass service, the MCP host to its singleton, and the web host to MapAll's service. Each provider keeps its own data source and its own cache as the first tier, and consults the shared tier only on a miss.

When a shared entry answers:

• Analysis hour. An entry answers only the hour it was computed for.
• Window. The compute's window now ends on that hour, not on the analysis instant. The cache has always keyed on the rounded hour but computed over whichever instant inside the hour asked first, so a later caller could get a window up to 59 minutes off its own. Now every caller in the hour asks for the same rows, and a shared entry is exactly what a fresh compute returns. This is the one answer change in the PR: the window's end moves back by under an hour out of 720, and the detectors still ask four hours back, so those rows are settled.
• Time. An entry dies CacheTtl (1 h) after its compute, which bounds late rows, purges and a target changing zone. Dead entries are swept at most every quarter-TTL, so the tier holds about one TTL of computes. Measured on the rig: 150 entries for 12 servers took 1.74 MB, about 145 KB per server. Across an hour boundary the old hour lives until the sweep, so the peak is at most double that.
• Failure. Only successful computes are shared. A timeout, or a role that cannot read a relation, stays the failing provider's "no baseline this pass". A shared success beats a local failure.
• Engine. The provider type is part of the key, so SQL Server and PostgreSQL-target series never answer for each other.
• What does not invalidate. Thresholds and settings: nothing configurable reaches the baseline SQL. Removing or re-adding a server: remove_server keeps the history, and the hash id is stable.

Effect. The scheduled passes' baseline reads halve, from servers x 2 per hour to servers x 1. At the default 30-minute interval, each clock hour's two passes share one analysis hour. analyze_server and compare_analysis run inside an hour a pass already computed read no baseline. A persisted table was not needed, so there is no migration.

A long-lived provider's own cache is swept too. A per-pass provider dies before its cache matters. The MCP and web hosts' singletons, though, kept every keyed series they were ever asked for, because the keys follow each server's top statements. KeyedEntryCount counted those dead series toward the cardinality note's 500 bar, so a consumer that did bound itself would still have tripped the warning after enough days. The provider now runs the tier's sweep, through the same gate and removal, over its own cache. Lite's provider has no keyed series, so its cache is already bounded by servers x metrics.

Lite parity. Lite had the same shape: a fresh AnalysisService per scheduled pass and per Recommendations-tab run, an MCP singleton, and per-tab LocalDataService providers. It gets the same tier, one per DuckDB store (BaselineCache.For(duckDb), never a process static), handed to all four production sites, with the same window snap.

Test plan

• Live: two private providers at H:05 and H:50 compute identical baselines for every PostgreSQL arm. On the old window shape this fails (pg_tps: 245 samples vs 290).
• Live: a shared entry equals a fresh compute and issues no statement. This holds for unkeyed arms and for the keyed statement sets.
• Live: two fresh DarlingAnalysisServices in one analysis hour. The first reads ≥ 10 baselines, the second 0, and a private one reads them all again. The anomaly detector's facts from the warm shared provider equal those from a cold private one. Without the snap they differ (baseline_samples=50 vs 58).
• Live: a failed compute (a store the provider cannot read) is not shared, and the next provider computes for itself. The failed provider then takes the shared success.
• Live: the next analysis hour recomputes, and InvalidateCache clears the shared tier.
• Live: a long-lived provider's keyed series leave KeyedEntryCount once they are a TTL old, and stay while they are live.
• Unit: hour, engine, TTL, sweep gate and removal, and invalidation rules. Wiring pins cover the DI singleton, the worker, the MCP host, the web host, and exactly three construction sites.
• Lite: the snap; shared == fresh, with no read proved by mutating the store under the tier; a compute with no buckets is not shared; one tier per store; every production site wired.
• Pins updated deliberately, each because its lookup sat inside the analysis hour its rows came from:
  • PgTargetClockLiveTests and both LocalClockBucketKey live tests (Darling and Lite): lookups moved one week later to the same local hour-of-week, so the planted rows are wholly inside the window. The DST smear is still demonstrated (UTC twin: median 10, mean 32.5).
  • Source-text pins now name Store(...) and windowEnd: PgBaselineProviderKeyedTests, PgTargetClockTests.
  • FleetSweepWebFeedTests: MapAll now takes _baselineCache.
• Full Darling.Tests (with DARLING_TEST_PG): 12,968 total, 0 failed, 29 skipped (environment-gated), after merging origin/dev and after the sweep commit.
• Full Lite.Tests: 5,167 total, 0 failed.
• 0 warnings on both builds.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GdmA4ND1wLSqA91ax1m4xv