get_fleet_overview: one collection-health scan per minute per host, however many overview calls race (#3735)

[erikdarlingdata]

Mechanism

DarlingFleetReader.GetFleetOverviewAsync fans out thirteen bounded cross-server reads. Twelve of them depend on the call — they carry the caller's blocking/deadlock window, or they are latest-snapshot DISTINCT ON reads that cost one index descent per server. One does not: ReadFailingCollectorCountsAsync runs FleetCollectionHealthSql over v_collection_log with $1 = now - 7 days, whatever hours_back was. Three overview calls with three windows were three identical copies of one 7-day GROUP BY server_id, collector_name aggregate, and because the web fleet page reaches the same read through /api/fleet, a browser's auto-refresh and an agent's tick stack the same way.

This PR single-flights that one read and memoizes it for a minute, per data source:

• CollectionHealthMemo (nested in DarlingFleetReader, one per NpgsqlDataSource through a ConditionalWeakTable): concurrent callers share ONE in-flight scan; a completed scan younger than CollectionHealthMemoLifetime = 60 s is served from memory. Sixty because the rollup counts per-collector RUNS over seven days and the fastest collector cadence in the ladder is one minute — between two reads a minute apart at most one run per collector has landed, and no band CollectorHealth.HealthStatus computes over a week of them can have moved. Shorter re-runs a 7-day aggregate to learn nothing; longer makes the fleet's collector dots lag get_collection_health, which reads the same rows uncached.
• Per data source, not per process. The MCP host and the web host each build their own NpgsqlDataSource (different roles, different pools), so each holds its own memo and one scan per minute per host is the bound — the same reason ComposeStoreAvailability keys its cache this way, and the same protection for gated-live tests spinning several stores in one process.
• The caller's nowUtc is the memo's clock. It is already the reader's freshness reference (the instant generated_at carries) and both production callers pass DateTime.UtcNow, so the age published beside the payload is measured from the instant the payload itself publishes, and a test can drive expiry without sleeping. A caller whose now reads behind the memo's is served at age 0 — it is holding the freshest reading there is.
• The shared scan runs on CancellationToken.None, bounded by the 20 s McpCommandDeadlines.ReadSeconds and the role's 15 s statement_timeout; a caller's own token releases only that caller (Task.WaitAsync(token)). When the first caller cancels while two others wait: that caller is thrown to at once, the statement keeps running for the two, and its result is memoized for everyone who arrives inside the next minute. Reasons, in weight order: the MCP surface never hands a token down at all (McpCommandDeadlines documents it), so on the surface the photo was taken the scan already ran to its deadline whatever the caller did; the web surface threads RequestAborted, and a tab that navigates away mid-refresh must not discard a scan two other waiters or the next tick are about to use — the alternative, a linked token cancelled when the last waiter leaves, turns an auto-refreshing browser into cancel-and-restart churn, N sequential copies of the statement instead of N concurrent ones; and the worst case under None is bounded to one orphaned statement per host per minute, which is the bound this change promises anyway.
• A failed scan is not memoized, and cannot become an unobserved exception. The shared task carries its outcome (ScanOutcome with an ExceptionDispatchInfo) rather than faulting: waiters present rethrow the original with its own stack; the memo keeps whatever it last read successfully (still served while under a minute old — one timed-out re-read must not turn into a burst of retries from every caller of the next minute); the next caller past that starts a fresh scan. A scan every waiter had abandoned completes quietly instead of surfacing on the finalizer thread.
• FleetOverviewResult.collection_health_age_seconds — trailing, additive, on both JSON surfaces (the MCP tool and /api/fleet serialize the same object through the same options): 0 when this call ran the scan, else the memo's age in whole seconds. generated_at minus this is the collector-health half's own reference instant. Its doc names the half it qualifies (per-card healthy_collector_count / failed_collector_count / collector_count / collector_severity / deadlock_collector_band / deadlock_source; the fleet's servers_with_collection_failures and deadlock_coverage), and the tool description and the MCP instructions row say the same in agent-facing words, plus the caller-side hygiene: call once with the widest window and derive, or sequence the calls.

Evidence

• The live catch (get_fleet_overview's collection-health rollup times out at :35 on the largest store — phase-grid signature, discriminate compression-band contention vs plan before fixing #3735, second comment): 13:36:03Z on the largest production store, the collection-health statement running three times concurrently — identical text, same start, one per get_fleet_overview call a single caller had raced with different hours_back values — each with two parallel workers, nine backends on collection_log in pure I/O waits while the collectors' flush band was mid-COPY. The slowest crossed the mcp role's statement_timeout = 15 s; the retry landed after the flush drained and succeeded. Solo, the plan runs in 680 ms as a healthy parallel aggregate. Hit-vs-miss hours track write-band intensity (3 of 9).
• Cleared by that evidence and therefore untouched here: the plan, the The twenty continuous aggregates join the compression ladder: compress_after derived from each tier's refresh window, a once-a-day band off the hourly grid, and the backlog staged one aggregate per night, largest first (#3581) #3611 compression band (wrong tables, ~0 s runs), and the 15 s cap, which did its job — raising it would have hidden the fan-out.
• Store-side confirmation on this branch, PostgreSQL 18 + TimescaleDB 2.28.1 in Docker with pg_stat_statements loaded: the gated live test's six GetFleetOverviewAsync calls (three raced with windows 1/6/24 h, one at +30 s, one at +61 s, one on a second data source) left pg_stat_statements reading 6 calls of the window-bound sibling FleetLastCollectionSql (unchanged: per call) and 3 calls of FleetCollectionHealthSql — one for the three raced calls, one for the re-read past the minute, one for the second host's memo. Before this change that row would have read 6.

Why this shape

• Single-flight rather than a background refresher: the read is on-demand on both surfaces and a host nobody is asking should not scan a 7-day window every minute forever.
• A memo on the one window-independent read rather than on the whole overview: the other twelve reads change with the caller's window or with every collection cycle, and a fleet band that lagged a minute behind a fresh collection_time would trade a real signal for a cheaper one. This read's answer genuinely cannot move inside a minute.
• Outcome-as-value rather than a faulting shared task: the departed-waiter case is the one the cancellation design creates, and a faulted task nobody awaits is exactly what UnobservedTaskException is for.
• Func seam rather than a mocked Npgsql: the memo takes the statement as a function of the instant and the token it decides, so the unit tests count executions on the same object the reader routes through, and the live test counts them on the real store.

What it does NOT do

• Does not change FleetCollectionHealthSql, McpCommandDeadlines.ReadSeconds, the role's statement_timeout, or hours_back semantics — pinned by source anchors in the new test file.
• Does not memoize any other overview read.
• Does not move the compression band or the collectors' flush band — the issue's phase-grid hypothesis was the first reading; the second comment's source read superseded it and this PR builds on that.
• Does not change the web page: fleet.js ignores the new field, which is what trailing-and-additive is for.

Tests

FleetCollectionHealthMemoTests (new, Darling.Tests), through the memo's seam with a gated counting scan so N callers can be parked before it produces anything:

• five concurrent callers → one statement, one shared dictionary, age 0 for all;
• served with its age at +30 s and +59.9 s (age 59), re-read at +60 s with a fresh dictionary and age 0, CollectionHealthMemoLifetime asserted equal to 60 s;
• a caller whose now is behind the memo is served at age 0 with no statement;
• a failed scan: every waiter present gets the ORIGINAL exception (same instance), and the next caller at the same instant runs statement Lite overview: show Online/Offline status for servers #2;
• a failed RE-read does not evict the last good reading (a caller inside its minute still gets it; a caller past it starts Add check for updates feature #3 rather than inheriting the failure);
• the first caller cancels while two wait: thrown to at once, the scan was handed a token with CanBeCanceled == false, the two complete on the same statement, a fourth caller 20 s on is a hit, ScansStarted == 1;
• a scan every waiter abandoned still completes and is still memoized;
• one memo per NpgsqlDataSource (same instance → same memo; second instance → different memo);
• the age rides BuildRollup (default 0, explicit 42), serializes as "collection_health_age_seconds":42, and is the LAST key;
• source anchor: the reader reaches ReadFailingCollectorCountsAsync at exactly one site, inside the memo's scan lambda; the SQL text and the CommandTimeout = McpCommandDeadlines.ReadSeconds line are byte-unchanged.

RepoFileAdoptionTests.s_lfReaders gains the new file (one entry, with its reason): the source pin's anchors span line breaks, so the file reads LF-normalised and the bounded roster has to name it — the one red of 11,472 on the first CI run, executed green here before the push.

FleetCollectionHealthMemoLivePostgresTests (gated DARLING_TEST_PG, live-postgres collection, plants nothing): three real GetFleetOverviewAsync calls raced with windows 1/6/24 h → CollectionHealthMemoFor(postgres).ScansStarted == 1 and every payload at age 0; +30 s → still 1, age 30; +61 s → 2, age 0; a second data source over the same store → its own memo at 1, the first still at 2.

Executed here, not only in CI: a net10.0 xunit harness compiling the test sources against the built Service dll (the mactest pattern), against PostgreSQL 18 + TimescaleDB in Docker — 363/363 across the new file, every DarlingFleetReaderTests class including the two live ones, FleetCardPostgresCpuLivePostgresTests (the other live GetFleetOverviewAsync caller), McpPayloadContractCensusTests, McpReadCommandTimeoutTests + CommandDeadlineScannerAdoptionTests, DocCommentHygieneTests (whole tree, cref resolver over the new doc blocks), StoreSqlClockDisciplineTests, FleetPageAttentionFilterTests, FleetCardCollectionStaleNamesItsPopulationTests, AbandonedRunEraInvariantReadTests, NothingReadBandsTests, DarlingMcpToolsTests, DarlingMcpFleetSweepToolsTests, RepoFileAdoptionTests. PerformanceMonitor.Darling.Service builds -c Release -p:EnableWindowsTargeting=true with 0 warnings. The classes that instantiate the WPF Viewer (UnmeasuredMetricsAreNotHealthyTests, FleetCardPostgresCpuTests, DeadlockRateBandRungTests, …) cannot run on this Mac and are CI's; none of them touches the changed lines beyond BuildRollup's unchanged positional prefix.

Closes #3735