FinOps Server Inventory: one fleet query instead of one per server (#4227)

[erikdarlingdata]

Part of #4227.

Lane V3a wrote the code and the source/unit pins, then ran out of budget before the live tests. Lane V3e (this pass) ran the live tests, the before/after numbers, the census gates, and the full suite. It found and fixed one real bug the fleet rewrite introduced, and fixed one census pin the change legitimately moves.

Why

FinOps Server Inventory called GetServerMetricsAsync once per server (43 round trips on a 43-server fleet). Each call ran a raw 7-day v_query_stats scan just to answer one boolean per database: did it run anything. Across a fleet that is roughly the whole raw query_stats table, every 30 seconds the sub-tab was visible.

The coordinator's ruling (posted on #4227 and in lane V1's handoff comment) set the shape: one fleet statement instead of 43. The idle check answers from the stitched query_stats_db_hourly and query_stats_db_interval_hourly rollup plus raw only for the two edges the rollup cannot cover. The rollup routing uses RollupCoverage.StitchedRelationSql, the same routing ViewerDataService.FinOps.Workload.cs already uses. That rollup pair is materialized_only = true. #4186 froze the legacy relation. It is never read alone.

What changes

• Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FinOps.Inventory.cs: ServerMetricsSql is now one fleet-wide statement (no server_id parameter) instead of a per-server template. cpu_24h and grants group by server_id. mem_latest, storage_totals and latest_dbs use a per-server CROSS JOIN LATERAL ... LIMIT 1 descent off servers (the same shape ServerFreshnessSql already uses, Fleet overview reads every retained row of four tables on every call, and the web viewer calls it twice a minute per open tab #3895). GetServerMetricsAsync() dropped its serverId parameter and now returns Dictionary<int, ServerMetricsRow> for the whole fleet in one round trip.
  • New: ServerMetricsSqlFor(RollupCoverage, DateTime idleCutoffUtc, DateTime watermarkUtc) swaps the raw active_dbs CTE for a three-arm union. Arm 1: the stitched rollup for whole hours from the idle cutoff's ceiling hour onward. Arm 2: raw for the partial leading hour [idleCutoff, ceilingHour). Arm 3: raw for the trailing unmaterialized range [watermark, now).
  • Bug found and fixed: idle_dbs originally grouped latest_dbs EXCEPT active_dbs by server_id. A server whose every known database is active has zero rows surviving that EXCEPT. GROUP BY then contributes no row for it at all, and the outer LEFT JOIN reads idle_db_count as NULL instead of 0. The old per-server statement never had this trap: a bare COUNT(DISTINCT ...) with no GROUP BY always returns exactly one row. FinOpsTab.Loaders.cs's overlay only assigns a value that arrived (if (row.IdleDbCount.HasValue)). A healthy, all-active server then keeps showing whatever idle count it last had instead of updating to 0. Fixed by rooting idle_dbs at servers (a LEFT JOIN + COUNT(...) FILTER (...), like size_latest/mem_latest already do) so every server gets a row. A live database, not a code read, surfaced it.
• Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs: new RollupMaterializationWatermark class beside RollupCoverage. GetAsync reads a CAGG's materialization watermark via _timescaledb_functions.cagg_watermark, falling back to max(bucket) + one bucket width off the view itself on any Postgres/cast failure. The fallback is never later than the true watermark, so it only ever reads more raw, never less. V3a confirmed on the rig (as a role carrying only the production viewer grants) that both the function and _timescaledb_catalog.continuous_agg are callable and readable.
• Darling/PerformanceMonitor.Darling.Viewer/FinOpsTab.Loaders.cs: LoadFinOpsServerInventoryAsync now calls GetServerMetricsAsync() once and overlays the returned dictionary in a plain loop. This replaces the ViewerReadFanOut.Lanes/Task.WhenAll fan-out that used to run one call per server.
• Darling/Darling.Tests/ViewerCommandTimeoutTests.cs: NoDeclaredFanOutWidth_OutrunsThePermitsOrTheMeasurement counted two runtime (Lanes(...).Count-derived) fan-out declarations project-wide, one of which was this loader's now-removed one. Dropped the floor from 2 to 1. This is a deliberate pin update, not a weakening. The other runtime fan-out (MainWindow.xaml.cs's fleet-totals read) is untouched.
• Two new live test files added by this pass: Darling/Darling.Tests/ServerInventoryFleetMetricsLiveTests.cs (below).
• Lite is out of scope for this lane (V3d's part of WPF FinOps Server Inventory re-runs a 7-day raw query_stats DISTINCT for every server on the 30 s timer (~43 × 44 k buffers per refresh); query_stats_db_hourly answers it in 32 ms / 175 buffers #4227). The FinOps refresh timer/cadence is lane V3b's part.

Numbers

Seeded probe database: 40 servers, 7 days of hourly CPU/memory-grant/query_stats data, one database_size_stats snapshot per server. Each server had 5 databases, one permanently idle. Rollups refreshed to a 6-hour unmaterialized tail. EXPLAIN (ANALYZE, BUFFERS), one run each:

Statement	Round trips	Total execution time	Total shared buffer hits
Old per-server (origin/dev's ServerMetricsSql, summed over 40 runs)	40	54.2 ms	170,812
New fleet statement (ServerMetricsSqlFor, routed through the rollup + raw tail)	1	14.9 ms	8,322

About 3.6x less total execution time and 20.5x fewer buffer hits, on top of the 40:1 round-trip reduction. Each of the 40 old round trips also paid its own network/connection-pool overhead in production. This in-process measurement does not capture that cost.

Test plan

• dotnet build Darling/Darling.Tests/Darling.Tests.csproj -c Debug: 0 Warning(s), 0 Error(s).
• Darling.Tests.exe -class "*ViewerFinOpsSqlTests*": 49/49 pass.
• Darling.Tests.exe -class "*ServerInventoryFleetMetricsTests*": 6/6 pass. One assertion updated to match the idle_dbs fix's SQL shape (LEFT JOIN/FILTER instead of EXCEPT).
• New live tests, ServerInventoryFleetMetricsLiveTests.cs, both against the rig (TimescaleDB, ScratchPostgres-isolated):
  • ServerMetricsSqlFor_FindsActiveDatabasesInEveryRegion_AndOnlyTheNeverExecutedOneIsIdle: seeds legacy rollup for one hour, successor rollup for the next two, a partial-leading-hour row, and a raw-tail row after the watermark. Over a 5-database universe, idle_db_count reads exactly 1. A miss in any active region (legacy, successor, raw tail, or partial leading hour) pushes the count above 1. Also reads the watermark two ways (cagg_watermark and the max(bucket) fallback) and confirms both give the same idle_db_count.
  • GetServerMetricsAsync_FleetResult_MatchesTheOldPerServerStatement_ServerByServer: 3 servers, deliberately different shapes (mixed active/idle, all-active, all-idle with no CPU/memory/grant evidence). Every field (AvgCpuPct, StorageTotalGb, IdleDbCount, ProvisioningStatus) matches the old per-server text (frozen verbatim from origin/dev) for all 3 servers. This is the test that caught the idle_db_count NULL-vs-0 bug.
  • Both pass: 2/2.
• Census gates: McpPayloadContractCensusTests, StartupCommandTimeoutTests, StorageCommandTimeoutTests, AlertReadFailureSurfaceTests, DocCommentHygiene, LivePostgresCollectionHygieneTests, LiveCleanupConversionRatchetTests: 154/154 pass.
• git merge origin/dev: clean, no conflicts.
• Full suite, rig-backed (DARLING_TEST_PG, fresh darlingtest): 13,814 total, 0 errors, 2 failed, 47 skipped, 1 not run.
  • ViewerCommandTimeoutTests.NoDeclaredFanOutWidth_OutrunsThePermitsOrTheMeasurement: the census pin above. Fixed. Re-ran the class alone afterward: 57/57 pass.
  • TrendPayloadBudgetLiveTests.EveryDefaultAnswer_StaysNearTheBudget_AndTheLargestAnswerStaysUnderTheCap: not mine, not confirmed pre-existing. It fails on an MCP trend tool each time ("Exception while reading from stream"), a different tool each run. Nothing in this PR's diff touches trend tools or MCP payloads. It also fails alone on a freshly created database. I did not have budget to check dev's own CI run to call it pre-existing. Reporting it as unrelated and needing a look.
• Not re-run: the complete full suite after the last two fixes. Both were verified individually, but a second 10-minute full run was outside the budget. Worth one more full run before merging.

CHANGELOG entry

SECTION: Fixed
ENTRY:
- **FinOps Server Inventory: one fleet query instead of one per server** ([#4241]). The Server Inventory grid ran a per-server round trip on every refresh (43 on a 43-server fleet). Each round trip included a raw multi-day scan just to flag idle databases. It now runs one fleet-wide statement. The idle check answers from the hourly rollup plus raw only for the parts the rollup has not materialized yet. Also fixed, before it ever shipped: a server with zero idle databases showed a blank idle count instead of 0.
REF:
[#4241]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/4241

Generated with Claude Code

https://claude.ai/code/session_01FVjn4PBJN71NQXdFo6ZxNQ