FinOps refresh: take the FinOps tab off the 30s timer, merge its doubled query_stats and bounds reads (#4227)

[erikdarlingdata]

Part of #4227.

Why

The Mac's second-pass field measurements (issue comment) showed the FinOps tab's cost was not limited to
Server Inventory's idle-database read. That read belongs to lane V3a, on its own branch, and is untouched here.
The whole tab ran on the 30 second fleet timer while visible. Every figure on it moves hourly (utilization) or
daily (sizes, inventory, growth) at the fastest, so that timer bought nothing. Two of its reads also duplicated
work. The two top-consumer grids aggregated the same 24h of raw query_stats independently, and the Storage
Growth object drill's summary and series reads each recomputed the same MAX/MIN(collection_time) bounds.

What changes

1. Cadence. FinOpsTab joins RecommendationsTab and OverviewTab in OnRefreshTimerTick's tab
   early-return (MainWindow.xaml.cs). Every FinOps sub-tab already reloads on tab activation, on its own
   sub-tab switch, and on the FinOps Refresh button. That is FinOpsTab.xaml.cs's existing
   RefreshActiveSubTabAsync, FinOpsSubTabs_SelectionChanged, and FinOpsRefresh_Click, none of which
   changed. Only the 30 second poll is removed.
2. Utilization's two top-consumer grids. TopResourceConsumersByTotalSql and TopResourceConsumersByAvgSql
   (ViewerDataService.FinOps.Workload.cs) are replaced by one TopResourceConsumersSql, grouped by
   database_name, computing both the total and the CASE WHEN execution_count > 0 average in one pass. A new
   GetTopResourceConsumersAsync reads it once and ranks and limits both grids client-side, with
   OrderByDescending (a stable sort) plus Take(topN). One statement can only ORDER BY/LIMIT one column,
   and the two grids rank by different ones, so that ranking step moved to C#. The tiered-retention routing
   (TopResourceConsumersSqlFor for raw/hourly/daily, via RouteOrThrow/ConsumerCteForCagg) keeps its shape. It
   it now runs once instead of twice, because the merged query's workload CTE is byte-identical to the
   fragment the routing already swaps.
3. Storage Growth object drill. ObjectGrowthSummarySql and ObjectGrowthSeriesSql
   (ViewerDataService.FinOps.Storage.cs) no longer each carry their own bounds CTE. A new
   ObjectGrowthBoundsSql computes MAX/MIN(collection_time) once. GetObjectGrowthHeatmapDataAsync reads
   it first and passes both instants as plain parameters to both statements. A window with nothing in it
   (collection never started, or stopped before the window) now short-circuits to an empty result without
   issuing either statement, instead of both independently matching zero rows the long way.
4. Live equality pins and rig numbers (this lane, V3c). Added Darling/Darling.Tests/FinOpsMergedReadsLiveTests.cs:
   two [Collection("live-postgres")] classes that seed real data and compare the new reads against the exact
   old SQL text (copied verbatim from origin/dev, since the old constants no longer exist in source). See
   "Diagnosis" and "Test plan" below.
5. Job 4 (DatabaseSizeLatestSql planning time) is not done. Diagnosed but not implemented: see "What's left".
6. Lite parity is out of scope for this PR. Lite is a separate lane's (V3d) own PR against WPF FinOps Server Inventory re-runs a 7-day raw query_stats DISTINCT for every server on the 30 s timer (~43 × 44 k buffers per refresh); query_stats_db_hourly answers it in 32 ms / 175 buffers #4227. The brief
   for this PR explicitly excludes it. V3b's inspection notes (cadence needs no fix, the two double-reads are
   present in Lite in the same shape) are left below for that lane's reference.

Behavior notes worth a second look

• The old TopResourceConsumersByTotalSql excluded a NULL (unattributed) database_name. The old
  TopResourceConsumersByAvgSql never did. The merged statement bakes in neither filter: it returns every
  database, NULL name included, and GetTopResourceConsumersAsync applies each grid's own old filter
  client-side (skip NULL name for ByTotal; skip zero-execution, via a NULL avg_cpu_ms, for ByAvg). The two
  grids keep their previous, different inclusion rules. Live-pinned below.
• Tie-breaking changes. The old behavior was whichever order Postgres's plan happened to return, from
  ORDER BY <metric> DESC LIMIT $3 with no secondary key, which was never a documented contract. The new
  behavior is database name ascending: the merged query's ORDER BY c.database_name, broken by a stable
  OrderByDescending in C#. cpu_time_ms and avg_cpu_ms are sums of independent per-database deltas, so an
  exact tie is not expected in practice, but the live pin below seeds one deliberately and asserts same rows /
  same values (canonicalized), rather than asserting an old order that was never a real contract.

Diagnosis

Confirmed on a rig this lane stood up (port 55973): PostgreSQL 18.6 + TimescaleDB 2.30.1, darlingtest seeded
with 40 servers (server_id 90001-90040), the target server (90001) carrying 24h of query_stats/file_io_stats
at 5-minute density across 30 databases (NULL name, zero-execution, and a forced tie baked in), and a
GrowthDb with 31 days of daily index_object_stats across 50 tables. EXPLAIN (ANALYZE, BUFFERS), one run
each (not averaged: see caveat below):

Utilization top-consumers (server 90001, 24h raw window):

statement	execution time	planning time	buffers (exec / planning)
OLD ByTotal	10.079 ms	8.242 ms	280 / 744
OLD ByAvg	8.648 ms	0.505 ms	277 / 24
OLD pair, summed	18.727 ms	8.747 ms	557 / 768
NEW merged	8.985 ms	0.546 ms	280 / 21

~2.1x less execution time, ~16x less planning time (one statement planned instead of two), same execution
buffers as the more expensive of the two old statements (not doubled), ~37x fewer planning buffers.

Storage Growth drill bounds (server 90001, GrowthDb, 30-day window, topN 20):

statement	execution time	planning time	buffers (exec / planning)
OLD ObjectGrowthSummarySql (own bounds CTE)	0.432 ms	3.433 ms	23 / 140
OLD ObjectGrowthSeriesSql (own bounds CTE)	2.581 ms	0.961 ms	49 / 117
OLD pair, summed	3.013 ms	4.394 ms	72 / 257
NEW ObjectGrowthBoundsSql (computed once)	0.055 ms	0.268 ms	6 / 4
NEW ObjectGrowthSummarySql (bounds as params)	0.290 ms	0.353 ms	11 / 6
NEW ObjectGrowthSeriesSql (bounds as params)	2.250 ms	0.566 ms	40 / 8
NEW total (bounds + pair)	2.595 ms	1.187 ms	57 / 18

~14% less execution time, ~3.7x less planning time, ~14x fewer planning buffers (bounds priced once instead
of embedded in each of two statements).

Next to the issue's own Mac field numbers (#4227's comment, a real fleet, not this rig's synthetic seed):
Utilization was ~1.5 s execution + 0.23 s planning per refresh; each drill bounds pass was 89 ms and 21.3k
buffers. This rig's absolute numbers are far smaller (single-digit ms, tens to low hundreds of buffers) because
the seed is a fraction of a real fleet's scale (8,640 query_stats rows for the target server vs. a real
server's weeks of raw retention). The point proved here is the shape of the fix: one statement instead of
two, one bounds pass instead of two: not a claim of matching the Mac's absolute numbers. A single EXPLAIN ANALYZE run each, not averaged over repeated runs, so treat the precise digits as indicative, not a
benchmark-grade measurement.

What is left

• Job 4 (DatabaseSizeLatestSql planning time). Diagnosed only, not implemented. Skipped per this
  lane's brief ("skip if your context is past 150k"): this lane's context watchdog fired at 200k before
  reaching it. See V3b's diagnosis (still accurate, unchanged by this lane): a collection_time >= probe
  bound, WatermarkPolicy.RecentWatermarkWindow-style, as a second statement tried first, falling back to
  today's unbounded statement on a miss (a server whose collection stopped longer ago than the probe window).
• Lite parity for jobs 2 and 3. Explicitly out of scope for this PR: lane V3d ports Lite in its own PR
  against WPF FinOps Server Inventory re-runs a 7-day raw query_stats DISTINCT for every server on the 30 s timer (~43 × 44 k buffers per refresh); query_stats_db_hourly answers it in 32 ms / 175 buffers #4227. V3b's inspection notes: cadence (job 1) needs no Lite fix (Lite's only fleet timer never
  touches FinOpsTab). The top-consumer double read and the object-growth bounds double read are both
  present in Lite in the same shape as the pre-fix Darling code.
• Live rollup-tier (hourly/daily) execution was not added. ConsumerCteRaw/ConsumerCteForCagg and the
  RouteOrThrow substitution are byte-identical to origin/dev (diffed by hand) and are already live-pinned
  at the SQL-text level by RetentionTierRouterTests/ViewerFinOpsSqlTests. This rig's darlingtest has the
  timescaledb extension but migrations did not materialize any continuous aggregates/hypertables on it (no
  rows in timescaledb_information.hypertables), so seeding a live hourly-tier comparison would have needed
  standing up that provisioning path first: out of this lane's remaining time. Only the raw-tier merge and
  the new client-side ranking are new risk surface versus origin/dev, and both are fully live-pinned below.

Test plan

• dotnet build Darling/PerformanceMonitor.Darling.Viewer/PerformanceMonitor.Darling.Viewer.csproj: 0
  Warning(s), 0 Error(s).
• dotnet build Darling/Darling.Tests/Darling.Tests.csproj: 0 Warning(s), 0 Error(s) (both before and after
  git merge origin/dev).
• New live equality pins (FinOpsMergedReadsLiveTests.cs), against a real rig (PostgreSQL 18.6 +
  TimescaleDB 2.30.1, DARLING_TEST_PG):
  • FinOpsTopConsumersMergeLiveTests: the merged TopResourceConsumersSql plus
    GetTopResourceConsumersAsync's client-side ranking returns the same rows as the old ByTotal/ByAvg
    statements (copied verbatim from origin/dev), covering a NULL database_name, a zero-execution database,
    and a genuine cpu_time_ms/avg_cpu_ms tie between two other databases, on the raw tier. Ties are compared
    canonicalized (metric DESC, name ASC) rather than by raw returned order, since the old tie order was never a
    documented contract (see "Behavior notes" above): same rows, same values, proven either way. Pass.
  • FinOpsObjectGrowthBoundsMergeLiveTests: the bounds-computed-once form returns exactly what the old
    per-statement bounds CTE pair returned, including exact row order (this read's ORDER BY has a full
    tie-break, so no canonicalization needed), for both a real two-object growth/shrink window and the
    empty-window case (a database whose only data predates the window). Pass.
  • Both proven against literal git show origin/dev:... copies of the retired SQL constants, since those
    constants no longer exist in current source.
• Every class touched or added to: ViewerFinOpsSqlTests, RetentionTierRouterTests,
  ViewerFleetTimerFanOutPositionTests, FinOpsTopConsumersMergeLiveTests,
  FinOpsObjectGrowthBoundsMergeLiveTests.
• Sibling/regression + census gates run together: ViewerFleetTimerGuardTests, ViewerFinOpsRecommendationsTests,
  McpPayloadContractCensusTests, StartupCommandTimeoutTests, StorageCommandTimeoutTests,
  AlertReadFailureSurfaceTests, DocCommentHygiene, LivePostgresCollectionHygieneTests,
  LiveCleanupConversionRatchetTests. 278/278 passed, 0 failed.
• New source pin FinOpsTab_IsInTheRefreshTicksTabEarlyReturn (ViewerFleetTimerFanOutPositionTests.cs).
  Proven red by hand against the pre-fix source by V3b (reverted the exemption locally, confirmed the new
  fact failed, then restored it).
• git merge origin/dev: merged cleanly, no conflicts (19 files, mostly MCP fleet-response-budget work
  unrelated to FinOps).
• Full suite with the rig (darlingtest dropped and recreated first): 13,801 total, 0 errors, 12 failed,
  47 skipped, 1 not run. None of the 12 failures are in a file this PR (or V3b's commit) touches. They
  span RollupCoverageRoutingTests, PgTargetWaitLiveTests, ViewerCalendarRetentionLivePostgresTests,
  TopCpuQueriesTextLiveTests, ViewerSessionStatsLivePostgresTests, CaptureDownChunkOrderTests,
  DefaultTraceEventFrameLivePostgresTests (3 of its offset cases), and
  PgTargetTileBehaviourIoReplicationBlockingTests (2 cases): all unrelated subsystems, and most are named
  around UTC offsets/shifts, consistent with this rig's initdb defaulting to America/New_York for part of
  the run (see next item). Not independently re-verified against a fresh dev + CI per lane-orders' full
  pre-existing-failure protocol: flagged here rather than silently dropped. This needs a coordinator
  double-check, not a claim of "pre-existing" on my say-so alone.
  • Mid-run timezone fix (coordinator follow-up). This rig's initdb took the machine's local zone
    (America/New_York); CI runs UTC, and this lane appended timezone = 'UTC' / log_timezone = 'UTC' to postgresql.conf
    and restarted (pg_ctl restart -m fast) partway through the full-suite run above (confirmed via the
    post-restart log switching to UTC timestamps); the in-flight run's connection pool reconnected and kept
    going, so the run mixes a short pre-restart window with a UTC majority. Of the three tests the coordinator
    named, ServerListAndSummaryPlanShapeTests and ViewerW2aLivePostgresTests.ServerSummary_... did not
    appear in the failure list. CaptureDownChunkOrderTests did. Re-ran it alone afterward, confirmed
    server-side UTC by then: it still failed alone. Per the brief, not chased further. It is in
    CaptureDownChunkOrderTests.cs, a collection-log chunk-ordering test this PR's diff does not touch.
    darlingtest was NOT re-dropped/recreated after the restart before the run above finished (the run was
    already past its halfway point when the restart landed, and redoing a ~13-minute full run risked missing
    this lane's deadline): a clean full run entirely under UTC is worth a coordinator re-check.

CHANGELOG entry

SECTION: Fixed
ENTRY:

• FinOps tab no longer polls every 30 seconds ([FinOps refresh: take the FinOps tab off the 30s timer, merge its doubled query_stats and bounds reads (#4227) #4240]): The FinOps tab's Utilization, Server Inventory,
  Storage Growth, and other sub-tabs re-ran their reads on the fleet's 30 second timer, even though the
  underlying numbers move hourly at the fastest. It now refreshes only when you open it, switch its sub-tabs,
  or click Refresh, the same as the Recommendations tab already did. The Utilization tab's two "top resource
  consumer" grids, and the Storage Growth drill's summary and trend views, also stopped scanning the same data
  twice to build what used to be two separate reads.
  REF:
  [FinOps refresh: take the FinOps tab off the 30s timer, merge its doubled query_stats and bounds reads (#4227) #4240]: FinOps refresh: take the FinOps tab off the 30s timer, merge its doubled query_stats and bounds reads (#4227) #4240

🤖 Generated with Claude Code

https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3