Skip to content

Overview reads collection-health rollup instead of scanning raw data every 30 s (#4226) - #4236

Merged
erikdarlingdata merged 7 commits into
devfrom
fix/4226-viewer-timer-reads
Sep 25, 2026
Merged

erikdarlingdata merged 7 commits into
devfrom
fix/4226-viewer-timer-reads

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Closes #4226.

Regression fixed

ViewerW2aLivePostgresTests.ServerSummary_ReadsEnrichedThreadsMemoryBlockingCollectors_AgainstDevPostgres failed
at Darling/Darling.Tests/ViewerW2aTests.cs:1008 (HealthyCollectorCount expected 1, got 0). Dev's CI passes
this test, so this PR broke it.

Cause

FleetCollectionHealthByServerSql is the new shared read. The Overview cards, the status bar and the
server-tab badge all now call it through GetFleetCollectionHealthByServerAsync. It scoped its rows to
server_id IN (SELECT server_id FROM config_monitored_servers WHERE is_enabled). That scope was copied from
FleetCollectionHealthSql, the viewer's older, unrelated fleet-cumulative total.

The two statements do not share the same scope. Every caller of the new by-server read looks up one server's own
rows. The old raw per-server scans it replaced (CollectionHealthSql, PermissionDeniedCollectorCountSql)
never filtered by enable state. They read by server_id alone.

The test's EnrichServerId is a synthetic id with no config_monitored_servers row at all, so the is_enabled
subquery dropped it entirely. That is also a real production gap, not just a test artifact. Two cases hit it. One: a disabled server whose card, badge and tab need its last 7-day history. Two: the
bootstrap window, where config_monitored_servers is empty before the config store is seeded.

Both were confirmed by running PgMigrations.MigrateAsync alone on a fresh rig database.
collect.collection_health_hourly does not exist yet, so the raw fallback runs, not the composed read.
config.config_monitored_servers has 0 rows, so the is_enabled scope drops every server, not just the
test's synthetic one.

The right precedent was sitting right next to the moved code. The service's own by-server fleet read
(DarlingFleetReader.FleetCollectionHealthSql, now CollectionHealthRollupSupport's twin) scopes to
server_id <> 0 only. The continuous aggregate it composes with (CreateCollectionHealthHourlySql) does the
same. The service's own consumer (GetFleetOverviewAsync) reads the by-server dictionary unscoped, and lets its
own caller decide which server ids to keep. This fix adopts that same pattern for the viewer.

Fix

  • FleetCollectionHealthByServerSql now scopes to server_id <> 0 only, matching the CAGG and the service's
    read exactly. Overview cards and the badge key their lookup by one server_id. A per-server status-bar tab
    does too. Each now gets that server's real rows regardless of enable or registration state, matching the old
    raw scans exactly.
  • The status bar's fleet-cumulative branch (no tab selected) is the one place that still wants enabled servers
    only. That is the old FleetCollectionHealthSql behavior, where a removed server's aged-out rows must not
    read as erroring. The filter now lives in UpdateCollectorHealthTextAsync itself, against the fleet list
    already in memory (_fleet.All.Where(s => s.IsEnabled)). It no longer lives inside the shared SQL every
    other caller also paid for.
  • Updated FleetCollectionHealthByServerSql's doc comment and the OverviewCollectionHealthRollupRoutingTests
    pin (renamed, re-asserted on server_id <> 0) to state the corrected scope and why.
  • Added a disabled-server case and an unregistered-server case to
    FleetCollectionHealthByServerLiveEqualityTests. The composed and raw fleet-by-server reads are now checked
    against the old raw per-server scans for both, alongside the existing three enabled servers.

Lite was checked for the same class of bug. Its only change in this PR memoizes its own existing per-server
GetPermissionDeniedCollectorCountAsync scan. That scan was never is_enabled-scoped. Lite has no fleet-wide
read to share. No parity fix was needed there.

What changed

  • Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs: FleetCollectionHealthByServerSql
    scoped to server_id <> 0, doc comment corrected.
  • Darling/PerformanceMonitor.Darling.Viewer/MainWindow.ServerManagement.cs: the status bar's fleet-cumulative
    branch now filters to enabled servers itself.
  • Darling/Darling.Tests/OverviewCollectionHealthRollupRoutingTests.cs: pin renamed and re-targeted at the
    corrected scope.
  • Darling/Darling.Tests/FleetCollectionHealthByServerLiveEqualityTests.cs: added a disabled-server and an
    unregistered-server case.

Test plan

  • ServerSummary_ReadsEnrichedThreadsMemoryBlockingCollectors_AgainstDevPostgres alone: failed before the
    fix (HealthyCollectorCount 0, expected 1), passes after.
  • FleetCollectionHealthByServerLiveEqualityTests, OverviewCollectionHealthRollupRoutingTests,
    CollectionHealthAggregateTests: 18/18 passed, including the two new regression cases.
  • Every class in ViewerW2aTests.cs (ViewerOverviewSqlTests, ViewerServerSummaryDisplayTests,
    ViewerAlertHistoryW2aSqlTests, ViewerW2aLivePostgresTests) plus FreshStoreWatermarkTests: 84/84 passed.
  • Lite's PermissionDeniedBadgeMemoTests: 2/2 passed.
  • Census and pin gates (McpPayloadContractCensusTests, StartupCommandTimeoutTests,
    StorageCommandTimeoutTests, AlertReadFailureSurfaceTests, DocCommentHygiene,
    LivePostgresCollectionHygieneTests, LiveCleanupConversionRatchetTests, ReadmeDerivedCountPinTests):
    154 + 5 passed.
  • Merged origin/dev (clean, no conflicts) and ran the full Darling.Tests suite once against a freshly
    dropped and recreated darlingtest. Result: 13811 total, 1 failed, 47 skipped, 1 not run, 599s.
    The one failure was ForcePlanFailuresAccessPathTests.TheShippedRead_PlansAsAnIndexOnlyScanOnTheCoveringIndex_AgainstDevPostgres.
    It is an unrelated query-store-stats planner-shape test with zero references to collection health in its
    file. It passed when re-run alone right after. It is full-suite-only, not caused by this diff. It did not
    reproduce in isolation to file cleanly. It is worth a census if it recurs.

CHANGELOG entry

SECTION: Fixed
ENTRY:
- **Collection health by server: fixed scope so disabled and unregistered servers show their history** ([#4236]). `FleetCollectionHealthByServerSql` filtered to enabled servers only. A disabled server showed no collection history at all. A server onboarded before the config store was seeded also showed nothing. The read now scopes to `server_id <> 0` only, matching the service's own read and its CAGG exactly. The enabled-only filter moved to the status bar's fleet-cumulative branch, where it belongs.
REF:
[#4236]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/4236

Generated with Claude Code

https://claude.ai/code/session_01FVjn4PBJN71NQXdFo6ZxNQ

…er tick, not 43 raw scans (#4226)

The Overview loader called GetCollectionHealthAsync (raw 7-day collection_log)
once per server every 30 s tick, plus the status bar ran a second raw fleet
scan on the same tick. Moved the service's collection_health_hourly guard and
composer (DarlingFleetReader) into Storage as CollectionHealthRollupSupport so
the viewer can share it; DarlingFleetReader now delegates to it (same names,
same values, existing pins unchanged). Added a per-server-breakdown fleet read
(ViewerDataService.GetFleetCollectionHealthByServerAsync), memoized 20s, that
both the Overview cards and the status bar now read instead of the raw scans.
The ranked per-server CollectionHealthSql stays as-is for the Collection
Health tab, which needs the exemplar messages this fleet read does not carry.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
erikdarlingdata and others added 4 commits September 25, 2026 00:54
…llup read (#4226)

Every server-tab refresh (1 min auto, plus tab activation) ran PermissionDeniedCollectorCountSql,
its own 7-day raw collection_log scan per open tab. GetFleetCollectionHealthByServerAsync already
carries permission_denied_count per (server, collector) over the same window, so the badge now
filters that shared, memoized read instead. Pinned by a new source-text test, proven to fail on
the old raw-scan body before the fix landed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
… reads (#4226)

Per-server numbers and badge counts from the rollup-composed FleetCollectionHealthByServerSql must
equal CollectionHealthSql (Overview's old per-server scan) and PermissionDeniedCollectorCountSql
(the badge's old scan) over a window with a partial leading hour and rows in the still-open current
hour, so both the raw head slice and the materialized union are exercised. Verified the assertions
actually discriminate by temporarily comparing against the wrong server_id and watching it fail,
then reverted.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
…0 ms bar (#4226)

Lite's Overview/status-bar equivalent (RemoteCollectorService.GetHealthSummary) is already
in-memory and never touches DuckDB, so #4226's rollup fix has no Lite counterpart there. The
one DB-backed read every server-tab refresh still runs is GetPermissionDeniedCollectorCountAsync,
measured at 46-58 ms on a seeded 20-collector/7-day/1-minute-cadence DuckDB (Lite's realistic
per-server scale) -- over the lane brief's ~50 ms bar even without a fleet to amortize across. It
now memoizes per server for 20 seconds, the same TTL shape Darling's fleet-by-server read uses.
Pinned by two new tests proving the memo is served (not re-scanned) within the TTL and keyed
correctly per server.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
erikdarlingdata added a commit that referenced this pull request Sep 25, 2026
ViewerW2aLivePostgresTests was listed as a time-zone failure, but CI
showed it was a real regression in #4236. Keep only the two plan-shape
tests, say the time zone is the likely cause rather than a proven one,
and say a UTC rig does not excuse a failure.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
@erikdarlingdata erikdarlingdata mentioned this pull request Sep 25, 2026
2 of 3 tasks
erikdarlingdata added a commit that referenced this pull request Sep 25, 2026
* Lane rigs run in UTC, as CI does

initdb takes the machine's time zone, so local lane rigs ran in
America/New_York while CI runs in UTC. Three live tests failed only
on the local rigs on 2026-09-25, and lanes spent turns chasing them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

* Lane-orders UTC note: name only the two plan-shape tests

ViewerW2aLivePostgresTests was listed as a time-zone failure, but CI
showed it was a real regression in #4236. Keep only the two plan-shape
tests, say the time zone is the likely cause rather than a proven one,
and say a UTC rig does not excuse a failure.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
erikdarlingdata and others added 2 commits September 25, 2026 01:56
CollectionHealthByServerSql copied FleetCollectionHealthSql's
is_enabled/config_monitored_servers scope, but every per-server caller
(Overview cards, the badge, a per-server status-bar tab) needs that
server's own rows whether or not it is currently enabled or even
registered yet - the same as the raw per-server scans it replaced,
which never filtered by enable state.

Scope it to server_id <> 0 instead, matching the CAGG it composes with
and the service's own by-server fleet read. The status bar's
fleet-cumulative branch (no tab selected) still wants enabled-only, so
that filter moves to the one call site that needs it, against the
registry already in memory.

Adds a disabled-server and an unregistered-server case to
FleetCollectionHealthByServerLiveEqualityTests so this regresses
loudly next time.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
@erikdarlingdata erikdarlingdata changed the title DO NOT MERGE (viewer timers): Overview reads the collection-health rollup instead of scanning raw data every 30 s (#4226) Overview reads collection-health rollup instead of scanning raw data every 30 s (#4226) Sep 25, 2026
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 25, 2026 06:19
@erikdarlingdata
erikdarlingdata merged commit 96a4e6a into dev Sep 25, 2026
17 of 18 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/4226-viewer-timer-reads branch September 25, 2026 06:19
erikdarlingdata added a commit that referenced this pull request Sep 25, 2026
…and Service (#4228) (#4255)

* AG reads: two-step, floor-bounded newest-snapshot reads, shared across Viewer/Service (#4228)

The Viewer's and Service's AG replica/database reads each found "the newest snapshot per
server" by joining to an unbounded MAX(collection_time) GROUP BY server_id, which reads
every retained row for every server (2,803 ms / 441k buffers for 398 rows measured on a
production store). Move the reads into one shared PerformanceMonitor.Darling.Storage.DarlingAgStatesReader
(the Viewer has no route to the Service assembly), split each into a per-server newest-instant
step (ordered LATERAL descent, driven from servers, not scoped to is_enabled per the #4236
lesson) and a floor-bounded step two identical to the shipped statement plus collection_time >= $floor
on both arms. A staleness horizon (one chunk width) keeps a server whose AG rows stopped a day
ago from dragging the shared floor back; it is read on its own instead, so results stay
identical to today for every server, including a disabled or removed one.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

* G2 verify #4228: live row-equality and chunk-bound pins for the AG reader move

Adds DarlingAgStatesReaderLiveEqualityTests: ten seeded server shapes (six
current, one stopped 2 days ago, one stopped 2 hours ago, one disabled with
current rows, one removed from config_monitored_servers but still present in
servers) proving every entry point (viewer, /api/ag, /api/ag/count,
get_ag_health with and without a server filter) matches the pre-#4228
unbounded statements row for row, plus an EXPLAIN pin proving the shared
floor and the stale server's own point read both plan meaningfully fewer
chunks than the unbounded oracle. Verified once that the equality test
fails when the stale-server merge is neutralized, then reverted.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

* G3 verify #4228: give Lite's AG reads the same two-step newest-instant shape

G1's PR body dismissed Lite parity because "a Lite install monitors one server" - wrong,
Lite can monitor several, and LocalDataService.AvailabilityGroups.cs reads per server with
WHERE server_id = $1 and an unbounded per-server MAX(collection_time), which can scan that
one server's whole retained history regardless of fleet size.

Timed against a DuckDB table seeded at the real ScheduleManager defaults for both AG
collectors (1-minute frequency, 30-day retention, several servers interleaved by collection
tick): the old correlated-MAX query measured 47-93 ms, over the ~20 ms parity bar.

GetLatestAgReplicaStatesAsync / GetLatestAgDatabaseReplicaStatesAsync now use the same
two-step newest-instant split DarlingAgStatesReader uses: an ordered-descent step one
(ORDER BY collection_time DESC LIMIT 1), then a step two bound by that exact instant. Lite's
read is already scoped to one server per call, so step two uses a plain equality bound
instead of Darling's >= floor aggregate (no fleet-wide shared floor to protect). Query time
dropped to 2-4 ms in the same seed.

New AgAvailabilityGroupNewestSnapshotEqualityTests.cs pins the new code against the old SQL
text, row for row, across a tie (several rows sharing the newest collection_time) and a NULL
(a WSFC-quorum-loss row with NULL ag_name/replica_server_name that both old and new code must
drop). Proved once by hand that loosening the new bound to <= makes the test fail
(NODE-OLD leaks in); reverted, git diff confirms the reader matches this commit.

Filed #4262 for a separate, pre-existing finding: every LocalDataService call opens a fresh
DuckDB connection (37-50 ms alone), which is why the full method call still isn't under
20 ms even after this fix. That's orthogonal to #4228's unbounded-scan diagnosis and a much
larger, systemic change - out of scope for this lane.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant