Repository navigation
Overview reads collection-health rollup instead of scanning raw data every 30 s (#4226) - #4236
Merged
Merged
Conversation
…er tick, not 43 raw scans (#4226) The Overview loader called GetCollectionHealthAsync (raw 7-day collection_log) once per server every 30 s tick, plus the status bar ran a second raw fleet scan on the same tick. Moved the service's collection_health_hourly guard and composer (DarlingFleetReader) into Storage as CollectionHealthRollupSupport so the viewer can share it; DarlingFleetReader now delegates to it (same names, same values, existing pins unchanged). Added a per-server-breakdown fleet read (ViewerDataService.GetFleetCollectionHealthByServerAsync), memoized 20s, that both the Overview cards and the status bar now read instead of the raw scans. The ranked per-server CollectionHealthSql stays as-is for the Collection Health tab, which needs the exemplar messages this fleet read does not carry. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
…llup read (#4226) Every server-tab refresh (1 min auto, plus tab activation) ran PermissionDeniedCollectorCountSql, its own 7-day raw collection_log scan per open tab. GetFleetCollectionHealthByServerAsync already carries permission_denied_count per (server, collector) over the same window, so the badge now filters that shared, memoized read instead. Pinned by a new source-text test, proven to fail on the old raw-scan body before the fix landed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
… reads (#4226) Per-server numbers and badge counts from the rollup-composed FleetCollectionHealthByServerSql must equal CollectionHealthSql (Overview's old per-server scan) and PermissionDeniedCollectorCountSql (the badge's old scan) over a window with a partial leading hour and rows in the still-open current hour, so both the raw head slice and the materialized union are exercised. Verified the assertions actually discriminate by temporarily comparing against the wrong server_id and watching it fail, then reverted. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
…0 ms bar (#4226) Lite's Overview/status-bar equivalent (RemoteCollectorService.GetHealthSummary) is already in-memory and never touches DuckDB, so #4226's rollup fix has no Lite counterpart there. The one DB-backed read every server-tab refresh still runs is GetPermissionDeniedCollectorCountAsync, measured at 46-58 ms on a seeded 20-collector/7-day/1-minute-cadence DuckDB (Lite's realistic per-server scale) -- over the lane brief's ~50 ms bar even without a fleet to amortize across. It now memoizes per server for 20 seconds, the same TTL shape Darling's fleet-by-server read uses. Pinned by two new tests proving the memo is served (not re-scanned) within the TTL and keyed correctly per server. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
erikdarlingdata
added a commit
that referenced
this pull request
Sep 25, 2026
ViewerW2aLivePostgresTests was listed as a time-zone failure, but CI showed it was a real regression in #4236. Keep only the two plan-shape tests, say the time zone is the likely cause rather than a proven one, and say a UTC rig does not excuse a failure. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
2 of 3 tasks
erikdarlingdata
added a commit
that referenced
this pull request
Sep 25, 2026
* Lane rigs run in UTC, as CI does initdb takes the machine's time zone, so local lane rigs ran in America/New_York while CI runs in UTC. Three live tests failed only on the local rigs on 2026-09-25, and lanes spent turns chasing them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3 * Lane-orders UTC note: name only the two plan-shape tests ViewerW2aLivePostgresTests was listed as a time-zone failure, but CI showed it was a real regression in #4236. Keep only the two plan-shape tests, say the time zone is the likely cause rather than a proven one, and say a UTC rig does not excuse a failure. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3 --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
CollectionHealthByServerSql copied FleetCollectionHealthSql's is_enabled/config_monitored_servers scope, but every per-server caller (Overview cards, the badge, a per-server status-bar tab) needs that server's own rows whether or not it is currently enabled or even registered yet - the same as the raw per-server scans it replaced, which never filtered by enable state. Scope it to server_id <> 0 instead, matching the CAGG it composes with and the service's own by-server fleet read. The status bar's fleet-cumulative branch (no tab selected) still wants enabled-only, so that filter moves to the one call site that needs it, against the registry already in memory. Adds a disabled-server and an unregistered-server case to FleetCollectionHealthByServerLiveEqualityTests so this regresses loudly next time. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
erikdarlingdata
marked this pull request as ready for review
September 25, 2026 06:19
This was referenced Sep 25, 2026
erikdarlingdata
added a commit
that referenced
this pull request
Sep 25, 2026
…and Service (#4228) (#4255) * AG reads: two-step, floor-bounded newest-snapshot reads, shared across Viewer/Service (#4228) The Viewer's and Service's AG replica/database reads each found "the newest snapshot per server" by joining to an unbounded MAX(collection_time) GROUP BY server_id, which reads every retained row for every server (2,803 ms / 441k buffers for 398 rows measured on a production store). Move the reads into one shared PerformanceMonitor.Darling.Storage.DarlingAgStatesReader (the Viewer has no route to the Service assembly), split each into a per-server newest-instant step (ordered LATERAL descent, driven from servers, not scoped to is_enabled per the #4236 lesson) and a floor-bounded step two identical to the shipped statement plus collection_time >= $floor on both arms. A staleness horizon (one chunk width) keeps a server whose AG rows stopped a day ago from dragging the shared floor back; it is read on its own instead, so results stay identical to today for every server, including a disabled or removed one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3 * G2 verify #4228: live row-equality and chunk-bound pins for the AG reader move Adds DarlingAgStatesReaderLiveEqualityTests: ten seeded server shapes (six current, one stopped 2 days ago, one stopped 2 hours ago, one disabled with current rows, one removed from config_monitored_servers but still present in servers) proving every entry point (viewer, /api/ag, /api/ag/count, get_ag_health with and without a server filter) matches the pre-#4228 unbounded statements row for row, plus an EXPLAIN pin proving the shared floor and the stale server's own point read both plan meaningfully fewer chunks than the unbounded oracle. Verified once that the equality test fails when the stale-server merge is neutralized, then reverted. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3 * G3 verify #4228: give Lite's AG reads the same two-step newest-instant shape G1's PR body dismissed Lite parity because "a Lite install monitors one server" - wrong, Lite can monitor several, and LocalDataService.AvailabilityGroups.cs reads per server with WHERE server_id = $1 and an unbounded per-server MAX(collection_time), which can scan that one server's whole retained history regardless of fleet size. Timed against a DuckDB table seeded at the real ScheduleManager defaults for both AG collectors (1-minute frequency, 30-day retention, several servers interleaved by collection tick): the old correlated-MAX query measured 47-93 ms, over the ~20 ms parity bar. GetLatestAgReplicaStatesAsync / GetLatestAgDatabaseReplicaStatesAsync now use the same two-step newest-instant split DarlingAgStatesReader uses: an ordered-descent step one (ORDER BY collection_time DESC LIMIT 1), then a step two bound by that exact instant. Lite's read is already scoped to one server per call, so step two uses a plain equality bound instead of Darling's >= floor aggregate (no fleet-wide shared floor to protect). Query time dropped to 2-4 ms in the same seed. New AgAvailabilityGroupNewestSnapshotEqualityTests.cs pins the new code against the old SQL text, row for row, across a tie (several rows sharing the newest collection_time) and a NULL (a WSFC-quorum-loss row with NULL ag_name/replica_server_name that both old and new code must drop). Proved once by hand that loosening the new bound to <= makes the test fail (NODE-OLD leaks in); reverted, git diff confirms the reader matches this commit. Filed #4262 for a separate, pre-existing finding: every LocalDataService call opens a fresh DuckDB connection (37-50 ms alone), which is why the full method call still isn't under 20 ms even after this fix. That's orthogonal to #4228's unbounded-scan diagnosis and a much larger, systemic change - out of scope for this lane. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3 --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4226.
Regression fixed
ViewerW2aLivePostgresTests.ServerSummary_ReadsEnrichedThreadsMemoryBlockingCollectors_AgainstDevPostgresfailedat
Darling/Darling.Tests/ViewerW2aTests.cs:1008(HealthyCollectorCountexpected 1, got 0). Dev's CI passesthis test, so this PR broke it.
Cause
FleetCollectionHealthByServerSqlis the new shared read. The Overview cards, the status bar and theserver-tab badge all now call it through
GetFleetCollectionHealthByServerAsync. It scoped its rows toserver_id IN (SELECT server_id FROM config_monitored_servers WHERE is_enabled). That scope was copied fromFleetCollectionHealthSql, the viewer's older, unrelated fleet-cumulative total.The two statements do not share the same scope. Every caller of the new by-server read looks up one server's own
rows. The old raw per-server scans it replaced (
CollectionHealthSql,PermissionDeniedCollectorCountSql)never filtered by enable state. They read by
server_idalone.The test's
EnrichServerIdis a synthetic id with noconfig_monitored_serversrow at all, so theis_enabledsubquery dropped it entirely. That is also a real production gap, not just a test artifact. Two cases hit it. One: a disabled server whose card, badge and tab need its last 7-day history. Two: the
bootstrap window, where
config_monitored_serversis empty before the config store is seeded.Both were confirmed by running
PgMigrations.MigrateAsyncalone on a fresh rig database.collect.collection_health_hourlydoes not exist yet, so the raw fallback runs, not the composed read.config.config_monitored_servershas 0 rows, so theis_enabledscope drops every server, not just thetest's synthetic one.
The right precedent was sitting right next to the moved code. The service's own by-server fleet read
(
DarlingFleetReader.FleetCollectionHealthSql, nowCollectionHealthRollupSupport's twin) scopes toserver_id <> 0only. The continuous aggregate it composes with (CreateCollectionHealthHourlySql) does thesame. The service's own consumer (
GetFleetOverviewAsync) reads the by-server dictionary unscoped, and lets itsown caller decide which server ids to keep. This fix adopts that same pattern for the viewer.
Fix
FleetCollectionHealthByServerSqlnow scopes toserver_id <> 0only, matching the CAGG and the service'sread exactly. Overview cards and the badge key their lookup by one
server_id. A per-server status-bar tabdoes too. Each now gets that server's real rows regardless of enable or registration state, matching the old
raw scans exactly.
only. That is the old
FleetCollectionHealthSqlbehavior, where a removed server's aged-out rows must notread as erroring. The filter now lives in
UpdateCollectorHealthTextAsyncitself, against the fleet listalready in memory (
_fleet.All.Where(s => s.IsEnabled)). It no longer lives inside the shared SQL everyother caller also paid for.
FleetCollectionHealthByServerSql's doc comment and theOverviewCollectionHealthRollupRoutingTestspin (renamed, re-asserted on
server_id <> 0) to state the corrected scope and why.FleetCollectionHealthByServerLiveEqualityTests. The composed and raw fleet-by-server reads are now checkedagainst the old raw per-server scans for both, alongside the existing three enabled servers.
Lite was checked for the same class of bug. Its only change in this PR memoizes its own existing per-server
GetPermissionDeniedCollectorCountAsyncscan. That scan was neveris_enabled-scoped. Lite has no fleet-wideread to share. No parity fix was needed there.
What changed
Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs:FleetCollectionHealthByServerSqlscoped to
server_id <> 0, doc comment corrected.Darling/PerformanceMonitor.Darling.Viewer/MainWindow.ServerManagement.cs: the status bar's fleet-cumulativebranch now filters to enabled servers itself.
Darling/Darling.Tests/OverviewCollectionHealthRollupRoutingTests.cs: pin renamed and re-targeted at thecorrected scope.
Darling/Darling.Tests/FleetCollectionHealthByServerLiveEqualityTests.cs: added a disabled-server and anunregistered-server case.
Test plan
ServerSummary_ReadsEnrichedThreadsMemoryBlockingCollectors_AgainstDevPostgresalone: failed before thefix (
HealthyCollectorCount0, expected 1), passes after.FleetCollectionHealthByServerLiveEqualityTests,OverviewCollectionHealthRollupRoutingTests,CollectionHealthAggregateTests: 18/18 passed, including the two new regression cases.ViewerW2aTests.cs(ViewerOverviewSqlTests,ViewerServerSummaryDisplayTests,ViewerAlertHistoryW2aSqlTests,ViewerW2aLivePostgresTests) plusFreshStoreWatermarkTests: 84/84 passed.PermissionDeniedBadgeMemoTests: 2/2 passed.McpPayloadContractCensusTests,StartupCommandTimeoutTests,StorageCommandTimeoutTests,AlertReadFailureSurfaceTests,DocCommentHygiene,LivePostgresCollectionHygieneTests,LiveCleanupConversionRatchetTests,ReadmeDerivedCountPinTests):154 + 5 passed.
origin/dev(clean, no conflicts) and ran the fullDarling.Testssuite once against a freshlydropped and recreated
darlingtest. Result: 13811 total, 1 failed, 47 skipped, 1 not run, 599s.The one failure was
ForcePlanFailuresAccessPathTests.TheShippedRead_PlansAsAnIndexOnlyScanOnTheCoveringIndex_AgainstDevPostgres.It is an unrelated query-store-stats planner-shape test with zero references to collection health in its
file. It passed when re-run alone right after. It is full-suite-only, not caused by this diff. It did not
reproduce in isolation to file cleanly. It is worth a census if it recurs.
CHANGELOG entry
Generated with Claude Code
https://claude.ai/code/session_01FVjn4PBJN71NQXdFo6ZxNQ