Unify the collector-health ROW banding + fix the daily-collector STALE/FAILING false-positive (#1573) - #1578
Merged
Conversation
…ve (#1573) The per-collector NEVER_RUN/NO_PERMISSIONS/FAILING/STALE/WARNING/HEALTHY status was three byte-identical copies -- Lite's grid, the Darling viewer's grid + Overview "collectors failing" count, and the service's get_collection_health MCP tool + web fleet failing-count reader -- with flat 4h-STALE / 24h-FAILING thresholds that assumed a ~1-min collector. index_object_stats is a daily (1440-min) collector that succeeds every run, so it always read STALE then FAILING between successes: a real field false-positive that also inflated the Overview failing count (which reuses the same row banding). - Extract one pure classifier, CollectorHealthClassifier, into PerformanceMonitor.Common (beside the #1562 card classifier). It centralizes the on-load set and makes the staleness thresholds relative to each collector's cadence: FAILING > max(24, 2 x freqHours), STALE > max(4, 1.5 x freqHours). The floors keep every frequent collector byte-for-byte identical; only a slow collector relaxes (daily: stale 36h, fail 48h), so index_object_stats at 27h now reads HEALTHY. - Refactor all three consumers onto it; each keeps its own SQL / row model / display and resolves the collector's cadence from the shared CollectorScheduleDefaults default (no consumer cheaply has a per-install override at the row level, so all three use the same default -- the parity guarantee). - Mirrored decision-table tests in Lite.Tests + Darling.Tests pin the whole table, including index_object_stats HEALTHY at 27h (the exact bug) and every frequent / on-load / NEVER_RUN / NO_PERMISSIONS / WARNING case unchanged. Closes #1573 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
erikdarlingdata
force-pushed
the
feature/1573-collector-health-banding
branch
from
July 18, 2026 21:32
3cd78a1 to
f20f3c8
Compare
erikdarlingdata
added a commit
that referenced
this pull request
Jul 18, 2026
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pull Bot
pushed a commit
to ehtick/PerformanceMonitor
that referenced
this pull request
Jul 22, 2026
The [erikdarlingdata#1578] and [erikdarlingdata#1536] reference definitions were glued onto one line (pre-existing, unrelated to erikdarlingdata#1592), so [erikdarlingdata#1536] never resolved. Split them while touching this file for the erikdarlingdata#1592 entry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 27, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1573.
The bug
index_object_statsis a daily collector (CollectorScheduleDefaultscadence 1440 min) that succeeds every run, but the collector-health row banding used flat thresholds - STALE past 4h since last success, FAILING past 24h - that assumed a frequent (~1-min) collector. So between its once-a-day successes it always crossed 4h then 24h and read STALE, then FAILING. Because the Overview card's "Collectors: N failing" count reuses the same row banding (GetCollectorHealthCountsAsynccountsHealthStatus == "FAILING"), a healthy daily collector also showed the fleet Overview a phantom failing collector.The fix
Staleness thresholds are now relative to each collector's own cadence, with floors set to the original flat values:
> max(24, 2 x freqHours)(1-min -> 24h floor, unchanged; daily -> 48h)> max(4, 1.5 x freqHours)(1-min -> 4h floor, unchanged; daily -> 36h)Every frequent collector bands byte-for-byte identically (the floors dominate up to a ~2.7h cadence for STALE / 12h for FAILING); only a slow collector relaxes.
index_object_statsat 27h now reads HEALTHY. On-load config snapshots stay staleness-exempt; NEVER_RUN / NO_PERMISSIONS / WARNING are unchanged.Unification (the #1573 core)
The banding was three byte-identical copies that nothing pinned together. All three now delegate to one pure classifier,
CollectorHealthClassifierinPerformanceMonitor.Common(beside the #1562 per-server card classifier), which also centralizes the on-load set so the surfaces can't drift:Lite/Services/LocalDataService.CollectionHealth.csDarling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.csget_collection_healthMCP tool + web fleet failing-count readerDarling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.csEach surface keeps its own SQL / row model / display and resolves the collector's cadence from the shared
CollectorScheduleDefaultsdefault (no consumer cheaply has a per-install schedule override at the row level, so all three use the same default - which is itself the parity guarantee; an unknown collector name falls to the 24h/4h floors = the old flat behavior).Tests
A mirrored decision-table test (
CollectorHealthClassifierTests) in both Lite.Tests and Darling.Tests pins the whole table - frequent/hourly/daily/on-load/NEVER_RUN/NO_PERMISSIONS/WARNING and the threshold-helper math - including thatindex_object_stats' default 1440-min cadence yields HEALTHY at 27h (the exact bug), so the two SKUs can never drift.Verification
PerformanceMonitor.Common, Lite, Darling viewer, Darling service.🤖 Generated with Claude Code