Skip to content

Unify the collector-health ROW banding + fix the daily-collector STALE/FAILING false-positive (#1573) - #1578

Merged
erikdarlingdata merged 1 commit into
devfrom
feature/1573-collector-health-banding
Jul 18, 2026
Merged

erikdarlingdata merged 1 commit into
devfrom
feature/1573-collector-health-banding

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Closes #1573.

The bug

index_object_stats is a daily collector (CollectorScheduleDefaults cadence 1440 min) that succeeds every run, but the collector-health row banding used flat thresholds - STALE past 4h since last success, FAILING past 24h - that assumed a frequent (~1-min) collector. So between its once-a-day successes it always crossed 4h then 24h and read STALE, then FAILING. Because the Overview card's "Collectors: N failing" count reuses the same row banding (GetCollectorHealthCountsAsync counts HealthStatus == "FAILING"), a healthy daily collector also showed the fleet Overview a phantom failing collector.

The fix

Staleness thresholds are now relative to each collector's own cadence, with floors set to the original flat values:

  • FAILING when hours-since-success > max(24, 2 x freqHours) (1-min -> 24h floor, unchanged; daily -> 48h)
  • STALE when > max(4, 1.5 x freqHours) (1-min -> 4h floor, unchanged; daily -> 36h)

Every frequent collector bands byte-for-byte identically (the floors dominate up to a ~2.7h cadence for STALE / 12h for FAILING); only a slow collector relaxes. index_object_stats at 27h now reads HEALTHY. On-load config snapshots stay staleness-exempt; NEVER_RUN / NO_PERMISSIONS / WARNING are unchanged.

Unification (the #1573 core)

The banding was three byte-identical copies that nothing pinned together. All three now delegate to one pure classifier, CollectorHealthClassifier in PerformanceMonitor.Common (beside the #1562 per-server card classifier), which also centralizes the on-load set so the surfaces can't drift:

Consumer File
Lite grid Lite/Services/LocalDataService.CollectionHealth.cs
Darling viewer grid + Overview failing count Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs
Service get_collection_health MCP tool + web fleet failing-count reader Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs

Each surface keeps its own SQL / row model / display and resolves the collector's cadence from the shared CollectorScheduleDefaults default (no consumer cheaply has a per-install schedule override at the row level, so all three use the same default - which is itself the parity guarantee; an unknown collector name falls to the 24h/4h floors = the old flat behavior).

Tests

A mirrored decision-table test (CollectorHealthClassifierTests) in both Lite.Tests and Darling.Tests pins the whole table - frequent/hourly/daily/on-load/NEVER_RUN/NO_PERMISSIONS/WARNING and the threshold-helper math - including that index_object_stats' default 1440-min cadence yields HEALTHY at 27h (the exact bug), so the two SKUs can never drift.

Verification

  • Builds clean (0 errors): PerformanceMonitor.Common, Lite, Darling viewer, Darling service.
  • Full Darling.Tests: 2436 passed / 137 gated-live skipped / 0 failed.
  • Full Lite.Tests: 1428 passed / 0 failed / 0 skipped.
  • No existing collector-health banding test changed (they all use frequent / on-load collectors, unaffected by the floors).

🤖 Generated with Claude Code

…ve (#1573)

The per-collector NEVER_RUN/NO_PERMISSIONS/FAILING/STALE/WARNING/HEALTHY status
was three byte-identical copies -- Lite's grid, the Darling viewer's grid +
Overview "collectors failing" count, and the service's get_collection_health MCP
tool + web fleet failing-count reader -- with flat 4h-STALE / 24h-FAILING
thresholds that assumed a ~1-min collector. index_object_stats is a daily
(1440-min) collector that succeeds every run, so it always read STALE then
FAILING between successes: a real field false-positive that also inflated the
Overview failing count (which reuses the same row banding).

- Extract one pure classifier, CollectorHealthClassifier, into
  PerformanceMonitor.Common (beside the #1562 card classifier). It centralizes
  the on-load set and makes the staleness thresholds relative to each
  collector's cadence: FAILING > max(24, 2 x freqHours), STALE >
  max(4, 1.5 x freqHours). The floors keep every frequent collector byte-for-byte
  identical; only a slow collector relaxes (daily: stale 36h, fail 48h), so
  index_object_stats at 27h now reads HEALTHY.
- Refactor all three consumers onto it; each keeps its own SQL / row model /
  display and resolves the collector's cadence from the shared
  CollectorScheduleDefaults default (no consumer cheaply has a per-install
  override at the row level, so all three use the same default -- the parity
  guarantee).
- Mirrored decision-table tests in Lite.Tests + Darling.Tests pin the whole
  table, including index_object_stats HEALTHY at 27h (the exact bug) and every
  frequent / on-load / NEVER_RUN / NO_PERMISSIONS / WARNING case unchanged.

Closes #1573

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@erikdarlingdata
erikdarlingdata force-pushed the feature/1573-collector-health-banding branch from 3cd78a1 to f20f3c8 Compare July 18, 2026 21:32
@erikdarlingdata
erikdarlingdata merged commit af6c169 into dev Jul 18, 2026
5 checks passed
@erikdarlingdata
erikdarlingdata deleted the feature/1573-collector-health-banding branch July 18, 2026 21:52
erikdarlingdata added a commit that referenced this pull request Jul 18, 2026
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pull Bot pushed a commit to ehtick/PerformanceMonitor that referenced this pull request Jul 22, 2026
The [erikdarlingdata#1578] and [erikdarlingdata#1536] reference definitions were glued onto one line
(pre-existing, unrelated to erikdarlingdata#1592), so [erikdarlingdata#1536] never resolved. Split
them while touching this file for the erikdarlingdata#1592 entry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant