Skip to content

Revert cadence-aware fleet staleness (#3268): it deletes the amber band fleet-wide - #3270

Merged
erikdarlingdata merged 1 commit into
devfrom
revert/3236-fleet-stale-cadence
Sep 10, 2026
Merged

erikdarlingdata merged 1 commit into
devfrom
revert/3236-fleet-stale-cadence

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Reverts #3268 (and its CHANGELOG entry from #3269). Reopens #3236.

Why

A field review against the pgmon store found #3268 is a net regression, verified from source before this revert:

The amber band becomes unreachable fleet-wide. ServerHealthClassifier.EffectiveStaleThreshold takes the max over a server's enabled collectors, and a current collector contributes ~its own cadence (StaleThresholdFor = 2x cadence). Real PostgreSQL targets carry 9 hourly + 4 daily collectors, so the daily ones alone hold the per-server stale cutoff at >=24h (measured 40h15m live). But ClassifyFreshness checks the flat 30-minute Offline band first, then the stale threshold. With the stale cutoff at 40h > 30min, the amber Stale window [stale, offline) is empty: a server reads Fresh until 30 minutes fully dark, then jumps to Offline. A multi-minute collection gap — the exact thing #3236 was filed about — now shows green. That fails toward the quiet label, which is the wrong direction for a staleness signal. The change's own doc comment admits the no-amber outcome but scopes it to "only slow-cadence collectors enabled"; max makes it "any slow-cadence collector present," which is every real target.

The original diagnosis was wrong. FleetLastCollectionSql is MAX(collection_time) over all collectors, so on a healthy server the ~70-second collectors keep it current and the flat 2-minute threshold never trips spuriously. The 14-target amber event in #3236 was a real ~4-minute coverage collapse during the nightly service restart — a true positive the band reported correctly — not the false positive the fix assumed.

What's reverted

All 20 code/test files from #3268 (the EffectiveStaleThreshold path across the fleet overview, list_servers, the viewer cards/dots, and Lite's Overview card) plus the 2-line CHANGELOG entry. CHANGELOG census verified CRLF-only, entry removed. Pre-#3268 flat-threshold behavior is restored.

#3236 is reopened for re-diagnosis with fleet measurement in the loop; any real fix must pin that the amber band stays reachable (EffectiveStaleThreshold(...) < OfflineThreshold) over a representative collector mix.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Dqza1QNL4YCFS1MpzezERn

Reverts PR #3268 and its CHANGELOG entry (#3269). Field review against the pgmon store showed the change is a net regression: EffectiveStaleThreshold takes the max over a server's collectors, so any enabled slow-cadence collector (these PG targets carry 9 hourly + 4 daily) pushes the per-server stale cutoff to tens of hours, while ClassifyFreshness checks the flat 30-minute Offline band first. That leaves the amber Stale band unreachable on every real target: a server reads Fresh until 30 minutes fully dark, then jumps to Offline, so a multi-minute collection gap now shows green. The original #3236 signature was a true positive (a real coverage collapse during the nightly restart), not the false positive the fix assumed, because last_collection is MAX over all collectors and the ~70s collectors keep it current on a healthy server. Reopening #3236 to re-diagnose with fleet measurement in the loop.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dqza1QNL4YCFS1MpzezERn
@claude

claude Bot commented Sep 10, 2026

Copy link
Copy Markdown

Reviewed. This is a clean, symmetric revert of #3268 across PerformanceMonitor.Common, Darling (service + viewer + tests), and Lite (+ tests), plus the CHANGELOG entry.

Verification performed:

  • Grepped the full tree for every symbol Make the server stale band honor each collector's own cadence (#3236) #3268 introduced (CollectorCadenceSample, EffectiveStaleThreshold, StaleThresholdMinutes/stale_threshold_minutes, EffectiveCadenceLookup, FleetCollectorCadenceSql, FleetScheduleOverridesSql, StaleThresholdFor, StaleCadenceGraceMultiplier, ResolveEffectiveCadencesForFreshness, GetCollectorCadenceSamplesAsync, CollectorCadenceSamplesSql) — no leftover references anywhere in the tree. No dangling using PerformanceMonitor.Common; or orphaned helper methods.
  • CollectorScheduleOverlay/ScheduleOverride (used by the diff in ViewerDataService.ServerStatus.cs/CollectorScheduleEditorWindow) predate Make the server stale band honor each collector's own cadence (#3236) #3268 and remain used elsewhere (schedule editor, StoreConfigProvider), so they're correctly left in place rather than reverted.
  • CHANGELOG: both the ### Fixed bullet and its [#3236] link-reference footer are removed together; no dangling reference.
  • Lite/Darling parity: the revert removes the feature symmetrically from both apps (Lite's LocalDataService/MainWindow.AlertEngine/MainWindow.xaml.cs and Darling's service reader/MCP tools/viewer), so parity is preserved — neither app is left with orphaned cadence-aware logic the other lacks.
  • Spot-checked the PR body's math (EffectiveStaleThreshold = max over collectors of LastRunUtc + StaleThresholdFor(freq) - lastCollection, with StaleThresholdFor = max(2×freq, flat floor)): a daily collector (1440 min) on schedule pins the per-server threshold at ~48h, and since ClassifyFreshness checked the flat 30-minute Offline band before the (now enormous) stale threshold, the amber Stale band was indeed unreachable on any server with a daily-cadence collector enabled — which is every real target carrying the shipped daily collectors. The regression claim checks out from the code as written in Make the server stale band honor each collector's own cadence (#3236) #3268.

No correctness, parity, or security issues found in the revert itself. No T-SQL changed (pure C#/CHANGELOG diff), so the T-SQL style conventions don't apply here.

@erikdarlingdata
erikdarlingdata merged commit 778a355 into dev Sep 10, 2026
8 checks passed
@erikdarlingdata
erikdarlingdata deleted the revert/3236-fleet-stale-cadence branch September 10, 2026 07:36
@erikdarlingdata erikdarlingdata mentioned this pull request Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant