You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
On-load collectors are now staleness-exempt forever in CollectorHealthClassifier, even though they recapture daily #4000
Found while fixing #3929/#3930. CollectorHealthClassifier.Classify (PerformanceMonitor.Common/ServerHealthBands.cs)
permanently exempts an on-load collector from staleness bands (STOPPED/FAILING/STALE), banding it by
failure/abandon rate only. Now that on-load collectors also recapture daily (#3929/#3930), that exemption
masks exactly the kind of regression those two issues were about.
Why
if(isOnLoad){// On-load collectors are staleness-exempt and banded by rate only ...returnfailureRatePercent>WarningFailureRatePercent||abandonRatePercent>WarningAbandonRatePercent?Warning:Healthy;}
This was correct when an on-load collector's only capture was the last connect - it really could be weeks
old on a healthy server, so "hours since last success" was meaningless as a health signal. After #3929/#3930,
these collectors run at least once a day. A collector whose daily reschedule silently breaks (a bug in the
NextDue seeding, an exception swallowed somewhere before the run) but whose historical runs were all
successes would still read HEALTHY forever under the current exemption - the exact blind spot that let #3930's field defect go unnoticed.
Fix shape (needs a decision, so not folded into #3929/#3930)
Once on-load collectors have a real cadence
(CollectorScheduleDefaults.EffectiveRecurringIntervalMinutes), Classify could apply the same FailingThresholdHours/StaleThresholdHours logic to them that scheduled collectors get, using that
effective interval instead of the raw 0. Open questions this needs a ruling on:
Whether to widen the STALE/FAILING thresholds specifically for the on-load tier (a day's cadence is much
coarser than the per-minute collectors those thresholds were tuned against), or reuse the existing
day-or-twice-the-interval math as-is.
Every SQL call site that currently passes isOnLoad/computes frequencyMinutes for these five collectors
(the fleet rollup, the per-server health read, and their live tests) would need auditing - isOnLoad
might become a dead parameter, or the threshold math might need a distinct on-load-aware path.
Whether removing the "on-load collectors never go STALE" property is a user-visible change worth a
CHANGELOG entry of its own (a fleet that has been silently STOPPED for months, unnoticed because it read
HEALTHY, would newly show up as STALE/FAILING/STOPPED after this ships).
This is wide-blast-radius (the classifier is consumed by the fleet health rollup, the per-server viewer page,
and several live/pin tests) and needs a threshold ruling before it's mechanical, so it was parked rather than
attempted in #3929/#3930's PR.
Ruled (Erik, 2026-09-23): no new threshold. Classify the on-load collectors like any scheduled collector, on the effective 1440-minute cadence (CollectorScheduleDefaults.EffectiveRecurringIntervalMinutes) with the existing staleness bands. A daily config capture that keeps failing should read stale.
Activity
erikdarlingdata commentedon Sep 23, 2026
Ruled (Erik, 2026-09-23): no new threshold. Classify the on-load collectors like any scheduled collector, on the effective 1440-minute cadence (
CollectorScheduleDefaults.EffectiveRecurringIntervalMinutes) with the existing staleness bands. A daily config capture that keeps failing should read stale.Only a catalog on-load collector bands on the daily cadence; a name t…
On-load collectors band on the same staleness ladder as any other col…
erikdarlingdata commentedon Sep 23, 2026
Closed by the watcher: delivered in PR #4029, merged to dev.