Skip to content

On-load collectors are now staleness-exempt forever in CollectorHealthClassifier, even though they recapture daily #4000

Closed

Description

@erikdarlingdata

Summary

Found while fixing #3929/#3930. CollectorHealthClassifier.Classify (PerformanceMonitor.Common/ServerHealthBands.cs)
permanently exempts an on-load collector from staleness bands (STOPPED/FAILING/STALE), banding it by
failure/abandon rate only. Now that on-load collectors also recapture daily (#3929/#3930), that exemption
masks exactly the kind of regression those two issues were about.

Why

if (isOnLoad)
{
    // On-load collectors are staleness-exempt and banded by rate only ...
    return failureRatePercent > WarningFailureRatePercent || abandonRatePercent > WarningAbandonRatePercent
        ? Warning
        : Healthy;
}

This was correct when an on-load collector's only capture was the last connect - it really could be weeks
old on a healthy server, so "hours since last success" was meaningless as a health signal. After #3929/#3930,
these collectors run at least once a day. A collector whose daily reschedule silently breaks (a bug in the
NextDue seeding, an exception swallowed somewhere before the run) but whose historical runs were all
successes would still read HEALTHY forever under the current exemption - the exact blind spot that let
#3930's field defect go unnoticed.

Fix shape (needs a decision, so not folded into #3929/#3930)

Once on-load collectors have a real cadence
(CollectorScheduleDefaults.EffectiveRecurringIntervalMinutes), Classify could apply the same
FailingThresholdHours/StaleThresholdHours logic to them that scheduled collectors get, using that
effective interval instead of the raw 0. Open questions this needs a ruling on:

  1. Whether to widen the STALE/FAILING thresholds specifically for the on-load tier (a day's cadence is much
    coarser than the per-minute collectors those thresholds were tuned against), or reuse the existing
    day-or-twice-the-interval math as-is.
  2. Every SQL call site that currently passes isOnLoad/computes frequencyMinutes for these five collectors
    (the fleet rollup, the per-server health read, and their live tests) would need auditing - isOnLoad
    might become a dead parameter, or the threshold math might need a distinct on-load-aware path.
  3. Whether removing the "on-load collectors never go STALE" property is a user-visible change worth a
    CHANGELOG entry of its own (a fleet that has been silently STOPPED for months, unnoticed because it read
    HEALTHY, would newly show up as STALE/FAILING/STOPPED after this ships).

This is wide-blast-radius (the classifier is consumed by the fleet health rollup, the per-server viewer page,
and several live/pin tests) and needs a threshold ruling before it's mechanical, so it was parked rather than
attempted in #3929/#3930's PR.

Activity

  1. erikdarlingdata commented on Sep 23, 2026

    @erikdarlingdata
    OwnerAuthor

    Ruled (Erik, 2026-09-23): no new threshold. Classify the on-load collectors like any scheduled collector, on the effective 1440-minute cadence (CollectorScheduleDefaults.EffectiveRecurringIntervalMinutes) with the existing staleness bands. A daily config capture that keeps failing should read stale.

  2. erikdarlingdata commented on Sep 23, 2026

    @erikdarlingdata
    OwnerAuthor

    Closed by the watcher: delivered in PR #4029, merged to dev.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions