Skip to content

On-load collectors band on the same staleness ladder as any other collector (#4000) - #4029

Merged
erikdarlingdata merged 2 commits into
devfrom
fix/4000-onload-staleness-bands
Sep 23, 2026
Merged

erikdarlingdata merged 2 commits into
devfrom
fix/4000-onload-staleness-bands

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Closes #4000.

Why

CollectorHealthClassifier.Classify (PerformanceMonitor.Common/ServerHealthBands.cs) permanently
exempted an on-load collector (server_config, database_config, database_scoped_config,
trace_flags, server_properties) from every staleness band, reading it by failure/abandon rate
alone. That was correct when an on-load collector's only capture was the last connect. Since
#3929/#3930 gave every one of them a real daily reschedule (CollectorScheduleDefaults.OnLoadRecaptureMinutes
= 1440) on top of the connect-time capture, "hours since last success" became a meaningful signal
again - but the exemption stayed, so a daily reschedule that silently breaks (a bug in the NextDue
seeding, an exception swallowed before the run) whose historical runs were all successes still read
HEALTHY forever. That is the exact blind spot that let #3930's field defect go unnoticed.

Erik's ruling (posted on the issue): no new thresholds or settings. Classify on-load collectors like
any scheduled collector, on the effective 1440-minute cadence
(CollectorScheduleDefaults.EffectiveRecurringIntervalMinutes), with the existing staleness bands.

What changes

  • CollectorHealthClassifier.Classify - removed the isOnLoad parameter and the exemption branch
    entirely. The method is now a pure function of run counts and cadence; on-load-ness is no longer a
    distinct input, only a distinct CADENCE the caller resolves before calling in. Parameter count: 10 -> 9.
  • Three FrequencyMinutes properties (one per surface) now route the catalog lookup through
    CollectorScheduleDefaults.EffectiveRecurringIntervalMinutes, so a 0 default (on-load, or a
    collector no longer in the catalog) resolves to the 1440-minute daily cadence instead of the 24h/4h
    floors a raw 0 used to fall to:
    • Lite/Services/LocalDataService.CollectionHealth.cs (CollectorHealthRow)
    • Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs (CollectorHealthRow)
    • Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs (CollectorHealth)
  • Fleet rollup covered for free: DarlingFleetReader.cs constructs the SAME CollectorHealth type
    from FleetCollectionHealthSql's rows, so the fix reaches the fleet overview without a separate change.
    Confirmed by reading DarlingFleetReader.cs - no direct Classify call there to update.
  • IsOnLoadCollector/OnLoadCollectorNames stay - still consumed by ProducedThenStopped's "three
    tab opens are not three consecutive cycles" exclusion, which is a different concept (consecutive-run
    semantics, not staleness) and out of this issue's scope.
  • Doc comments updated everywhere the old exemption was described as permanent (the IsOnLoadCollector
    summary, Classify's order-of-bands summary, and one stale cross-reference in FormatOutputFinding's
    remarks that cited Classify's isOnLoad pattern by name).

Test plan

  • Darling.Tests and Lite.Tests both build with 0 Warning(s).
  • Decision-table pins (Classify_BandsTheDecisionTable, both SKUs) updated: isOnLoad column
    removed from every row; the old "on-load exemption" demonstration rows (500h dark -> Healthy/Warning)
    now assert the corrected Stopped outcome instead of being deleted, so the removed behavior stays
    pinned as a regression rather than silently disappearing.
  • New regression coverage, both SKUs, mirroring the existing index_object_stats boundary facts
    but through a REAL on-load collector name (server_config) so the catalog resolution
    (EffectiveRecurringIntervalMinutes) and the ladder are exercised together, not just the pure
    Classify function:
    - 27h since last success -> HEALTHY (well inside the new 36h stale line)
    - 37h -> STALE (was HEALTHY forever before this fix)
    - 49h -> STOPPED (was HEALTHY forever before this fix - the A Darling server connected for more than 30 days loses its config facts: on-load snapshots age out and nothing re-collects them #3930-shaped blind spot)
    - Darling additionally checks both CollectorHealthRow (viewer) and CollectorHealth (MCP
    service) at 49h in one fact, matching the suite's existing "on both row types" discipline.
  • Pin tests caught and fixed deliberately, not weakened: TheBandingSignature_TakesNoOutputAndNoDenialCurrency
    (CollectionOutputBesideCostTests.cs, both SKUs) reflects Classify's exact parameter list by
    name/count - updated from 10 to 9 parameters with the reason (On-load collectors are now staleness-exempt forever in CollectorHealthClassifier, even though they recapture daily #4000 removed isOnLoad) in the
    comment, per this repo's "update a pin deliberately, never weaken one" rule.
  • Targeted class runs, both executables (-class per touched class, not by file - some of these
    files hold exactly one class each, verified by grep before running):
    - Darling.Tests: CollectorHealthClassifierTests 72/72, {RegressedFromProductiveTests, ProductiveZeroBandingTests, AbandonedRunEraInvariantReadTests, CollectionOutputBesideCostTests}
    49/49 (1 skipped, pre-existing/unrelated), DocCommentHygiene* 77/77 (doc comment edits didn't
    trip the hygiene pin).
    - Lite.Tests: {CollectorHealthClassifierTests, RegressedFromProductiveTests, CollectionOutputBesideCostTests} 90/90.
    - All: 0 failed.
  • Full suite NOT run locally (Darling.Tests.exe / Lite.Tests.exe with no filter) - flagging
    for the coordinator rather than silently skipping: this lane hit a hard 13:10Z deadline shared
    with a second issue (MCP get_trace_flags and the web viewer share a narrower version of #3929's stale-flag defect #3999) on its own branch, and every class touched by this diff (by grep,
    not by filename) is covered above. CI will run the full suite on this PR; please treat that as
    the gate before arming auto-merge rather than assuming my local run covered it.

What the coordinator should double-check

  • The parameter-count pin change (10 -> 9) in CollectionOutputBesideCostTests.cs is a deliberate,
    reasoned update (see comment), not a weakening - it now asserts a SMALLER number, which a careless
    "make the test pass" edit could get backwards (e.g. by leaving 10 and hoping for a compile error
    instead of a runtime pin failure). Worth a second look.
  • Lite parity: confirmed by reading Lite/Services/LocalDataService.CollectionHealth.cs alongside both
    Darling surfaces - all three FrequencyMinutes properties had byte-for-byte identical logic before
    this change and now have byte-for-byte identical fixes.
  • No store migration - this is pure C# classifier/read logic, no schema change.
  • CI's full-suite run is the one gate this PR has not seen locally (see Test plan above).

🤖 Generated with Claude Code

https://claude.ai/code/session_01GdmA4ND1wLSqA91ax1m4xv

…lector, on their effective daily cadence (#4000)

CollectorHealthClassifier.Classify no longer exempts on-load collectors (server_config,
database_config, database_scoped_config, trace_flags, server_properties) from STOPPED/FAILING/STALE.
Now that #3929/#3930 give them a real daily reschedule, a broken reschedule stayed invisible behind
the old exemption - the exact blind spot #3930's field defect exploited.

Callers resolve FrequencyMinutes through CollectorScheduleDefaults.EffectiveRecurringIntervalMinutes,
so an on-load collector's catalog 0 reads as the 1440-minute recapture cadence and lands on the SAME
ladder as index_object_stats (HEALTHY to 36h, STALE to 48h, STOPPED past it). Fixed in Lite's
CollectorHealthRow, the Darling viewer's CollectorHealthRow, and the MCP service's CollectorHealth -
the fleet rollup reuses the MCP type so it is covered without a separate change.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdmA4ND1wLSqA91ax1m4xv
@erikdarlingdata
erikdarlingdata marked this pull request as draft September 23, 2026 12:52
…he catalog doesn't know keeps the floor ladder, and the tests match the ruling

- The three FrequencyMinutes resolutions (Darling MCP, Darling viewer, Lite)
  sent every name missing from the catalog through the on-load 0 -> 1440
  mapping too. That judged an unknown collector on a daily cadence, so one
  30 hours dark read HEALTHY instead of STOPPED, and the live seven-day
  aggregate test failed. Only a catalog entry is resolved now; an unknown
  name keeps 0 and the classifier's floor thresholds, as before #4000.
- AlertReadFailureSurfaceTests pins Classify at 9 parameters (isOnLoad removed).
- ViewerDailyHealthRowTests follow the ruling: an on-load collector is HEALTHY
  2 hours after a success and STOPPED after 100; its failure-rate WARNING is
  asserted with a recent success; a new fact pins an uncataloged name to the
  floor ladder.

Targeted Darling classes 140/140, live on PG 18.6 + TimescaleDB 2.30.1,
including ViewerDailyHealthLivePostgresTests. Full Lite suite 5179/5179.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdmA4ND1wLSqA91ax1m4xv
@erikdarlingdata

Copy link
Copy Markdown
Owner Author

Coordinator fixes (d60435d), after CI failed 4 tests the lane never ran:

  • Product fix: FrequencyMinutes (Darling MCP, Darling viewer, Lite) had mapped every name missing from the catalog to the daily cadence too, so a collector 30 h dark read HEALTHY instead of STOPPED. That was the live seven-day aggregate failure. Only a catalog on-load entry resolves to 1440 now. Unknown names keep the floor ladder, as before On-load collectors are now staleness-exempt forever in CollectorHealthClassifier, even though they recapture daily #4000, pinned by a new fact.
  • Tests aligned to the ruling:
    • the Classify signature pin is 9;
    • an on-load collector is HEALTHY at 2 h and STOPPED at 100 h;
    • the failure-rate WARNING is asserted with a recent success.
  • Targeted Darling classes 140/140 live (PG 18.6 + TimescaleDB 2.30.1); full Lite suite 5179/5179.

@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 23, 2026 12:58
@erikdarlingdata
erikdarlingdata enabled auto-merge (squash) September 23, 2026 12:59
@erikdarlingdata
erikdarlingdata merged commit 5abd34f into dev Sep 23, 2026
15 of 16 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/4000-onload-staleness-bands branch September 23, 2026 13:06
erikdarlingdata added a commit that referenced this pull request Sep 23, 2026
…ntries in their sections (#4080)

Adds 42 entries and 42 link refs (#3992, #3995, #3996, #3998, #4001, #4002, #4003, #4007, #4010, #4011, #4013, #4015, #4020, #4022, #4025, #4029, #4030, #4031, #4036, #4038, #4039, #4040, #4044, #4047, #4048, #4049, #4050, #4051, #4055, #4061, #4063, #4064, #4065, #4066, #4067, #4068, #4069, #4070, #4071, #4073, #4074, #4078). Each PR's entry was buffered, and this lands every entry whose PR was merged on origin/dev when it ran.

#3989 left 26 entries under bare 'Changed' and 'Fixed' lines above '### Added'. They move into '### Changed' and '### Fixed', below the new entries, and one blank line stays under [Unreleased].


Claude-Session: https://claude.ai/code/session_01Ua31ugERL5DmhFVRtf6keQ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant