Repository navigation
On-load collectors band on the same staleness ladder as any other collector (#4000) - #4029
Merged
Merged
Conversation
…lector, on their effective daily cadence (#4000) CollectorHealthClassifier.Classify no longer exempts on-load collectors (server_config, database_config, database_scoped_config, trace_flags, server_properties) from STOPPED/FAILING/STALE. Now that #3929/#3930 give them a real daily reschedule, a broken reschedule stayed invisible behind the old exemption - the exact blind spot #3930's field defect exploited. Callers resolve FrequencyMinutes through CollectorScheduleDefaults.EffectiveRecurringIntervalMinutes, so an on-load collector's catalog 0 reads as the 1440-minute recapture cadence and lands on the SAME ladder as index_object_stats (HEALTHY to 36h, STALE to 48h, STOPPED past it). Fixed in Lite's CollectorHealthRow, the Darling viewer's CollectorHealthRow, and the MCP service's CollectorHealth - the fleet rollup reuses the MCP type so it is covered without a separate change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdmA4ND1wLSqA91ax1m4xv
erikdarlingdata
marked this pull request as draft
September 23, 2026 12:52
…he catalog doesn't know keeps the floor ladder, and the tests match the ruling - The three FrequencyMinutes resolutions (Darling MCP, Darling viewer, Lite) sent every name missing from the catalog through the on-load 0 -> 1440 mapping too. That judged an unknown collector on a daily cadence, so one 30 hours dark read HEALTHY instead of STOPPED, and the live seven-day aggregate test failed. Only a catalog entry is resolved now; an unknown name keeps 0 and the classifier's floor thresholds, as before #4000. - AlertReadFailureSurfaceTests pins Classify at 9 parameters (isOnLoad removed). - ViewerDailyHealthRowTests follow the ruling: an on-load collector is HEALTHY 2 hours after a success and STOPPED after 100; its failure-rate WARNING is asserted with a recent success; a new fact pins an uncataloged name to the floor ladder. Targeted Darling classes 140/140, live on PG 18.6 + TimescaleDB 2.30.1, including ViewerDailyHealthLivePostgresTests. Full Lite suite 5179/5179. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdmA4ND1wLSqA91ax1m4xv
Owner
Author
|
Coordinator fixes (d60435d), after CI failed 4 tests the lane never ran:
|
erikdarlingdata
marked this pull request as ready for review
September 23, 2026 12:58
erikdarlingdata
enabled auto-merge (squash)
September 23, 2026 12:59
erikdarlingdata
added a commit
that referenced
this pull request
Sep 23, 2026
…ntries in their sections (#4080) Adds 42 entries and 42 link refs (#3992, #3995, #3996, #3998, #4001, #4002, #4003, #4007, #4010, #4011, #4013, #4015, #4020, #4022, #4025, #4029, #4030, #4031, #4036, #4038, #4039, #4040, #4044, #4047, #4048, #4049, #4050, #4051, #4055, #4061, #4063, #4064, #4065, #4066, #4067, #4068, #4069, #4070, #4071, #4073, #4074, #4078). Each PR's entry was buffered, and this lands every entry whose PR was merged on origin/dev when it ran. #3989 left 26 entries under bare 'Changed' and 'Fixed' lines above '### Added'. They move into '### Changed' and '### Fixed', below the new entries, and one blank line stays under [Unreleased]. Claude-Session: https://claude.ai/code/session_01Ua31ugERL5DmhFVRtf6keQ Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4000.
Why
CollectorHealthClassifier.Classify(PerformanceMonitor.Common/ServerHealthBands.cs) permanentlyexempted an on-load collector (
server_config,database_config,database_scoped_config,trace_flags,server_properties) from every staleness band, reading it by failure/abandon ratealone. That was correct when an on-load collector's only capture was the last connect. Since
#3929/#3930 gave every one of them a real daily reschedule (
CollectorScheduleDefaults.OnLoadRecaptureMinutes= 1440) on top of the connect-time capture, "hours since last success" became a meaningful signal
again - but the exemption stayed, so a daily reschedule that silently breaks (a bug in the NextDue
seeding, an exception swallowed before the run) whose historical runs were all successes still read
HEALTHY forever. That is the exact blind spot that let #3930's field defect go unnoticed.
Erik's ruling (posted on the issue): no new thresholds or settings. Classify on-load collectors like
any scheduled collector, on the effective 1440-minute cadence
(
CollectorScheduleDefaults.EffectiveRecurringIntervalMinutes), with the existing staleness bands.What changes
CollectorHealthClassifier.Classify- removed theisOnLoadparameter and the exemption branchentirely. The method is now a pure function of run counts and cadence; on-load-ness is no longer a
distinct input, only a distinct CADENCE the caller resolves before calling in. Parameter count: 10 -> 9.
FrequencyMinutesproperties (one per surface) now route the catalog lookup throughCollectorScheduleDefaults.EffectiveRecurringIntervalMinutes, so a 0 default (on-load, or acollector no longer in the catalog) resolves to the 1440-minute daily cadence instead of the 24h/4h
floors a raw 0 used to fall to:
Lite/Services/LocalDataService.CollectionHealth.cs(CollectorHealthRow)Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.CollectionHealth.cs(CollectorHealthRow)Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs(CollectorHealth)DarlingFleetReader.csconstructs the SAMECollectorHealthtypefrom
FleetCollectionHealthSql's rows, so the fix reaches the fleet overview without a separate change.Confirmed by reading
DarlingFleetReader.cs- no directClassifycall there to update.IsOnLoadCollector/OnLoadCollectorNamesstay - still consumed byProducedThenStopped's "threetab opens are not three consecutive cycles" exclusion, which is a different concept (consecutive-run
semantics, not staleness) and out of this issue's scope.
IsOnLoadCollectorsummary,
Classify's order-of-bands summary, and one stale cross-reference inFormatOutputFinding'sremarks that cited
Classify'sisOnLoadpattern by name).Test plan
Darling.TestsandLite.Testsboth build with0 Warning(s).Classify_BandsTheDecisionTable, both SKUs) updated:isOnLoadcolumnremoved from every row; the old "on-load exemption" demonstration rows (500h dark -> Healthy/Warning)
now assert the corrected
Stoppedoutcome instead of being deleted, so the removed behavior stayspinned as a regression rather than silently disappearing.
index_object_statsboundary factsbut through a REAL on-load collector name (
server_config) so the catalog resolution(
EffectiveRecurringIntervalMinutes) and the ladder are exercised together, not just the pureClassifyfunction:- 27h since last success -> HEALTHY (well inside the new 36h stale line)
- 37h -> STALE (was HEALTHY forever before this fix)
- 49h -> STOPPED (was HEALTHY forever before this fix - the A Darling server connected for more than 30 days loses its config facts: on-load snapshots age out and nothing re-collects them #3930-shaped blind spot)
- Darling additionally checks both
CollectorHealthRow(viewer) andCollectorHealth(MCPservice) at 49h in one fact, matching the suite's existing "on both row types" discipline.
TheBandingSignature_TakesNoOutputAndNoDenialCurrency(
CollectionOutputBesideCostTests.cs, both SKUs) reflectsClassify's exact parameter list byname/count - updated from 10 to 9 parameters with the reason (On-load collectors are now staleness-exempt forever in CollectorHealthClassifier, even though they recapture daily #4000 removed
isOnLoad) in thecomment, per this repo's "update a pin deliberately, never weaken one" rule.
-classper touched class, not by file - some of thesefiles hold exactly one class each, verified by grep before running):
-
Darling.Tests:CollectorHealthClassifierTests72/72,{RegressedFromProductiveTests, ProductiveZeroBandingTests, AbandonedRunEraInvariantReadTests, CollectionOutputBesideCostTests}49/49 (1 skipped, pre-existing/unrelated),
DocCommentHygiene*77/77 (doc comment edits didn'ttrip the hygiene pin).
-
Lite.Tests:{CollectorHealthClassifierTests, RegressedFromProductiveTests, CollectionOutputBesideCostTests}90/90.- All: 0 failed.
Darling.Tests.exe/Lite.Tests.exewith no filter) - flaggingfor the coordinator rather than silently skipping: this lane hit a hard 13:10Z deadline shared
with a second issue (MCP get_trace_flags and the web viewer share a narrower version of #3929's stale-flag defect #3999) on its own branch, and every class touched by this diff (by grep,
not by filename) is covered above. CI will run the full suite on this PR; please treat that as
the gate before arming auto-merge rather than assuming my local run covered it.
What the coordinator should double-check
CollectionOutputBesideCostTests.csis a deliberate,reasoned update (see comment), not a weakening - it now asserts a SMALLER number, which a careless
"make the test pass" edit could get backwards (e.g. by leaving 10 and hoping for a compile error
instead of a runtime pin failure). Worth a second look.
Lite/Services/LocalDataService.CollectionHealth.csalongside bothDarling surfaces - all three
FrequencyMinutesproperties had byte-for-byte identical logic beforethis change and now have byte-for-byte identical fixes.
🤖 Generated with Claude Code
https://claude.ai/code/session_01GdmA4ND1wLSqA91ax1m4xv