Give Collection Stopped, Capture Down and Collector Cost Regression an authored self-monitor notebook (#4223)

[erikdarlingdata]

Refs #4223.

Why

The alert notebook's mechanical fallback converts a metric's generic triage sections into read cells with no awareness of what actually drives the alert. The three self-monitor alerts — Collection Stopped, Capture Down, Collector Cost Regression — are about the monitoring tool's own health on one real server, and each has a specific, known-shape set of evidence a reader needs: the collection log, and for a cost regression, that one collector's own cost trend and stall history. This gives them an authored template instead of the fallback.

What changes

• AlertContext.CollectorName (new field, PerformanceMonitor.Notifications/AlertContext.cs): a structured collector name, mirroring the existing WaitType field added for Poison Wait. Persisted as the trailing, nullable member of AlertContextDto, with AlertContextSerializer.TryReadCollectorName(contextJson) reading it back without rehydrating the whole context. Set at the Collector Cost Regression fire site in DarlingSelfAlertEvaluator.cs (ApplyCostRegressionsAsync, ~line 1640): context: new AlertContext { CollectorName = regression.CollectorName }. FireAsync (~line 6981) passes a caller-supplied context straight through with no merge, so this is the only context data this fire site carries.
• New file AlertNotebookEndpoint.Templates.SelfMonitor.cs: one builder, BuildSelfMonitorCells, registered once in s_authoredTemplates for all three metric names, routing to authored/self-monitor. Cells: HeaderCell, StatusCell, get_collection_health for every metric, then a per-metric get_collection_log read:
  • Collection Stopped — no status filter, hours=2, limit=50. ApplyCollectionStoppedAsync fires off "no SUCCESS/SKIPPED run recently" (ReadCollectionSignalsAsync), so the recent runs it covers can be ERROR, PERMISSIONS, EXTENSION_MISSING, or simply absent — no single status value covers all of those, so none is applied (documented in the file's doc comment).
  • Capture Down — status=SESSION_MISSING, hours=24, limit=50. Traced the firing condition directly: ApplyCaptureDownAsync (DarlingSelfAlertEvaluator.cs ~1420) only ever receives collector names from ReadMissingCaptureSessionsAsync (~6691), which runs MissingCaptureSessionsSql (~6637) — and that statement's own WHERE x.status = 'SESSION_MISSING' clause (line ~6655) is the ONLY predicate that can put a collector into the missing list. No other status can reach this alert's firing path, so one status value covers every case.
  • Collector Cost Regression — collector_name-scoped get_collection_log, plus get_collector_cost (days_back=14) and get_collector_stall_probes (days_back=7) for the collector named in the row's persisted context (read the same way BuildPoisonWaitCells reads WaitType — off row.ContextJson, never re-derived from prose). When the context carries no collector name (a row written before this member existed), the two collector-scoped reads are replaced with a note cell, and the log read falls back to the unscoped form — never a read with an empty collector_name.
• No trend panel, for any of the three: get_collector_cost and get_collector_stall_probes have no ComposeSpec/chart-catalog source, so unlike every other authored family (which plots the condition it fired on) this family is read cells only.
• Coverage: AlertNotebookAuthoredTemplateTests' AllAuthoredMetrics/AllAuthoredMetricsWithExpectedId theories are data-driven off s_authoredTemplates itself, so the new row is automatically covered by every existing shared theory (routing, ValidateNotebookDefinition, budget) with no list to update.
• s_authoredLimitlessTrendReads gained get_collector_cost (declares only days_back/collector_name, no limit param at all).

Test plan

New class AlertNotebookTemplateSelfMonitorTests.cs (20 cases): routing for all three metrics to authored/self-monitor; Collection Stopped's log cell carries no status; Capture Down's carries status=SESSION_MISSING; Collector Cost Regression's collector-scoped reads carry the name from two different row contexts (not a literal); the no-collector-name case degrades to a note with no collector-scoped reads and no empty collector_name param on any cell, for any of the three metrics; no panel-type cell for any of the three; every param every read cell passes is declared in that read's catalog descriptor (DarlingWebEndpoints.CatalogDescriptors); an AlertContext round trip for CollectorName, plus the legacy-JSON-has-no-member case.

Fire-site pin, through the product's own path (not a direct helper call): extended CollectorCostRegression_FiresOnEntry_SuppressedWithinCooldown_ResolvesWhenGone in DarlingSelfAlertTests.cs to assert the AlertOutcome the deliverer actually received (ApplyCostRegressionsAsync → FireAsync → RecordingDeliverer) carries Context.CollectorName == "query_store".

RED / mutation, run in-process on macOS (Darling.Tests.dll, Microsoft.WindowsDesktop.App stripped from the runtimeconfig):

• Mutation (real, reverted): removed the CollectorName = regression.CollectorName assignment at the fire site (context: new AlertContext { }), rebuilt, ran the fire-site pin: Darling.Tests.DarlingSelfAlertTests.CollectorCostRegression_FiresOnEntry_SuppressedWithinCooldown_ResolvesWhenGone [FAIL] — Total: 248, Errors: 0, Failed: 1. Restored the assignment, rebuilt: Total: 248, Errors: 0, Failed: 0.
• Full run after restore, all green, 0 build errors both projects: AlertNotebookTemplateSelfMonitorTests (20/20), AlertNotebookAuthoredTemplateTests (152/152), AlertNotebookEndpointTests (29/29), AlertNotebookTemplateCustomRulesTests (11/11), AlertNotebookTemplateAnalysisFindingsTests (9/9), DarlingSelfAlertTests (248/248, 3 skipped — pre-existing, unrelated to this change), DocCommentHygieneTests (part of the combined 99/99 run alongside the new class).
• AlertNotebookAuthoredContextTests (live, own rig): not run — this PR adds no new live class and does not touch that file; its coverage is unaffected by a plain-field addition to AlertContext.
• RED on origin/dev @ 64ea4a2a5 (detached worktree, new test file copied in, no other source change): the new class doesn't compile against dev as-is, since AlertContext.CollectorName, AlertContextSerializer.TryReadCollectorName, and the AlertNotebookEndpoint.SelfMonitorTemplateVersion constant don't exist there yet. Added three minimal, unwired compile shims in the scratch worktree only (never committed): a CollectorName auto-property on AlertContext; a TryReadCollectorName that always returns null; and a SelfMonitorTemplateVersion constant (value -1, satisfies the compiler only — routing for the three metrics isn't registered on dev, so the version-equality assertion fails at runtime, not the constant). Built clean, ran AlertNotebookTemplateSelfMonitorTests: Total: 22, Errors: 0, Failed: 21, Skipped: 0. Failing (21 of 22; the routing/build, per-metric status-filter, collector-name, empty-param, no-panel-cell, and declared-param theories all fail, plus the round trip): AuthoredTemplate_SelfMonitorMetrics_AreNotNull and _RouteToOneTemplate (all 3 metrics), BuildSelfMonitorCells_CollectionStopped_LogCellHasNoStatusFilter, BuildSelfMonitorCells_CaptureDown_LogCellFiltersSessionMissing, BuildSelfMonitorCells_CostRegression_CollectorNameComesFromRowContext (both cases), BuildSelfMonitorCells_CostRegression_NoCollectorName_UsesNoteInsteadOfScopedReads, BuildSelfMonitorCells_NeverEmitsAnEmptyCollectorNameParam (all 3 metrics), BuildSelfMonitorCells_EmitsNoPanelCell (all 3 metrics), BuildSelfMonitorCells_EveryReadCellParamIsDeclaredInItsCatalogEntry (all 3 metrics), SelfMonitorContext_SerializesAndRehydratesTheCollectorName. Sample assertion (AuthoredTemplate_SelfMonitorMetrics_AreNotNull, metric "Collection Stopped"): Assert.NotNull() Failure: Value of type 'Nullable<AuthoredTemplateEntry>' does not have a value — dev has no routing entry for these metrics, so AuthoredTemplate(...) returns null. Round-trip sample (SelfMonitorContext_SerializesAndRehydratesTheCollectorName): Assert.Equal() Failure: Strings differ \ Expected: "query_store" \ Actual: null — the null-returning shim can't round-trip a value. The one passing case out of 22, SelfMonitorContext_LegacyJsonWithNoCollectorNameMember_RehydratesNull, isn't a false green: it asserts the reader returns null for context JSON with no CollectorName member, and the null-returning shim satisfies that unconditionally, so it carries no signal either way. Shims removed with the scratch worktree; nothing from this step is committed.

Lite (PerformanceMonitor.Notifications/AlertContext.cs is shared with Lite): Lite.Tests.dll cannot run in-process on this Mac — discovery dies on WindowsBase, per the repo's own documented limitation; build only. Lite.Tests/Lite.Tests.csproj built clean (0 errors, 0 warnings) against the new CollectorName field. Read Lite.Tests/AnalysisNotificationTests.cs, Lite.Tests/FindingRoutingTests.cs, Lite.Tests/GenericWebhookTests.cs and Lite.Tests/AlertContextBuildersTests.cs, which cover AlertContextSerializer.Serialize/TryDeserialize round trips (analogous to WaitType's Darling-side pin) — none of them assert on a fixed member list, so a plain nullable field addition can't red them; they weren't changed here since they're existing, generic round-trip coverage, not a per-field census. CI's Lite job should confirm this on Windows: it runs AnalysisNotificationTests, FindingRoutingTests, GenericWebhookTests and AlertContextBuildersTests, and none should change.

CHANGELOG entry

SECTION: Added
ENTRY:

• Self-monitoring alerts (Collection Stopped, Capture Down, Collector Cost Regression) open a notebook with the server's collection health and the collector's log and cost ([Give Collection Stopped, Capture Down and Collector Cost Regression an authored self-monitor notebook (#4223) #4437])
  REF:
  [Give Collection Stopped, Capture Down and Collector Cost Regression an authored self-monitor notebook (#4223) #4437]: Give Collection Stopped, Capture Down and Collector Cost Regression an authored self-monitor notebook (#4223) #4437