Skip to content

Fix the alert notebook's collector-freshness check (#4378) - #4416

Merged
erikdarlingdata merged 2 commits into
devfrom
fix/4378-notebook-freshness-success-only
Sep 26, 2026
Merged

erikdarlingdata merged 2 commits into
devfrom
fix/4378-notebook-freshness-success-only

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

Refs #4378.

Why

The alert-notebook endpoint's freshness check decides between "No resolution recorded" and "Unknown (not collected since …)" for a metric whose alert history gave no direct answer. It had three problems: it counted ANY collector run including ERROR rows as evidence the instrument was up, it ran without the sibling reads' command timeout, and its failure path always logged 0 ms elapsed regardless of how long the read actually took.

What changes

  • The freshness query now filters to status = 'SUCCESS' (see Freshness evidence below).
  • The freshness command sets CommandTimeout = McpCommandDeadlines.ReadSeconds (20s), the same budget the sibling MCP reads already use.
  • Both the freshness read and the status-history read now measure elapsed time with a Stopwatch and report ElapsedMilliseconds on failure instead of a hardcoded 0.
  • Reworded a stray internal comment marker in AlertNotebookAuthoredTemplateTests.cs to cite #4223 only.
  • Test-only, alongside this fix: AlertNotebookAuthoredTemplateTests' shared theories (validate / known-dispatch / compose-catalog / budget / binding / viz) were driven off a hand-written two-metric list (BothTemplates()). That list is now AllAuthoredMetrics(), generated by flattening AlertNotebookEndpoint.s_authoredTemplates, so a future authored family (Alert notebooks: authored templates for the remaining families, including custom rules and analysis findings #4223) is covered by these shared theories the moment its registration row lands, with no edit to this test file. The routing theory is now similarly data-driven, asserting each registered metric routes to its own row's Entry.Id.

Recovery edges

The AG family and the Server Unreachable/Restored pair fire under their own metric names rather than through a resolution-title alias, so StatusFromHistory's arm 1 (which only consulted DarlingTriageEndpoint.ResolutionAliases) never reached them — a notebook opened against "AG Replica Disconnected" could not read "Resolved at T" even when "AG Replica Reconnected" arrived later.

Added a small notebook-only pair list (AlertNotebookEndpoint.NotebookRecoveryEdges) consulted by arm 1 alongside ResolutionAliases:

  • AG Replica Disconnected → AG Replica Reconnected
  • AG Database Suspended → AG Data Movement Resumed
  • AG Sync Fell Behind → AG Sync Recovered
  • Server Unreachable → Server Restored

This is deliberately NOT added to ResolutionAliases itself, because that list also builds DarlingTriageEndpoint.SectionsByMetric, and every metric above already has its own entry there — adding them to ResolutionAliases would define the same section-map key twice.

AG Failover is left out on purpose: it is an event (a role change), not a condition that clears, so it has no natural recovery edge.

Freshness evidence

Only SUCCESS counts as evidence the collector reached the target. SKIPPED was dropped from the freshness predicate: it can be written before any query reaches the target — Lite/Services/RemoteCollectorService.cs's user-cancelled-MFA catch sets status = "SKIPPED" at the point the sign-in is cancelled, never having contacted the server. Darling's own per-run classifier, EnumeratedCollectorDriver.ClassifyReturnedRun, never returns SKIPPED either, so the status carries no server-reachability information on either code path.

Test plan

  • AlertNotebookEndpointTests, AlertNotebookAuthoredTemplateTests, DocCommentHygieneTests — all pass in-process on this machine (32, 77, 29 total respectively, 0 failed).
  • New pure pins in AlertNotebookEndpointTests.cs: CollectorFreshnessSql contains the status predicate and excludes 'ERROR'; a source-text pin confirms CommandTimeout = McpCommandDeadlines.ReadSeconds is set and that no Report(..., 0, ex) call remains for either read; three recovery-edge pins (AG reconnect, Server Restored, AG Failover absent from the pair list).
  • New live class AlertNotebookCollectorFreshnessLiveTests (3 facts), against my own throwaway container (timescale/timescaledb:2.30.1-pg18, removed after this run): only-ERROR-after-anchor reads Unknown; SUCCESS-after-anchor reads "No resolution recorded"; only-rows-before-anchor reads Unknown. All 3 GREEN on this branch.
  • RED confirmed on origin/dev (084cfb3) in a detached worktree: both the pure and the live test files fail to COMPILE against the old code (ResolveStatusAsync, CollectorFreshnessSql, and NotebookRecoveryEdges don't exist there — ResolveStatusAsync was private and is now internal for the live test to call it directly).
  • Both Darling.Tests.csproj and Lite.Tests.csproj build 0 errors with -p:EnableWindowsTargeting=true.
  • LivePostgresCollectionHygieneTests.EveryClassUsingTheSharedStore_IsSerializedOrDocumentsWhyNot fails on this Mac with a PresentationFramework load error — pre-existing platform gap unrelated to this change (it reflects over all loaded assemblies, which includes a WPF-referencing assembly this environment cannot load); left for CI, which runs on Windows.
  • Darling.Tests/Lite.Tests target net10.0-windows and cannot fully run here; CI decides the rest of the suite.

CHANGELOG entry

SECTION: Fixed
ENTRY:

- Freshness now counts only SUCCESS/SKIPPED collection_log rows, not
  ERROR runs, matching the other freshness reads in this codebase.
- The freshness command uses the sibling reads' 20s CommandTimeout.
- Elapsed time is measured with a Stopwatch instead of reporting 0.
- The AG and Server Unreachable/Restored recovery pairs now resolve
  the notebook status to "Resolved at T" (AG Failover has no natural
  recovery edge and stays out).
- Renamed a stray '(#4223 lane 0)' comment to '(#4223)'.
- Test-only: AlertNotebookAuthoredTemplateTests' shared theories are
  now data-driven off AlertNotebookEndpoint.s_authoredTemplates so a
  future authored family needs no edit to this file.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 26, 2026 13:21
@erikdarlingdata
erikdarlingdata merged commit 5899b85 into dev Sep 26, 2026
16 of 18 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/4378-notebook-freshness-success-only branch September 26, 2026 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant