Skip to content

Alert notebooks: authored templates for the remaining families, including custom rules and analysis findings #4223

Closed

Description

@erikdarlingdata

Part of #4219.

Problem

After the MVP, every alert opens a notebook. Only blocking and deadlocks are authored, though; every other family converts mechanically from DarlingTriageEndpoint.SectionsByMetric. Several families get nothing useful from that conversion, because they fall through to the two-section fallback (DarlingTriageEndpoint.cs:381-385) or because the read they need isn't on the web dispatch.

Custom rules (Custom:<id>) get only the fallback. The rule itself is a compose spec (CustomAlertRuleDefinition, validated by ComposeSpec.TryParsePanel). So the notebook could plot the exact measure the rule evaluates, with its threshold or band, over the window. Today the reader gets a server summary.

Analysis findings (Analysis: <category> [<hash8>]) get only the fallback.

  • The hash is the first 8 characters of story_path_hash (AnalysisNotificationService.cs:719-725), and get_analysis_findings already persists the evidence rows (include_drilldown).
  • The web dispatch entry doesn't expose include_drilldown (DarlingWebEndpoints.cs:1902: server, hours and as_of only).

These alerts are unmapped and get only the fallback:

  • Agent Not Running.
  • The per-server self-monitor alerts: Collection Stopped, Capture Down, Collector Cost Regression. PerServerSelfMonitorMetrics keeps them off the store sections, which is right, but no per-server framing replaces them.

Some framings need data the web dispatch doesn't expose:

Fix shape

Author templates. Each opens with the MVP's header and status cells, and every read carries a limit or goes through the chart bucket budget.

Family What happened To what Since when Still happening?
High CPU (both engines) get_cpu_utilization / get_pg_cpu_utilization (bucketed per #4193) get_top_queries_by_cpu, get_top_procedures_by_cpu; the maintenance/job naming the card already carries (#3495) Lookback trend, alert instant marked "CPU Resolved"; latest sample
Poison Wait (both engines) get_wait_stats / get_pg_wait_stats get_waiting_tasks; get_resource_semaphore + get_memory_grants when the wait is RESOURCE_SEMAPHORE get_wait_trend for the wait type the alert names "Poison Waits Cleared"
Long-Running Query get_active_queries get_plan_xml for the query get_long_query_completions Is the session still in the latest snapshot
Forced Plan Failing get_plan_corrections The query and plan Corrections timeline "…Resolved"; get_query_store_regressions only as click-to-run
Server Unreachable / Restored get_collection_health — get_collection_log with status (below) "Server Restored"
Agent jobs (Failed, Long-Running, Agent Not Running) The alert detail (job, step, message); get_running_jobs Job name Per-job history, once a read exists (below) Next run / "…Cleared"
PostgreSQL-only (Wraparound, Vacuum Horizon, Slot Retention) get_pg_wraparound_risk / get_pg_xmin_horizon / get_pg_replication_slots The xmin holder, slot, or database The same reads' 24 h trend Latest sample below threshold
Self-monitor, per-server (Collection Stopped, Capture Down, Collector Cost Regression) get_collection_health; get_collection_log with status The collector named in the detail get_collector_stall_probes, get_collector_cost Latest successful run of that collector
Custom rules The rule's compose spec (GET /api/alerts/{id}), re-rendered as a time series with its thresholds / band The rule's scope (servers or tag) The same spec over the lookback The rule's later firings; its evaluation health (the Custom Alert Rules Unhealthy family)
Analysis findings get_analysis_findings with include_drilldown, reduced to the chain whose story_path_hash starts with the hash The drill-down evidence rows first_seen, last_seen, occurrences The chain's last_seen after at
Reports (digests, sweep rollup) No notebook. The link targets the report's own surface (#/sweeps for the rollup) or is omitted; decide per report.

Supporting changes:

  • Web dispatch:
  • Agent job history: decide whether it's a new read or out of scope. If out of scope, the template says so in a markdown cell instead of implying the history is empty.
  • Poison Wait: confirm the wait type rides context_json as data, not only in prose. If it doesn't, add it to the context builder, so the trend cell can bind wait_type.
  • Custom rules:
    • the template reads the rule by the id parsed from the metric name;
    • a deleted rule becomes a note, never a 500;
    • a scalar-mode spec is re-rendered in time-bucket mode without changing its measure, filters or aggregate.
  • Button label: once this lands, relabel the link from "Open triage page" to "Investigate" in Slack, Teams, PagerDuty and email. It's a label, not a fact name; {{triage_url}} keeps its name.

Where

  • Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs (the template map that replaces SectionsByMetric, PerServerSelfMonitorMetrics ~:149-154)
  • DarlingWebEndpoints.cs dispatch: get_collection_log ~:1942, get_analysis_findings ~:1902
  • CustomAlertRuleDefinition.cs, CustomAlertEvaluator.cs (MetricNameFor), /api/alerts/{id} ~:547
  • PerformanceMonitor.Notifications/AnalysisNotificationService.cs:719-725; Mcp/DarlingMcpTools.cs get_analysis_findings ~:1045
  • PerformanceMonitor.Notifications/AlertFamily.cs (the census that decides which families exist)
  • PerformanceMonitor.Notifications/WebhookAlertService.cs button labels ~:1711, ~:765, ~:2418

Pins

  • Coverage: every metric in the AlertFamily census, and every prefix shape (Custom:, Analysis: ), resolves to an authored template or a deliberate "no notebook" (reports). None resolves to the generic fallback. It's the same census walk NotificationRoutingTests uses, so a new alert without a template reds CI.
  • Custom rule:
    • a fixed rule definition binds to a composed cell with the same measure, filters and aggregate as the rule;
    • the cell's thresholds equal the rule's threshold, or both bounds for Between/Outside;
    • an unknown id gives a note, not an error.
  • Analysis finding: given two findings in the window whose hashes share a 4-character prefix but differ at the 8th, the template picks the one matching all 8.
  • Web dispatch: get_collection_log?status=ERROR and get_analysis_findings?include_drilldown=true reach the tool with the flag set, with parity against the MCP tool's output for the same arguments.
  • The MVP budget pin covers every template added here.

Out of scope

User-editable override templates, an MCP read returning the bound notebook, and server tabs that accept an anchored window.

Activity

  1. added
    enhancementNew feature or request
    client-siteOwned by the client-site agents (other laptop). Local sessions never pick these up.
    on Sep 25, 2026
  2. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling

    Ruled (2026-09-26): how the remaining alert families get their notebooks. These settle the points the body leaves open.

    1. Agent job history is out of scope here. The Agent-job templates carry a markdown cell, in neutral words, saying that job history isn't read yet. They never imply the history is empty. A new read would be its own issue.
    2. Poison Wait: if the wait type doesn't already ride the firing's context_json as data, the lane adds it in the context builder. A pin checks that the firing's context carries it, so the trend cell can bind wait_type.
    3. Reports: the sweep rollup links to #/sweeps, and the digests carry no notebook link.
    4. Structure first: a small first change makes AlertNotebookEndpoint a partial class.
      • It's a pure move: each family's template builder goes in its own file (AlertNotebookEndpoint.Templates.<Family>.cs).
      • The dispatch table stays in one file, with one registration line per family, kept sorted.
      • Nothing changes in behaviour. The existing notebook pins stay green, and one new pin checks that every registered family resolves.
      • After that, family templates can be built in parallel, each touching only its own file and its registration line.

    The work starts after #4395 (the authored Blocking and Deadlocks templates, which set the pattern) is on dev.

  3. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    #4418 is merged (0dd9cfb). Long-Running Query and Forced Plan Failing alerts now open authored notebooks instead of the mechanical fallback.

  4. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    #4419 is merged (99aba09). Wraparound Risk, Vacuum Horizon Blocked and Replication Slot Retention alerts now open authored notebooks, each with a 24-hour gauge trend.

  5. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    #4422 is merged (463667f). Server Unreachable/Restored and the Agent-job alerts now open authored notebooks, and the collection-log web read accepts a status filter.

  6. 5 remaining items

  7. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    #4421 is merged (2b4f30e). Report alerts now link to the sweep timeline, or carry no link for the digests, and the notebook endpoint declares them as having no notebook. Ordinary alerts keep "Open triage page".

  8. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    #4424 is merged (b476262). Poison Wait alerts now open an authored notebook, and the alert context carries the wait type. It shows the resource semaphore and memory grants for RESOURCE_SEMAPHORE, and a wait trend bound to the incident's wait type.

  9. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    #4425 is merged (a74e857). The shared context step for authored notebooks is in: context-bearing templates, prefix routing for Custom and Analysis metrics, and a pre-fetch that runs only for families that need it. No notebook's output changes. The Custom rules, Analysis findings and self-monitor templates build on it, one at a time.

  10. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    #4433 is merged (e0232a4). Custom-rule alerts now open an authored notebook. It charts the rule's own measure by the hour, with its thresholds as reference lines, and says plainly when the rule was deleted or its definition can't be read. Analysis findings are next, then self-monitor.

  11. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    #4436 is merged (64ea4a2). Analysis-finding alerts now open an authored notebook with the finding's summary and its evidence, read with drill-down. A finding no longer in the store gives a plain note. The get_analysis_findings read's catalog entry now declares limit, include_drilldown and full_text. The self-monitor notebook is the last family.

  12. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    #4437 is merged (13329d1), so every alert family now opens an authored notebook, and this issue is closing. The PRs, in merge order:

    Metrics without an authored template still get the mechanical notebook.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    client-siteOwned by the client-site agents (other laptop). Local sessions never pick these up.enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions