Repository navigation
Alert notebooks: authored templates for the remaining families, including custom rules and analysis findings #4223
Description
Part of #4219.
Problem
After the MVP, every alert opens a notebook. Only blocking and deadlocks are authored, though; every other family converts mechanically from DarlingTriageEndpoint.SectionsByMetric. Several families get nothing useful from that conversion, because they fall through to the two-section fallback (DarlingTriageEndpoint.cs:381-385) or because the read they need isn't on the web dispatch.
Custom rules (Custom:<id>) get only the fallback. The rule itself is a compose spec (CustomAlertRuleDefinition, validated by ComposeSpec.TryParsePanel). So the notebook could plot the exact measure the rule evaluates, with its threshold or band, over the window. Today the reader gets a server summary.
Analysis findings (Analysis: <category> [<hash8>]) get only the fallback.
- The hash is the first 8 characters of
story_path_hash(AnalysisNotificationService.cs:719-725), andget_analysis_findingsalready persists the evidence rows (include_drilldown). - The web dispatch entry doesn't expose
include_drilldown(DarlingWebEndpoints.cs:1902: server, hours and as_of only).
These alerts are unmapped and get only the fallback:
- Agent Not Running.
- The per-server self-monitor alerts: Collection Stopped, Capture Down, Collector Cost Regression.
PerServerSelfMonitorMetricskeeps them off the store sections, which is right, but no per-server framing replaces them.
Some framings need data the web dispatch doesn't expose:
- No per-job history read. Failed Agent Job and Long-Running Job can show only
get_running_jobsand the default trace. get_collection_loghas nostatusfilter on the web. The MCP tool has one since get_collection_log has no status filter - the one question the log exists for cannot be asked #3869, but the web entry doesn't (DarlingWebEndpoints.cs:1942). So "which collectors failed around the firing" means reading 200 raw rows.
Fix shape
Author templates. Each opens with the MVP's header and status cells, and every read carries a limit or goes through the chart bucket budget.
| Family | What happened | To what | Since when | Still happening? |
|---|---|---|---|---|
| High CPU (both engines) | get_cpu_utilization / get_pg_cpu_utilization (bucketed per #4193) |
get_top_queries_by_cpu, get_top_procedures_by_cpu; the maintenance/job naming the card already carries (#3495) |
Lookback trend, alert instant marked | "CPU Resolved"; latest sample |
| Poison Wait (both engines) | get_wait_stats / get_pg_wait_stats |
get_waiting_tasks; get_resource_semaphore + get_memory_grants when the wait is RESOURCE_SEMAPHORE |
get_wait_trend for the wait type the alert names |
"Poison Waits Cleared" |
| Long-Running Query | get_active_queries |
get_plan_xml for the query |
get_long_query_completions |
Is the session still in the latest snapshot |
| Forced Plan Failing | get_plan_corrections |
The query and plan | Corrections timeline | "…Resolved"; get_query_store_regressions only as click-to-run |
| Server Unreachable / Restored | get_collection_health |
— | get_collection_log with status (below) |
"Server Restored" |
| Agent jobs (Failed, Long-Running, Agent Not Running) | The alert detail (job, step, message); get_running_jobs |
Job name | Per-job history, once a read exists (below) | Next run / "…Cleared" |
| PostgreSQL-only (Wraparound, Vacuum Horizon, Slot Retention) | get_pg_wraparound_risk / get_pg_xmin_horizon / get_pg_replication_slots |
The xmin holder, slot, or database | The same reads' 24 h trend | Latest sample below threshold |
| Self-monitor, per-server (Collection Stopped, Capture Down, Collector Cost Regression) | get_collection_health; get_collection_log with status |
The collector named in the detail | get_collector_stall_probes, get_collector_cost |
Latest successful run of that collector |
| Custom rules | The rule's compose spec (GET /api/alerts/{id}), re-rendered as a time series with its thresholds / band |
The rule's scope (servers or tag) | The same spec over the lookback | The rule's later firings; its evaluation health (the Custom Alert Rules Unhealthy family) |
| Analysis findings | get_analysis_findings with include_drilldown, reduced to the chain whose story_path_hash starts with the hash |
The drill-down evidence rows | first_seen, last_seen, occurrences |
The chain's last_seen after at |
| Reports (digests, sweep rollup) | No notebook. The link targets the report's own surface (#/sweeps for the rollup) or is omitted; decide per report. |
Supporting changes:
- Web dispatch:
- add
statustoget_collection_log; - add
include_drilldowntoget_analysis_findings. - Both already exist on the MCP tools, so this is the Darling Web: a browser viewer served by the service (Mac/cross-platform), riding the existing Kestrel + server-side read layer #1562 no-drift rule applied to the web.
- add
- Agent job history: decide whether it's a new read or out of scope. If out of scope, the template says so in a markdown cell instead of implying the history is empty.
- Poison Wait: confirm the wait type rides
context_jsonas data, not only in prose. If it doesn't, add it to the context builder, so the trend cell can bindwait_type. - Custom rules:
- the template reads the rule by the id parsed from the metric name;
- a deleted rule becomes a note, never a 500;
- a scalar-mode spec is re-rendered in time-bucket mode without changing its measure, filters or aggregate.
- Button label: once this lands, relabel the link from "Open triage page" to "Investigate" in Slack, Teams, PagerDuty and email. It's a label, not a fact name;
{{triage_url}}keeps its name.
Where
Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs(the template map that replacesSectionsByMetric,PerServerSelfMonitorMetrics~:149-154)DarlingWebEndpoints.csdispatch:get_collection_log~:1942,get_analysis_findings~:1902CustomAlertRuleDefinition.cs,CustomAlertEvaluator.cs(MetricNameFor),/api/alerts/{id}~:547PerformanceMonitor.Notifications/AnalysisNotificationService.cs:719-725;Mcp/DarlingMcpTools.csget_analysis_findings~:1045PerformanceMonitor.Notifications/AlertFamily.cs(the census that decides which families exist)PerformanceMonitor.Notifications/WebhookAlertService.csbutton labels ~:1711, ~:765, ~:2418
Pins
- Coverage: every metric in the
AlertFamilycensus, and every prefix shape (Custom:,Analysis:), resolves to an authored template or a deliberate "no notebook" (reports). None resolves to the generic fallback. It's the same census walkNotificationRoutingTestsuses, so a new alert without a template reds CI. - Custom rule:
- a fixed rule definition binds to a composed cell with the same measure, filters and aggregate as the rule;
- the cell's
thresholdsequal the rule's threshold, or both bounds for Between/Outside; - an unknown id gives a note, not an error.
- Analysis finding: given two findings in the window whose hashes share a 4-character prefix but differ at the 8th, the template picks the one matching all 8.
- Web dispatch:
get_collection_log?status=ERRORandget_analysis_findings?include_drilldown=truereach the tool with the flag set, with parity against the MCP tool's output for the same arguments. - The MVP budget pin covers every template added here.
Out of scope
User-editable override templates, an MCP read returning the bound notebook, and server tabs that accept an anchored window.
Activity
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling
Ruled (2026-09-26): how the remaining alert families get their notebooks. These settle the points the body leaves open.
context_jsonas data, the lane adds it in the context builder. A pin checks that the firing's context carries it, so the trend cell can bindwait_type.#/sweeps, and the digests carry no notebook link.AlertNotebookEndpointa partial class.AlertNotebookEndpoint.Templates.<Family>.cs).The work starts after #4395 (the authored Blocking and Deadlocks templates, which set the pattern) is on dev.
Split AlertNotebookEndpoint's authored templates into per-family files (
Add authored alert notebooks for Long-Running Query and Forced Plan F…
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling.
#4418 is merged (0dd9cfb). Long-Running Query and Forced Plan Failing alerts now open authored notebooks instead of the mechanical fallback.
Merge dev into the PostgreSQL alert notebooks (#4223)
Add authored notebook templates for the PostgreSQL-only alert families (
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling.
#4419 is merged (99aba09). Wraparound Risk, Vacuum Horizon Blocked and Replication Slot Retention alerts now open authored notebooks, each with a 24-hour gauge trend.
Merge dev into the Server and Agent notebooks (#4223)
Pin the pre-fetch gate and the eight-character finding match (#4223)
Add authored alert notebooks for Server Unreachable/Restored and Agen…
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling.
#4422 is merged (463667f). Server Unreachable/Restored and the Agent-job alerts now open authored notebooks, and the collection-log web read accepts a status filter.
Merge dev into the High CPU notebook (#4223)
Mark the authored-context live tests' store use (#4223)
5 remaining items
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling.
#4421 is merged (2b4f30e). Report alerts now link to the sweep timeline, or carry no link for the digests, and the notebook endpoint declares them as having no notebook. Ordinary alerts keep "Open triage page".
Merge dev into the Poison Wait notebook (#4223)
Add the authored Poison Wait alert-notebook template (#4424)
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling.
#4424 is merged (b476262). Poison Wait alerts now open an authored notebook, and the alert context carries the wait type. It shows the resource semaphore and memory grants for RESOURCE_SEMAPHORE, and a wait trend bound to the incident's wait type.
Merge dev into the authored-context step (#4223)
Add authored-notebook context plumbing (#4425)
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling.
#4425 is merged (a74e857). The shared context step for authored notebooks is in: context-bearing templates, prefix routing for Custom and Analysis metrics, and a pre-fetch that runs only for families that need it. No notebook's output changes. The Custom rules, Analysis findings and self-monitor templates build on it, one at a time.
Pin the registered context-family prefixes (#4223)
Add the authored Custom-rules alert-notebook template (#4433)
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling.
#4433 is merged (e0232a4). Custom-rule alerts now open an authored notebook. It charts the rule's own measure by the hour, with its thresholds as reference lines, and says plainly when the rule was deleted or its definition can't be read. Analysis findings are next, then self-monitor.
Add the authored analysis-finding alert-notebook template (#4436)
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling.
#4436 is merged (64ea4a2). Analysis-finding alerts now open an authored notebook with the finding's summary and its evidence, read with drill-down. A finding no longer in the store gives a plain note. The get_analysis_findings read's catalog entry now declares limit, include_drilldown and full_text. The self-monitor notebook is the last family.
Give Collection Stopped, Capture Down and Collector Cost Regression a…
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling.
#4437 is merged (13329d1), so every alert family now opens an authored notebook, and this issue is closing. The PRs, in merge order:
Metrics without an authored template still get the mechanical notebook.