Skip to content

Alert notebook MVP: #/triage renders a read-only notebook bound to the firing (blocking and deadlocks authored, every other family converted) #4222

Closed

Description

@erikdarlingdata

Part of #4219.

Problem

The alert link lands on #/triage (wwwroot/js/pages/triage.js, backed by GET /api/triage in DarlingTriageEndpoint.cs). That page doesn't frame the firing:

Sections are generic. Each section is one /api/read result, auto-rendered through derive.js (triage.js:71-97). No chart marks the alert instant, and nothing tells the reader whether the condition has cleared.

The firing is matched by nearest time. The history match takes the nearest same-metric row (DarlingTriageEndpoint.cs:612-659). The link's dedup is only echoed as a notice (triage.js:151-153), although each history row's context_json carries its incidents' dedup keys (Mcp/DarlingAlertReader.cs:48-51). So the page can't name the incident, its database, or its objects.

Sections run serially, and the page re-runs every minute.

The page can't become a notebook.

  • renderNotebookDoc renders only saved views (wwwroot/js/pages/views.js:487-540).
  • ValidateNotebookDefinition accepts only composed panel cells (DarlingWebEndpoints.cs:1658, root keys :1335-1338). The forensic reads an investigation needs (blocking chains, deadlock graphs) are v1 reads, which dashboards accept (:1561-1587) and notebooks reject.

Fix shape

1. GET /api/alert-notebook?server&metric&at&dedup (after the auth middleware, like every sibling) returns:

{ alert, status, notes[], template: { id, version }, definition: { kind: "notebook", cells: [...] } }
  • alert: the history row whose context_json has an incident with the link's dedup. Without one, fall back to today's nearest same-metric row. Include the row's incident facts (Incident Since, Involved Objects, Database, Total Occurrences).
  • status: "is it still happening":
    • a resolution row for this metric after at (use ResolutionAliases, DarlingTriageEndpoint.cs:99-135) reads "Resolved at T";
    • otherwise, later firings of the same metric read "Fired again at …";
    • otherwise the answer depends on collector freshness. If the relevant collector hasn't run since at, the status is "unknown (not collected since T)", never "ongoing". An absence of rows is not evidence while the instrument is down.
  • definition: a template bound server-side.

2. A read cell type in notebooks: {type:"read", read, params, viz, title}.

  • It's validated by the same v1 read-panel rules dashboards use (catalog read name, declared params).
  • renderCell routes it to renderPanel.
  • Without it, "Save as notebook" fails on exactly the cells that matter.

3. Templates live server-side, as the evolution of SectionsByMetric.

  • Authored for the MVP:
    • Blocking Detected / Blocking Wait Time:
      • header and status cells;
      • get_blocking (limit 20);
      • a blocked_process_reports timeline with a deadlock annotation;
      • most-blocked objects, by database, and lock modes, reusing the blocking-rca panel specs;
      • get_active_queries blocking_only.
    • Deadlocks Detected:
      • header and status cells;
      • get_deadlock_detail (limit 3);
      • the deadlocks timeline with a blocking annotation;
      • deadlocks by database, reusing the deadlock-postmortem specs;
      • get_deadlock_trend 24 h.
  • Every other metric converts mechanically: its current SectionsByMetric list (or fallback) becomes read cells under the same header and status cells. Every family gets the notebook shell, the dedup match and status from day one.
  • Each template carries an id and version.

4. #/triage renders the alert notebook through renderNotebookDoc, refactored to take an in-memory definition.

5. Cost.

  • Cells load client-side through /api/read and /api/compose/run, top to bottom, at most 3 in flight. That replaces the serial server-side assembly, and cells pass the render's AbortController signal (Web viewer: the 60 s poll and tab switches re-fire a page's reads while the previous render's reads are still running, and superseded reads are never cancelled #4191).
  • The triage route joins app.js's poll guard, so it doesn't re-render every 60 s. Only the status cell refreshes: one history read and one freshness read.
  • Template rules, enforced by pins:
    • every read cell has an explicit limit, or its trend goes through the chart bucket budget;
    • every window is at or under the family lookback;
    • no audit_config, and no analyze_*;
    • get_query_store_regressions, when present, is a collapsed click-to-run cell.
  • Nothing is computed or stored at fire time, as today.

Where

  • Darling/PerformanceMonitor.Darling.Service/DarlingTriageEndpoint.cs: map ~:167-334, fallback ~:381-385, ResolveAnchor ~:531-554, history match ~:612-659
  • DarlingWebEndpoints.cs:
    • ValidateNotebookDefinition ~:1658, root keys ~:1335-1338, per-cell range ~:1762
    • dashboard read-panel rules ~:1561-1587
    • /api/compose/run absolute window ~:991-1034
    • route map ~:322-324
  • wwwroot/js/pages/triage.js, wwwroot/js/pages/views.js (renderNotebookDoc ~:487, renderCell, panelOrError ~:744-767), wwwroot/js/notebook.js (blocking-rca ~:516, deadlock-postmortem ~:583), wwwroot/js/app.js (route ~:63-69, refresh ~:288-299)
  • Mcp/DarlingAlertReader.cs (context_json on history rows)

Pins

  • Every shipped alert template, bound against a fixed alert, passes ValidateNotebookDefinition. A renamed measure or read reds CI.
  • Every read cell names a read in BuildReadDispatch. This extends the existing map↔dispatch pin.
  • Budget pin over every template: limits and windows within caps, no barred reads, and Query Store regressions only as click-to-run.
  • Dedup match: among two same-metric rows 30 s apart with different incident dedup keys, the link's dedup selects the right row even when the other is nearer in time. With no dedup, today's nearest-row behaviour holds.
  • Status arms: resolution row after at gives resolved; later same-metric row gives fired again; neither plus a fresh collector gives no resolution recorded; neither plus a collector with no run since at gives unknown. The last one must not read "ongoing".
  • Binding: composed cells carry an absolute range equal to the computed window; read cells carry as_of = window_end. An at in the future clamps to now, as ResolveAnchor does today.
  • Notebook read cell: the validator accepts a valid read cell, rejects an unknown read or an undeclared param by cell index, and a saved notebook with read cells round-trips through /api/views.
  • Poll guard: refresh() doesn't call route() on the triage route. Copy the editor-route guard's pin shape.
  • Unauthenticated GET /api/alert-notebook returns the login response (or Web viewer: after a service restart, every open tab's API reads come back as the 200 HTML login page, which the page classifies as empty data #4187's 401) and issues zero store commands, on the Darling: a live HTTP test for both hosts' gates, including /core #4128 live-HTTP pipeline with a counting data source.

Out of scope

Authored templates for other families (slice D), link delivery (A), sign-in (B), user-editable override templates, and an MCP read returning the bound notebook.

Activity

  1. added
    enhancementNew feature or request
    client-siteOwned by the client-site agents (other laptop). Local sessions never pick these up.
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Sep 25, 2026
  2. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling

    Slice (d), the web viewer's shell-level pause/resume control for the 60 s auto-refresh (the triage page no longer re-renders on every tick): #4365, merged to dev (420033e). The alert-notebook endpoint (#4366), the notebook read cell (#4367) and the triage render (#4368) are still open, so this issue stays open.

  3. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling

    Slice (a2), the notebook read cell type (validated by the same rules as a dashboard's v1 read panel; those shared rules now also check a read panel's params on save, for dashboards too): #4367, merged to dev (ff0373b). The alert-notebook endpoint (#4366) and the triage render (#4368) are still open, so this issue stays open.

  4. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling

    Slice (a), GET /api/alert-notebook (an alert turned into a mechanical notebook: the dedup match nearest the alert, four status arms with a forward read for later resolutions, window binding, and read cells naming only dispatch reads): #4366, merged to dev (9aa1519). The triage render (#4368) is still open, so this issue stays open.

  5. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling

    Slice (c), #/triage renders the alert notebook read-only (current vs threshold and detail text in the header, at most 3 cells loading at once, results dropped after a route change, falling back to the old triage view on a 404): #4368, merged to dev (2797849). Slices (a), (a2), (c) and (d) are now on dev; the authored templates (b) remain, so this issue stays open.

  6. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    #4395 is merged. Blocking Detected, Blocking Wait Time and Deadlocks Detected now open an authored alert notebook: a timeline, ranked breakdowns and drill-down reads, bound to the alert's window and database. Every other alert type keeps the mechanical layout.

  7. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling

    Closing: the alert-notebook MVP has shipped.

    Follow-ups:

  8. removed
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    client-siteOwned by the client-site agents (other laptop). Local sessions never pick these up.enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions