Skip to content

Request events: journal event log, resolved detector, fallback delivery, MCP cursor and wake (#5632 Phase C C7) #6003

Closed

Description

@erikdarlingdata

Claude posting for Erik Darling.
Part of #5632 (Phase C, wave 2).

What it does

The journal becomes the event log and two readers consume it: a signed POST to the agent route and the MCP long-poll. Four parts: the event writer, the resolved detector, fallback delivery for an expired request, and the MCP cursor and wake.

Events: opened; joined only for a new metric or a higher severity; resolved; expired and claims_exhausted; evidence_ready (from #6001); follow_up (from #6008); deadlock_digest (from #6005). Each event row gets an event_id uuid, sent as webhook-id. The journal's once_key writes each event once. max_severity on the request decides whether a join is worth an event.

First check, an assumption the plan flagged (D7): resolution rows regardless of the route's SendResolutions flag. resolved reads resolution rows, not the route's SendResolutions flag, on the belief that a resolution row is written for every alert clear even when the route sends nothing. The doc comment on DeliverResolutionAsync says a clear with no delivered fire writes "today's none row" and a mute writes a muted row, which supports it. Before building the detector, prove it with a test on a route whose SendResolutions is off, a muted alert, a digested one and a throttled one: each must leave a resolution row the detector can read, and that row must tie back to the request's linked metric and server. If any case leaves no row, say so in the PR body and stop on that case rather than reading the flag.

Decisions this slice carries (agreed by Erik on 2026-10-16)

  • D5 (A). "The journal is the event log; a signed POST to the agent route and the MCP long-poll both read it. The journal's once_key writes each event once."
  • D6. "opened; joined only for a new metric or a higher severity (every join is noise); resolved; expired and claims_exhausted; evidence_ready (D9); follow_up (D16); deadlock_digest (D13). Ids: a uuid per event row, sent as webhook-id", "with a max_severity column on the request."
  • D7 (A). "Every metric linked to it has a resolution row after its last alert. Exact. It reads resolution rows, not the route's SendResolutions flag." (B, the first metric resolved, "fires early on a request that grew.")
  • D8 (A). "A fallback_route_id on the agent route; else the same route's human channels; else a self-monitor alert."

Files to touch

Pointers re-checked on dev at c12233ce3.

  • Darling/PerformanceMonitor.Darling.Storage/InvestigationRequestStore.cs:331-345 (SweepExpireSql: journals expired and claims_exhausted with once_key expired:<request_id> and delivers nothing; the sweep call is at line 974).
  • PerformanceMonitor.Common/Investigations/InvestigationWebhookToken.cs:42 (opened ? "opened" : "joined", rendered only on the alert's own page; callers PerformanceMonitor.Notifications/InvestigationDispatchCore.cs:236 and :245).
  • The long-poll: Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpInvestigationTools.cs:126-143 (wait_for_investigation_requests), woken only by Darling/PerformanceMonitor.Darling.Service/Investigations/InvestigationOpenSignal.cs:64 (Opened(routeId)), fired by InvestigationDispatcher.cs (doc at lines 63-64). The wake must also fire for the new events.
  • Resolution rows: Darling/PerformanceMonitor.Darling.Service/DarlingAlertDeliverer.cs:414-460 (DeliverResolutionAsync; the send at 437, SendResolutionAsync at 452); NotificationRouteResolutions.SendsResolutions; the record is built by DarlingSelfAlertEvaluator.BuildResolutionRecord (Darling/PerformanceMonitor.Darling.Service/DarlingSelfAlertEvaluator.cs:82).
  • The journal table: Darling/PerformanceMonitor.Darling.Storage/InvestigationTables.cs:174-190 (once_key unique index at 190); the linked metrics: investigation_request_alerts at 121.
  • Signing and the generic POST come from Signed webhooks: Standard Webhooks signatures, key rotation, key screens, receiver docs (#5632 Phase C C2) #5998; the columns from Phase C store: the one migration, grants and retention (#5632 Phase C C1) #5997. The fallback self-monitor alert goes through DarlingSelfAlertEvaluator.

Depends on

#5997 (event id, max_severity, fallback_route_id) and #5998 (the signed POST).

Behaviour to pin

  • A same-severity join sends nothing.
  • A higher-severity join sends joined.
  • resolved waits for every linked metric to have a resolution row after its last alert.
  • A route with SendResolutions off still resolves the request (D7's check).
  • An unclaimed request that expires reaches the fallback route; with no fallback route it reaches the same route's human channels; with neither, a self-monitor alert is raised.
  • A replayed event id is not delivered twice.
  • The MCP long-poll wakes for a new event and resumes from its cursor without repeats.

Tests

Behaviour pins only, one test per behaviour listed above. No census, roster or source-text test (the #5833 ruling in .github/REVIEW-TRAPS.md).

Notes

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions