Skip to content

Lite investigation requests journal a quiet join and wait for every metric to resolve (#6010) - #6057

Merged
erikdarlingdata merged 7 commits into
devfrom
fix/6010-lite-events-part2
Oct 18, 2026
Merged

erikdarlingdata merged 7 commits into
devfrom
fix/6010-lite-events-part2

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

Closes #6010.

Why

Part 1 (#6054) shipped Lite's evidence. This finishes the issue: a same-severity join sent the same joined as a real one, and nothing journaled resolved. Lite stays advise-only.

What changes

  • Schema v73 adds investigation_requests.max_severity (shared added-columns list, plus the CREATE TABLE text).
  • Quiet join (D6): OpenOrJoinAsync reads the request's max_severity and whether the metric is already linked. A join with a new metric or a higher severity journals joined (detail new_metric, severity); any other join journals joined_quiet. Uses the shared InvestigationEvents constants. InvestigationOpenResult.Evented is set. A new request is stamped with its severity (null reads as warning).
  • Resolved (D7): SweepAsync journals resolved (once key resolved:<id>) for a live request when every linked metric has a resolution row after its last alert on the same server, in its own transaction after the expiry.
  • Tests: new LiteInvestigationRequestEventsTests (quiet join, resolved, v73 on new and existing DB, second run no change). Updated one old pin (AlertsGroup_...): same-metric joins now count as joined_quiet. The v72 test now expects stamp 73.

How Lite stores resolution rows

Match checked against the code, not assumed. Lite's alert engine writes no history row on a clear today (ShowAlertResolutionToastAsync is toast and log only), so there is no resolution row to match in a live Lite database yet. The sweep reads v_config_alert_log (live plus archive) rows whose context_json has a top-level Resolves member naming the firing metric, the same rule as Darling. Until a Lite writer records such a row, resolved cannot fire in the app; the tests seed the row shape directly. Round 2 (below) writes that row.

Round 2: Lite records the resolution row

Moved, not twinned. BuildResolutionRecord, ResolutionContextJson and the Resolves constant now live in PerformanceMonitor.Alerting/AlertResolutionRecords.cs. Both products already reference that project, and it pulls in no Darling-only type (only AlertResolution, AlertHistoryRecord, AlertDelivery, SensitiveStatements). Darling's DarlingSelfAlertEvaluator.BuildResolutionRecord, ResolutionContextJson and ResolvesMember now forward to it, so every Darling call site and test is unchanged and the two shapes cannot drift. Lite's DuckDbInvestigationStore.ResolvesMember points at the shared constant.

Lite writer. New Lite/Services/LiteAlertResolutionRecorder.cs. MainWindow.xaml.cs wires LiteAlertResolutionRecorder.Callback(_alertHistoryStore, ShowAlertResolutionToastAsync) as the engine's resolutionCallback. It writes the row first, catches and logs a failure, then runs the old toast and log line. Lite sends nothing to email or webhooks for a resolution.

Row shape (same as Darling): metric = the resolution title, value resolved, notification_type none, alert_sent false, muted false, masked message, context_json with Resolves = the firing metric.

What a resolution row does in each Lite reader of config_alert_log / v_config_alert_log

Reader What the row does Changed
DuckDbAlertHistoryStore.RecordAlertAsync (writer) Writes it as-is. No
GetLastEmailSentUtcAsync / GetLastWebhookSentUtcAsync (cooldown seeds) Filter on notification_type email/webhook, so none never matches. Not a send. No
GetLastAlertTimeAsync (digest-excluded seed) Matches only the resolution's own title (CPU Resolved), never the firing metric's name, so it cannot cool down a fire. No
GetLastDeliveredPageUtcAsync Needs alert_sent; false. Not a delivered page. No
GetLastFiredTierAsync (escalation seed) Reads it as the clear that closes the fire (its resolutionMetricName read). That is the designed use, same as Darling. No
LocalDataService.AlertHistory (history tab, get_alert_history) Shown. IsResolved true (title is a resolution by name), StatusDisplay = no channel, alert_sent false, severity resolution. No
Dismiss / dismiss-all / verify (LocalDataService.AlertHistory) Act on rows by time, server, metric. A resolution row can be dismissed like any row. No
LocalDataService.DailySummary alert_count Excluded by the NOT LIKE '%Resolved%/%Cleared%'... list, which covers every title the shared engine emits. Not counted as a fired alert. No
DuckDbInvestigationBriefSourceReader Reads only rows the request linked, joined by metric name and time. A resolution row is not linked, so it never appears as an alert. No
DuckDbInvestigationStore.SweepAsync The reader this round feeds: Resolves match journals resolved. Constant only
ArchiveService Moves rows by age. Same for all rows. No
EmailAlertService / LiteAlertDeliverer Write fire rows only. No

No reader needed a code change, so there is no per-reader "not counted" pin beyond the shared ones below. Note: the daily-summary SQL suffix list lacks Renewed, which AlertMetricClassifier.IsResolution has. Lite's engine emits no Renewed title (only Darling's TLS self-alerts do), so it is not reachable here and I left it.

Pins (all behaviour, in LiteAlertForwardingTests.ResolutionRow.cs)

  • TheResolutionCallback_WritesOneHistoryRow_NamingTheMetricItResolves
  • AFailedHistoryWrite_NeverCostsTheToastHalf
  • EndToEnd_AnAlertOpensARequest_ItClears_AndTheNextSweepJournalsResolvedOnce: real engine fires High CPU and clears it, real recorder writes the row, a request opened for that metric journals resolved once on the next sweep and not again.
  • EveryReaderOfTheAlertLog_ReadsAResolutionRowAsAClearAndNotAsAFire: escalation seed, email/webhook/delivered-page seeds, history read.
  • TheDailySummaryAlertCount_CountsTheFire_AndNotItsResolutionRow (fixed past day, holds at any hour).
    Proved to fail on the old code: with the recorder's write replaced by a no-op (the old toast-only callback), VM run at 6fed54744 (throwaway branch, deleted): Total: 51, Failed: 2 (the callback pin and the end-to-end pin). The reader pins describe behaviour the old code also had for a hand-seeded row; they guard it now that the app writes the row.

Existing pins updated

  • AlertReadFailureSurfaceTests: the "resolution context that is not JSON" catch moved out of DarlingSelfAlertEvaluator.cs into the shared file, which that census does not scan. Per-file exempt 21 to 20, tree exempt total 44 to 43, exemption entry removed.
  • AlertDelivererInventoryPinTests: the RecordAlertAsync regex already allows BuildResolutionRecord as a reviewed argument; it now also allows AlertResolutionRecords.BuildResolutionRecord. Lite/Services/LiteAlertResolutionRecorder.cs calls it that way.

Test plan

  • Mac build, Darling.Tests and Lite.Tests: 0 warnings, 0 errors. Round 1 items below still hold.
  • Pins fail on old code (round 1): Total: 4, Failed: 3 at 5c126e93c. Round 2: Total: 51, Failed: 2 at 6fed54744 (recorder write removed).
  • VM at 88d173bde, Lite.Tests, 24 classes (every class grep -l names for DuckDbInvestigationStore, MainWindow.AlertEngine/MainWindow.xaml, plus LiteAlertForwardingTests, EscalationSeedHistoryStoreTests, DailySummaryHighCpuTests): Total: 407, Failed: 0
  • VM at 88d173bde, Darling.Tests, 71 classes (every class grep -l names for DarlingSelfAlertEvaluator, AlertEngine.cs, AlertResolutionRecords, includes AlertReadFailureSurfaceTests, AlertDelivererInventoryPinTests): Total: 2223, Failed: 0, Skipped: 83. The first run at bfa94e7ce had 1 failure (the inventory regex above), fixed.
  • VM VM_PG=1 at 88d173bde: InvestigationRequestEventsLiveTests, AlertNotebookResolutionReadLiveTests: Total: 41, Failed: 0
  • origin/dev merged at bfa94e7ce
  • Seat full suite at 88d173bde (round 2): Darling.Tests Total: 34065, Errors: 0, Failed: 0, Skipped: 3906 and Lite.Tests Total: 10672, Errors: 0, Failed: 0, Skipped: 5. Round 1 at 4c381a754: Lite.Tests Total: 10667, Failed: 0.

Build warnings: none.

CHANGELOG entry

SECTION: Added
ENTRY: - Lite investigation requests journal a quiet join and wait for every metric to resolve ([#6010]) - A join that brings no new metric and no higher severity is now journaled as a quiet join instead of joined, and a request is resolved only when every linked metric has resolved. Lite schema v73 adds max_severity.
REF: [#6010]: #6010

Deferred and follow-ups

Deferred (not ported, 3): event delivery to an agent route, the unclaimed fallback, and the MCP long-poll wake. Lite passes opened: null to dispatch, has no wait_for_investigation_requests tool and no agent-route delivery, so they have nothing to attach to. Also the delete of branch proof/6010-old-code (left for the seat).
Follow-ups proposed (1): add Renewed to the Lite daily-summary suffix list so it matches AlertMetricClassifier.IsResolution (not reachable today).

@erikdarlingdata
erikdarlingdata marked this pull request as ready for review October 18, 2026 03:51
@erikdarlingdata
erikdarlingdata merged commit e8955cd into dev Oct 18, 2026
50 of 54 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/6010-lite-events-part2 branch October 18, 2026 04:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant