Repository navigation
Lite investigation requests journal a quiet join and wait for every metric to resolve (#6010) - #6057
Merged
Merged
Conversation
added 7 commits
October 17, 2026 21:45
…etric to resolve (#6010)
erikdarlingdata
marked this pull request as ready for review
October 18, 2026 03:51
This was referenced Oct 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #6010.
Why
Part 1 (#6054) shipped Lite's evidence. This finishes the issue: a same-severity join sent the same
joinedas a real one, and nothing journaledresolved. Lite stays advise-only.What changes
investigation_requests.max_severity(shared added-columns list, plus the CREATE TABLE text).OpenOrJoinAsyncreads the request'smax_severityand whether the metric is already linked. A join with a new metric or a higher severity journalsjoined(detailnew_metric,severity); any other join journalsjoined_quiet. Uses the sharedInvestigationEventsconstants.InvestigationOpenResult.Eventedis set. A new request is stamped with its severity (null reads as warning).SweepAsyncjournalsresolved(once keyresolved:<id>) for a live request when every linked metric has a resolution row after its last alert on the same server, in its own transaction after the expiry.LiteInvestigationRequestEventsTests(quiet join, resolved, v73 on new and existing DB, second run no change). Updated one old pin (AlertsGroup_...): same-metric joins now count asjoined_quiet. The v72 test now expects stamp 73.How Lite stores resolution rows
Match checked against the code, not assumed. Lite's alert engine writes no history row on a clear today (
ShowAlertResolutionToastAsyncis toast and log only), so there is no resolution row to match in a live Lite database yet. The sweep readsv_config_alert_log(live plus archive) rows whosecontext_jsonhas a top-levelResolvesmember naming the firing metric, the same rule as Darling. Until a Lite writer records such a row,resolvedcannot fire in the app; the tests seed the row shape directly. Round 2 (below) writes that row.Round 2: Lite records the resolution row
Moved, not twinned.
BuildResolutionRecord,ResolutionContextJsonand theResolvesconstant now live inPerformanceMonitor.Alerting/AlertResolutionRecords.cs. Both products already reference that project, and it pulls in no Darling-only type (onlyAlertResolution,AlertHistoryRecord,AlertDelivery,SensitiveStatements). Darling'sDarlingSelfAlertEvaluator.BuildResolutionRecord,ResolutionContextJsonandResolvesMembernow forward to it, so every Darling call site and test is unchanged and the two shapes cannot drift. Lite'sDuckDbInvestigationStore.ResolvesMemberpoints at the shared constant.Lite writer. New
Lite/Services/LiteAlertResolutionRecorder.cs.MainWindow.xaml.cswiresLiteAlertResolutionRecorder.Callback(_alertHistoryStore, ShowAlertResolutionToastAsync)as the engine'sresolutionCallback. It writes the row first, catches and logs a failure, then runs the old toast and log line. Lite sends nothing to email or webhooks for a resolution.Row shape (same as Darling): metric = the resolution title, value
resolved,notification_typenone,alert_sentfalse,mutedfalse, masked message,context_jsonwithResolves= the firing metric.What a resolution row does in each Lite reader of
config_alert_log/v_config_alert_logDuckDbAlertHistoryStore.RecordAlertAsync(writer)GetLastEmailSentUtcAsync/GetLastWebhookSentUtcAsync(cooldown seeds)notification_typeemail/webhook, sononenever matches. Not a send.GetLastAlertTimeAsync(digest-excluded seed)CPU Resolved), never the firing metric's name, so it cannot cool down a fire.GetLastDeliveredPageUtcAsyncalert_sent; false. Not a delivered page.GetLastFiredTierAsync(escalation seed)resolutionMetricNameread). That is the designed use, same as Darling.LocalDataService.AlertHistory(history tab,get_alert_history)IsResolvedtrue (title is a resolution by name),StatusDisplay= no channel,alert_sentfalse, severityresolution.LocalDataService.AlertHistory)LocalDataService.DailySummaryalert_countNOT LIKE '%Resolved%/%Cleared%'...list, which covers every title the shared engine emits. Not counted as a fired alert.DuckDbInvestigationBriefSourceReaderDuckDbInvestigationStore.SweepAsyncResolvesmatch journalsresolved.ArchiveServiceEmailAlertService/LiteAlertDelivererNo reader needed a code change, so there is no per-reader "not counted" pin beyond the shared ones below. Note: the daily-summary SQL suffix list lacks
Renewed, whichAlertMetricClassifier.IsResolutionhas. Lite's engine emits noRenewedtitle (only Darling's TLS self-alerts do), so it is not reachable here and I left it.Pins (all behaviour, in
LiteAlertForwardingTests.ResolutionRow.cs)TheResolutionCallback_WritesOneHistoryRow_NamingTheMetricItResolvesAFailedHistoryWrite_NeverCostsTheToastHalfEndToEnd_AnAlertOpensARequest_ItClears_AndTheNextSweepJournalsResolvedOnce: real engine fires High CPU and clears it, real recorder writes the row, a request opened for that metric journalsresolvedonce on the next sweep and not again.EveryReaderOfTheAlertLog_ReadsAResolutionRowAsAClearAndNotAsAFire: escalation seed, email/webhook/delivered-page seeds, history read.TheDailySummaryAlertCount_CountsTheFire_AndNotItsResolutionRow(fixed past day, holds at any hour).Proved to fail on the old code: with the recorder's write replaced by a no-op (the old toast-only callback), VM run at
6fed54744(throwaway branch, deleted):Total: 51, Failed: 2(the callback pin and the end-to-end pin). The reader pins describe behaviour the old code also had for a hand-seeded row; they guard it now that the app writes the row.Existing pins updated
AlertReadFailureSurfaceTests: the "resolution context that is not JSON" catch moved out ofDarlingSelfAlertEvaluator.csinto the shared file, which that census does not scan. Per-file exempt 21 to 20, tree exempt total 44 to 43, exemption entry removed.AlertDelivererInventoryPinTests: theRecordAlertAsyncregex already allowsBuildResolutionRecordas a reviewed argument; it now also allowsAlertResolutionRecords.BuildResolutionRecord.Lite/Services/LiteAlertResolutionRecorder.cscalls it that way.Test plan
Total: 4, Failed: 3at5c126e93c. Round 2:Total: 51, Failed: 2at6fed54744(recorder write removed).88d173bde, Lite.Tests, 24 classes (every classgrep -lnames forDuckDbInvestigationStore,MainWindow.AlertEngine/MainWindow.xaml, plusLiteAlertForwardingTests,EscalationSeedHistoryStoreTests,DailySummaryHighCpuTests):Total: 407, Failed: 088d173bde, Darling.Tests, 71 classes (every classgrep -lnames forDarlingSelfAlertEvaluator,AlertEngine.cs,AlertResolutionRecords, includesAlertReadFailureSurfaceTests,AlertDelivererInventoryPinTests):Total: 2223, Failed: 0, Skipped: 83. The first run atbfa94e7cehad 1 failure (the inventory regex above), fixed.VM_PG=1at88d173bde:InvestigationRequestEventsLiveTests,AlertNotebookResolutionReadLiveTests:Total: 41, Failed: 0origin/devmerged atbfa94e7ce88d173bde(round 2):Darling.Tests Total: 34065, Errors: 0, Failed: 0, Skipped: 3906andLite.Tests Total: 10672, Errors: 0, Failed: 0, Skipped: 5. Round 1 at4c381a754:Lite.Tests Total: 10667, Failed: 0.Build warnings: none.
CHANGELOG entry
SECTION: Added
ENTRY: - Lite investigation requests journal a quiet join and wait for every metric to resolve ([#6010]) - A join that brings no new metric and no higher severity is now journaled as a quiet join instead of
joined, and a request isresolvedonly when every linked metric has resolved. Lite schema v73 addsmax_severity.REF: [#6010]: #6010
Deferred and follow-ups
Deferred (not ported, 3): event delivery to an agent route, the unclaimed fallback, and the MCP long-poll wake. Lite passes
opened: nullto dispatch, has nowait_for_investigation_requeststool and no agent-route delivery, so they have nothing to attach to. Also the delete of branchproof/6010-old-code(left for the seat).Follow-ups proposed (1): add
Renewedto the Lite daily-summary suffix list so it matchesAlertMetricClassifier.IsResolution(not reachable today).