Confidence chooses the channel: uncorroborated findings route to digest + web, corroboration pages (#3712)

[erikdarlingdata]

The lie

Confidence has measured corroboration since #3538 A6, and every finding that crossed the severity floor still took the same road to the same channel. Severity said "this is big"; the channel heard "page a human". The first day the recalibrated engine ran on a large production fleet: forty-plus anomaly pages in seven hours, severity 1.5–1.9, confidence 0.20–0.35, interleaved with the handful that actually needed a human, and the operator's own reading of the channel was no idea what to focus on. That is alert fatigue as a mechanism — not wrong alerts but unranked ones, where the cost lands on the true positives that stop being seen. The batch itself made the case sharper: the low-confidence pages were largely true and redundant — anomaly stories attached to a CPU flap already paging through the CPU family, to a server inside its own maintenance job, to a burst the platform alert had already delivered. Uncorroborated single-fact stories are, empirically, mostly echoes.

The mechanism

Confidence chooses the channel, and it reads the corroboration COMPONENTS, never the scalar (design point 1). FindingRouting.Classify (PerformanceMonitor.Notifications — one pure function both SKUs consult) pages a finding when any of these holds, and its reason names which:

• two or more facts in the story chain (FactCount ≥ 2 — each traversed edge is a fired predicate onto a fact that itself scored);
• a matched co-fire check on the root fact (MatchedAmplifiers ≥ 1 — every amplifier the scorer defines is a predicate over ANOTHER fact: a sibling anomaly in the same window, an absolute fact at its concerning bar, a wait that is significant; one match is one corroborating fact the engine went looking for and found);
• one of the two by-construction 1.0 stories, named by root key (absolution; the same-statement pileup, a directly observed convoy).

Everything else is a lone fact and takes the digest road under the shipped default. The two components the scalar folds together now ride the story and the finding as their own numbers — AnalysisStory/AnalysisFinding.MatchedAmplifiers and .DefinedAmplifiers, read in InferenceEngine.BuildStory where the root fact is in hand, copied across by both finding stores, ephemeral like RootFactMetadata. The inverse of the formula is exact today (a one-fact story above 0.20 has a matched check) and stops being exact the day a weight moves; a paging bar that silently moves with a formula constant is what design point 1 forbids.

The extremity escape keeps its severity power; its published intent becomes the enforced bar. FactScorer.IsExtremeAnomaly (#3526) is a private predicate that writes NO marker onto the fact or the finding — the brief's hypothesis that the escape marks something it routes on was checked at the source and is false — so the gate cannot see extremity and does not try to. An extreme single still clears the floor and still leads its story; with no matched check it is still a single and lands in the digest with its severity intact, which is exactly "extreme and corroborated" as the bar.

AnalysisNotificationService.NotifyAsync takes two roads. The severity floor stays upstream of the gate — a below-floor finding is not routed at all. Each notify-worthy incident is classified member by member: an incident with any page-routed member takes the page road, led by a PAGE member even when the single outranks it, and the single is named on the page as co-fired — the "true and redundant" shape the live batch was made of, accounted for exactly as before. An incident of uncorroborated singles takes the digest road: the same FindingAlert to the same IFindingAlertSender, with Route = Digest; the sender consults NO channel (no send core, no webhook fan-out, no tray) and writes the ledger row with the new AlertDelivery.RoutedToDigest() disposition (notification_type = 'digest', Sent = false as a measurement). The tray is a paging channel — it interrupts — so the digest road raises nothing there either.

The ledger records the decision the way it records the fired tier. AlertContext.Routing (AlertRoutingDto(Route, Reason)) is a trailing nullable member of the context JSON, set on BOTH roads — a page says why it earned the channel, a digest entry says why it did not — read off the row by AlertContextSerializer.TryReadRouting and published by get_alert_history on both SKUs as routing / routing_reason beside severity_source. Null on every engine alert and every pre-#3712 row, and the descriptions say so. The web alerts page carries routing_reason as the status cell's title; util.js / AlertDeliveryStatus label the disposition Digest.

Escalation is the corroboration event (design point 2). The route is a key component of the re-notify bucket: the digest road cools under a digest| namespace prefix and the page namespace keeps its pre-#3712 key text byte for byte, so a story that gains a fact (a new hash) or a matched check (the SAME hash, a flipped route) finds no page bucket and pages as a NEW firing. Across a restart the page seed is the store's GetLastAlertTimeAsync, and both stores (PG, DuckDB) now EXCLUDE digest rows from it — spelled through AlertDelivery.ChannelDigest, not a retyped literal — because a seed that read a digest row would hold the escalation as a repeat of a page that never happened. The digest namespace is deliberately not seeded: there is no route-filtered seed on the sender seam, the cost is bounded and channel-free (one re-recorded ledger row per standing single per restart), and the digest reads distinct stories over its span rather than counting rows.

The Analysis Singles Digest is the third daily document beside the Collector Cost Digest and the Fleet Sweep Rollup, seam for seam: family reports, declared INFO arm, one post per trailing day, the #3580 delivered-today stamp under its own key, empty day posts nothing without consuming the interval, a failed read skips the tick and is counted into the #3013 census (34 sites now). Its SOURCE is the ledger (AnalysisSinglesDigestReader, notification_type = 'digest' over config.config_alert_log), because a digest-routed ledger row exists for exactly the findings the gate decided about and already carries the reason and the #1140 fingerprints. The document: "Anomaly singles: N across M servers", the top movers by severity — each with the KEY a page would have carried (Analysis: <category> [<hash8>], the ledger metric_name and the story hash get_analysis_findings resolves), its reason and its dedup fingerprints (design point 3) — then by family, by server, every cap stating its remainder. Rows are deduplicated per (server, story) at the newest severity so a restart's second row counts once.

One knob, analysis.uncorroborated_route = digest | page, default digest, on IAlertSettings.UncorroboratedFindingRoute as a defaulted interface member (the NotificationRoutes precedent: the default IS the ruling). Lite: a live setting — App.AnalysisUncorroboratedRoute, analysis_uncorroborated_route in the settings file, a Settings → Alerts checkbox, get_alert_settings. Darling: analysis.uncorroboratedRoute in darling.json, file-level like web.publicBaseUrl — carried across StoreConfigProvider's wholesale Analysis swap from the held config (the BuildServerFromRow backfill shape), published to an ambient static by both config-loading hosts so get_alert_settings reports the value the running process applies rather than a constant, refused by name by update_alert_settings with the file key (a round-tripped unchanged value is accepted with a warning). Documented in a new ### analysis section of Darling's README and in Lite's Alert Behavior list.

Lite's grid says what was not delivered. FindingRoutingLens (notify floor + knob) judges a PERSISTED finding through FindingRouting.ClassifyPersisted, which answers three arms exactly from the persisted fields and re-derives the matched-check arm from the scalar — saying so in its reason, and pinned to agree with the write-side decision over the whole small-integer catalogue domain. A notify-worthy single reads Not paged — uncorroborated: … on its Recommendations card; a delivered finding and a below-floor one carry no line.

What it does NOT do

• It does not touch FactScorer, AnomalyGate, the reconciler or any PgTarget* file — the fence held. Severity is unchanged; only the channel a severity reaches is decided here.
• It adds no store column and no migration rung on either SKU: the routing record lives in the context JSON, the digest document reads the ledger, and Darling's knob is file-level for exactly that reason. The store column + Viewer Settings surface for the Darling knob are the residual (item 5 of the brief asked for "config table + viewer Settings + MCP"; the MCP half ships, the other two wait for a rung window and a config_alert_settings column, at which point the carry in StoreConfigProvider becomes a reader.GetString and update_alert_settings gains the field).
• It does not seed the digest namespace across a restart (stated above, with the bounded cost).
• It does not change what get_analysis_findings returns: digest-routed findings are persisted by the same InsertFindingsAsync path they always were, upstream of the notification service; the live pipeline test asserts the sender still receives them.
• Nothing hides a finding anywhere. Nothing is suppressed; nothing interrupts.

Evidence and tests

Executed on this Mac (mactest harnesses over the built assemblies):

• Lite.Tests/FindingRoutingTests.cs (new, 39 tests, Notifications-only harness): every Classify arm in order with its reason; the knob moves only the uncorroborated arm; ClassifyPersisted ≡ Classify over path 1–11 × catalogue 0–12 × both knobs; the legacy path-shape 1.0 reads uncorroborated; the routing record round-trips and reads off the row alone (and a foreign shape reads null); the digest disposition is state-carrying, never Sent, labelled Digest on both SKUs; through NotifyAsync: a single takes the digest road with the reason on the context and no toast, a chain pages and toasts, a below-floor single is not routed, the page knob pages a single with the knob named in the reason, a mixed incident is led by the page member with the single co-fired, an all-single incident writes ONE digest row, a single that gains a matched check on the same hash pages as a new firing, a seeded page bucket still holds (which is why the stores exclude digest rows), the digest road never consults the seed. Red-first: emptying the digest| prefix reds exactly the escalation pin; raising the matched-check arm to ≥ 2 reds five.
• Darling.Tests/AnalysisSinglesDigestTests.cs (new, 22 tests): extraction (distinct counts, top-movers order, restart-row collapse at newest severity, family-then-server grouping, FamilyOf shapes, unparseable context kept), the rendered document (headline, span, ranked movers with key/reason/fingerprints, family blocks, remainders stated), the lifecycle (metric/key/label/INFO/count, interval, an earlier process's stamp gates, failed delivery writes no stamp and retries, empty day posts nothing, master gate, mute seam), the taxonomy registration, the reader's predicate, the seed exclusion spelled through the constant.
• DarlingMcpAlertToolsTests.UncorroboratedRoute_IsReportedFromThePublish_WithItsNote_AndTheWriterRefusesToChangeIt (new): unpublished → null + note, null coming home claims nothing; a value with nothing published is refused with the file key; published → wire spelling; the same value coming home → one warning, no column; a different value → refused; an unparseable file value publishes nothing; both descriptions say it.
• Whole Darling.Tests on the Mac against a scratch TimescaleDB (DARLING_TEST_PG set, live tests lit): 8419 total, 255 failed, 54 skipped — versus 8241 / 254 / 54 on pristine dev at the same HEAD-minus-this-PR, and the failure SETS are identical (the platform reds: Windows paths, deploy, repo-root resolution); the +1 is PgTargetWaitTests.TheWalWaits_…, which is red on dev's own HEAD CI (run on 1acd6062, job Darling PostgreSQL tests) — a PostgreSQL I/O family: an operator now learns how slow each data-file read was, on measured Aurora-storage bars, and hears why when the pass cannot know — timing off or a pre-16 major (partial #3691) #3726/Replication lag and slot retention stories for the PostgreSQL-target engine: a standby's stage and drift, a slot's disk-fill and horizon, on the alert's own bars (partial #3691) #3727 lane's test, fenced, not touched here. Live: DarlingAnalysisPipelineTests.EndToEnd_FullAnalysisPipeline_AgainstDevPostgres now also asserts the components rode BuildStory → PgFindingStore (ANOMALY_WAIT_PROFILE defines five checks) and the gate's decision rides the alert; DarlingAlertingTests.PgAlertHistoryStore_GetLastAlertTimeAsync_ExcludesDigestRoutedRows_FromThePageSeed (new) proves the exclusion against Postgres in both directions and was proven red by blanking the filter.
• Census pins re-pinned and executed: AlertReadFailureSurfaceTests (10 counted in the evaluator, 34 tree-wide), NotificationRoutingTests (family census names the third document), AlertTrayStatusLoggedTests (util.js ↔ constants), AlertDeliveryChannelTests, AlertMetricClassifierTests, AlertHistoryRowSeverityTests, DarlingConfigTests.

CI-first (net10.0-windows only): Lite.Tests/LiteRecommendationsReaderTests.MapFinding_MarksANotifyWorthyUncorroboratedSingle_AndNothingElse (new), the Lite AlertSeverityTests / AlertReadFailureSurfaceTests / McpAlertSettingsKeyTests re-pins (the Darling↔Lite analysis key parity was replayed against both sources by hand: seven keys each side).

Every touched project builds -c Release -p:EnableWindowsTargeting=true --no-incremental with 0 warnings (the one xUnit2000 in Lite.Tests/LogTailOverlapThresholdPinTests.cs is dev's, from 40b13d5, untouched here).

Partial #3712 — residual: the Darling knob's store column and Viewer Settings surface (rung-fenced).