Maintenance-family anomalies fold onto the fired job's incident and name the job, so a server's own index-maintenance job no longer pages four times without saying its name (Fixes #3704)

[erikdarlingdata]

A maintenance window's sub-threshold anomalies now fold onto the incident that carries the fired RUNNING_JOBS, and each one names the job

The lie it stops telling

One dogfood server ran its own nightly index-maintenance Agent job for 116 minutes (a REORGANIZE … LOB_COMPACTION chain, LIMITED physical-stats scans, UPDATE STATISTICS; WRITELOG / HADR_SYNC_COMMIT / PAGEIOLATCH_SH waits). The analysis pass over that window produced four separate ANOMALY_* stories for the server — all one-fact stories, severity 1.5–1.9 — and delivered each as its own page. Not one of them said the job's name. #3538 A9 (PR #3632) had already ruled that a maintenance window's cards are ONE incident naming the job; the ruling did not reach these cards.

Why, at source

#3632 added three edges to RelationshipGraph.BuildMaintenanceEdges — SCH_M, IO_WRITE_LATENCY_MS and WRITELOG onto RUNNING_JOBS, each gated on the job having FIRED (BaseSeverity > 0: at least one job running past its own history; "mere presence would not do"). Those edges are keyed on the regular symptom facts. The anomaly path is separate by design: AnomalyIncidentReconciler ("fold, don't suppress, don't add graph edges" — the Round-2 vetted shape) rewrites an anomaly story's IncidentId onto the regular story whose Path carries the anomaly's mapped family key (ANOMALY_WRITE_LATENCY → IO_WRITE_LATENCY_MS; ANOMALY_WAIT_PROFILE → the family of its dominant contrib_*).

An anomaly is exactly the sub-threshold case: write latency or log flushes elevated against the server's own hour-of-week baseline while the absolute threshold held. So the regular symptom did NOT fire, there was no regular story to fold onto, and nothing in the reconciler or the graph connected the anomaly to the RUNNING_JOBS fact that had fired beside it. Four anomalies, four incidents, none naming the job that caused all of them.

What it does now

A conditional entry in the fold-target set, not a graph edge. When an anomaly's resolved family is one of the three maintenance symptoms (AnomalyIncidentReconciler.MaintenanceFamilies = SCH_M, IO_WRITE_LATENCY_MS, WRITELOG — the edge SOURCES of #3632), RUNNING_JOBS joins its candidate list LAST. The reconciler's index is already keyed on every member of a regular story's Path, so the anomaly folds onto the regular story that consumed the job: the job-rooted story, or a symptom story that reached it through #3632's edge. Its own family stays the first candidate; when the job fired, that story consumed the job through the edge, so both routes land on one incident.

The fired-gate is structural, not re-evaluated. A story carries RUNNING_JOBS in its Path only when the fact scored above zero — InferenceEngine.BuildStories roots at 0.5 or better, the edges' predicate is BaseSeverity > 0, and FactScorer.ScoreAll zeroes the severity of anything with base ≤ 0 — and FactScorer.ScoreJobFact scores the running-long count on a (1, 3) ramp, so ONE job running long is base 0.5: both "fired" and "roots a story". A present-but-quiet job scores 0, drops out of the working set, roots nothing and is consumed by nothing, so no story carries it and the anomalies are exactly as they were. This is the same predicate #3632 uses, seen through the story set instead of the fact set. It also answers the brief's open question: there is no "the job fired but no story carries it" case for the reconciler to mint an incident for, so no incident is minted — the fold onto a real story is the whole shape, and the end-to-end pin below runs the real arithmetic to hold that.

The folded anomaly names the job. The incident's e-mail is led by its highest-severity member and every viewer/MCP card renders its own frozen StoryText; one long job scores 0.5 while a fired anomaly scores 0.5–1.0 and amplifies from there, so the anomaly ordinarily leads and the job card may sit under the notify threshold and never reach the e-mail. Since #3653 (PR #3693) the job's name is on the RUNNING_JOBS fact's Fact.ObjectName, and it reaches the operator only through composed advice frozen into StoryText. So Reconcile gains a second parameter — the run's facts, the same list ClusterIntoIncidents and FactAdvice.PopulateStoryText already take — and, for a maintenance-family anomaly that lands on an incident carrying the job, appends one sentence to the anomaly's frozen Investigation:

RUNNING_JOBS fired in the same window — Agent job Nightly Index Maintenance was running well past its normal duration — so this anomaly and that job are ONE incident, not two: a long index rebuild, CHECKDB or reload drives exactly this write, log-flush and schema-lock pressure, and moving or shortening the job is the fix for both cards.

It reads the blob with FactAdvice.TryReadStoryText and re-freezes with FactAdvice.SerializeForStoryText — the same {h,i,r} shape every card deserializes, so no renderer and no store changes. The sentence follows the incident, not the parent's path: a lone SCH_M story unions onto the job's incident across its own edge without carrying the key, and an SCH_M-dominant wait profile folding onto it by family gets the sentence too. A fact with no name (a job_name that was NULL) gets the unnamed shape pointing at the Running Jobs view; the append is idempotent on its marker.

Non-maintenance anomalies stay solo. CPU spike, memory pressure, blocking/deadlock spikes, read latency, batch/session/query-duration, the db-scoped object pair, and a wait profile dominated by anything but WRITELOG/SCH_M never gain the job as a target and never get the sentence — #3632 declined those edges deliberately (a CPU spike during a rebuild is not the rebuild's card) and this arm honours the same line. Of the live four, only the maintenance-family ones fold; the rest were paging on their own evidence and still do.

Why this shape, and what was rejected

• Reconciler, not graph. A fourth edge ANOMALY_* → RUNNING_JOBS would put anomaly keys into the traversal and the union-find, which the reconciler's Round-2 header rejects for over-merging; the fold-target set is the one place the anomaly path already decides "whose incident is this", and one more candidate there changes nothing else.
• Fold onto a story, never mint. The arithmetic above makes the minted-incident arm unreachable; shipping it would be code that no run can execute and no test can honestly reach.
• Facts passed in, not parsed out of the job story's prose. The name is only ever on the fact; reading it back out of the job card's headline would be regex over prose. The two-argument Reconcile is the load-bearing edit at the one call site per SKU (DarlingAnalysisService.cs, Lite AnalysisService.cs), mirroring how ClusterIntoIncidents(stories, facts) is already called one line above.
• Sentence into Investigation, headline untouched. The headline is the anomaly's own measurement and FactAdvice's arm (out of this lane); the sentence is the anomaly-side counterpart of LinkedJobClause on the symptom cards, in the same register.
• A literal MaintenanceFamilies, pinned to the graph. The reconciler is static and graph-less by design; the test asserts every member has a "maintenance" edge onto RUNNING_JOBS in RelationshipGraph.GetAllEdges and that no other family an anomaly can resolve to has one, so the two cannot drift silently.

Blast radius

• PerformanceMonitor.Analysis is shared by both SKUs and the frozen deprecated/Dashboard. The one-argument Reconcile(stories) is kept and delegates with facts: null: the Dashboard's untouched call site compiles, gets the fold (same rule, structural gate), and writes no sentence — the sentence needs the fact, and the frozen twin is not modernised to pass it. Its AnomalyIncidentReconcilerTests twin is not extended (frozen; no census pins the two files identical).
• Persisted rows: a maintenance-family anomaly's incident_id now matches the job's, and its story_text Investigation carries one more sentence. Path, hash, severity, confidence and every other column are unchanged; mutes key on the path hash and are unaffected.
• RelationshipGraph.BuildMaintenanceEdges doc: its "does not name the job" sentence has been false since The analysis said an Agent job was running long and could not say which: the RUNNING_JOBS fact now names the job furthest past its own history on both SKUs, and the card, the frozen finding text and the headline say which one (partial #3653) #3693; corrected to say where the name lives and that the anomaly half is this reconciler arm. Doc-only.
• No schema change, no rung, no knobs, no FactScorer / FactAdvice / PgTarget* edits, no CHANGELOG edit (entry proposed to the coordinator).

What it does NOT do

• Does not model the maintenance schedule or score recurrence (A9's structural half, deferred on Brains-review campaign: deferred structural residue (from #3538 / #3539 / #3540 / #3541) #3653).
• Does not change the anomaly headline or the e-mail subject; the e-mail's "Co-fired in this incident" line names the job card whenever it clears the notify threshold, and the appended sentence carries the name when it does not.
• Does not carry the job's overrun figures into the sentence — max_percent_of_average / max_duration_seconds are window maxima over every running row (The analysis said an Agent job was running long and could not say which: the RUNNING_JOBS fact now names the job furthest past its own history on both SKUs, and the card, the frozen finding text and the headline say which one (partial #3653) #3693's stated caveat), and the job card in the same incident states them.
• Does not make SCH_M reachable: the collectors key wait facts on the raw wait_type, SQL Server reports schema-modification waits as LCK_M_SCH_M, and FactCollectorHelpers.IsGeneralLockWait groups that into LCK — so no fact keyed SCH_M is produced today and the SCH_M arms (scorer ramp, Story confidence measures corroboration instead of path length, and the nightly rebuild's three cards become one incident that names the job (#3538 A6/A9) #3632 edge, advice, this set) are dormant against real stores. Kept in the set for parity with the graph; reported to the coordinator as a separate finding, not fixed here.

Tests

New in Lite.Tests/AnomalyIncidentReconcilerTests.cs (the reconciler's home), all executed on this machine through a throwaway net10.0 xunit runner named Lite.Tests against the built PerformanceMonitor.Analysis.dll — 38/38 in the class, 61/61 with StoryConfidenceTests beside it — and proven red by emptying MaintenanceFamilies (6 failed: the fold, the name, the own-family-then-sentence case, the unnamed/idempotent case, the graph-parity pin and the end-to-end; the three negative pins stayed green as they should):

• MaintenanceAnomalies_FoldOntoTheFiredJobsIncident_AndTheirTextNamesTheJob — the issue's fixture: four one-fact maintenance-family anomalies (mapped family; dominant WRITELOG; dominant SCH_M; a WRITELOG/SCH_M tie) beside the fired job's story → one id across all four, each Investigation names `Nightly Index Maintenance`, headline/remediation/job story untouched.
• MaintenanceAnomalies_StayApart_WhenRunningJobsIsPresentButNotFired — the same four with the job present but quiet → four ids, text byte-identical.
• NonMaintenanceAnomaly_StaysSolo_BesideTheFiredJob (theory, 13 rows) — every other mapped key, the unmapped ones, and wait profiles dominated by PAGEIOLATCH_SH / HADR_SYNC_COMMIT / CXCONSUMER / LCK_M_X / LCK_M_SCH_M.
• MaintenanceAnomaly_PrefersItsOwnFamilyParent_AndStillNamesTheJob_WhenThatIncidentCarriesIt; MaintenanceAnomaly_FoldsByFamily_WithoutTheSentence_WhenTheIncidentHasNoJob; OneArgumentReconcile_FoldsOntoTheJob_ButCannotNameIt; UnnamedJob_GetsTheUnnamedSentence_AndTheAppendIsIdempotent.
• MaintenanceFamilies_AreExactlyTheGraphsMaintenanceEdgeSources — the set against RelationshipGraph.GetAllEdges, both directions.
• EndToEnd_FiredJobRootsAStory_MaintenanceAnomaliesFoldOntoIt_OthersStaySolo_QuietJobChangesNothing — the real FactScorer, InferenceEngine.BuildStories, FactAdvice.PopulateStoryText, ClusterIntoIncidents, IncidentId.StampClusters and the reconciler, in the SKUs' order, over the live window as raw facts (one named long job; write-latency and WRITELOG-dominant wait-profile anomalies at 3σ / 8×; a CPU spike and a query-duration anomaly beside them): five stories, the job at exactly 0.5, three incidents after the fold, the two maintenance anomalies naming the job and the two others not; with the job merely running, four stories, four incidents, no sentence anywhere.

Existing pins: the 17 pre-existing reconciler tests, StoryConfidenceTests (the #3632 edge and cluster pins) and Darling's PgTargetSharedSwitchRoutingTests (one-argument calls) are unchanged and green. No existing test pinned an anomaly's incident id or StoryText beside a fired RUNNING_JOBS.

Builds, zero warnings, -c Release -p:EnableWindowsTargeting=true: PerformanceMonitor.Analysis, PerformanceMonitor.Darling.Analysis, PerformanceMonitorLite, Lite.Tests, Darling.Tests, deprecated/Dashboard.Tests.

Which component(s) does this affect?

• Lite
• Darling
• Lite Tests
• Darling Tests
• SQL collection scripts
• Documentation
• Full Dashboard (deprecated)
• CLI Installer (deprecated)

How was this tested?

Executed on macOS against the built shared assembly (above); no SQL Server or store instance is involved — the change is engine wiring over hand-built and scorer-scored facts. CI's build job runs the same tests in Lite.Tests proper.

Checklist

• I have read the contributing guide
• My code builds with zero warnings (dotnet build -c Debug)
• I have tested my changes against at least one SQL Server version (not applicable — no SQL Server surface changed)
• I have not introduced any hardcoded credentials or server names

Fixes #3704