A chain that fires every Tuesday at the same hour was rated a fresh incident each week and nobody was told: the pass now labels it "recurring at this hour" at unchanged severity, and a weekly Agent job whose slot slid gets "maintenance window moved" (partial #3653 item 3, Q3)

[erikdarlingdata]

Partial #3653 — item 3 (Q3)

The lie

A diagnostic chain that fires every Tuesday at 14:00 — a report, a batch, an index rebuild — was rated by the engine as a fresh incident every week, and nothing on the card, in the e-mail or on the MCP wire said it was the third Tuesday running. #3538 A9 proposed the opposite fix: DISCOUNT the severity of a chain that recurs at the same hour×weekday for three consecutive weeks, because it is "expected". Erik ruled it the other way (Q3, verbatim): "Label, not discount: 'recurring at this hour' at unchanged severity. A weekly problem is still a problem." The sibling half of A9 — a weekly maintenance job whose window MOVES (Thursday 02:00 last week, Tuesday 03:00 this week) — defeats any same-slot rule by construction and defeats the operator the same way: last week's symptom "went away" at the old hour and a new one appeared at the new one.

Mechanism

A new pure class, PerformanceMonitor.Analysis/RecurrenceLabeler.cs, called as step 3.8 in both SKUs' pass methods (DarlingAnalysisService.AnalyzeAsync, Lite AnalysisService.AnalyzeAsync): after AnomalyIncidentReconciler.Reconcile has finished rewriting frozen StoryText (#3709) and BEFORE FilterMutedFindingsAsync copies it onto the finding — so the label is on the persisted row every card, e-mail and MCP read renders, with no renderer change (the #3709 pattern: TryReadStoryText → with { Investigation = … + sentence } → SerializeForStoryText).

One store read per pass, not per story. PgFindingStore.GetPriorOccurrencesAsync / Lite FindingStore.GetPriorOccurrencesAsync (new SQL const GetPriorOccurrencesSql, one token apart — server_properties vs Lite's v_server_properties view — and pinned to exactly that difference) returns, over the prior 21 days + 1 h of slack on idx_analysis_findings_time:

• every chain (story_path_hash) that fired in the reference instant's hour×weekday slot, and
• every RUNNING_JOBS-rooted card in ANY slot (a job that slid is by definition in a different slot),

collapsed to one row per (chain, root key, local hour) so the per-cycle re-persist (FindingOccurrences: 27.9× mean) counts as one "fired in that hour". story_text is read for the job rows only (MAX(CASE WHEN root_fact_key = 'RUNNING_JOBS' …)), because the frozen job card is the one place a prior week's job NAME survives — Fact.ObjectName is not a finding-row column and a rung for one is not this lane's. The reference instant is context.TimeRangeEnd: "now" for a scheduled pass, the anchor for an as-of pass, and the exclusive upper bound of the read (this pass has not persisted; earlier passes today are week 0 and are dropped).

The slot is on the target's clock. Rows and the reference are shifted by the latest server_properties.utc_offset_minutes (the same read PgFactCollector.QueryPerf / DuckDbFactCollector.QueryPerf make) before EXTRACT(HOUR) / EXTRACT(DOW). Q6 ruled buckets key on the target's local clock; this lane does not build a UTC-only rule for Q6 to invalidate. When the store carries no offset (a PostgreSQL target has no server_properties row; a SQL Server target's on-load collector may not have run) the read returns a null offset, the slot is UTC, and the sentence SAYS so. One offset across the whole window is a fixed offset, not a zone — a DST change inside the 21 days smears the older week by an hour, which is the Q6/Q8 defect (the zone-id rung), disclosed in the SQL comment, not fixed here.

Recurring at this hour. ≥ 3 CONSECUTIVE weeks counting this pass ({1,2} → 3; {1} → 2; {1,3} → 2, the gap ends the run; {1,2,3} → 4, the read's ceiling). Appended sentence (fixed phrasing, because TryReadLabel parses the count back out of it):

Recurring at this hour: this chain has fired in this hour×weekday slot — 10:00 Tuesday, server local time — for 3 consecutive weeks. The severity above is unchanged (a weekly problem is still a problem); the label says the cause keeps a schedule, so look at what runs in this slot — a job, a report, a batch — before treating this as a new incident.

(UTC fallback: — 14:00 Tuesday UTC (the store carries no UTC offset for this server, so the slot is keyed on UTC) —.) Metadata on the story's RootFactMetadata — a COPY, because InferenceEngine hands the story the root fact's own dictionary by reference — recurring_at_this_hour = 1, recurrence_weeks = N, recurrence_local_hour, recurrence_local_dow (0 = Sunday); the stores already copy it onto the in-memory finding for the notification layer.

Maintenance window moved. When this pass's fired RUNNING_JOBS fact (base severity > 0, the #3709 predicate) carries a name, and LAST week's job-rooted card names the SAME job (case-insensitive, read off the frozen headline Agent job \name`thatFactAdvice.ComposeRunningJobswrites — pinned against the live composer) in a slot that is not this one, every story this pass ties to the job —RUNNING_JOBSon itsPath, or the job link RUNNING_JOBS fired in the same windowin its frozen Investigation (theLinkedJobClause/ #3709 fold spelling, pinned equal toAnomalyIncidentReconciler.JobSentenceMarker`) — gets:

Maintenance window moved: Agent job \Nightly Index Maintenance` ran at 02:00 Thursday last week, 10:00 Tuesday this week (server local time; hour buckets, the job's earliest hour last week). The same weekly job in a different slot is why a symptom that "went away" at the old hour has appeared at this one — it moved with the job. The severity above is unchanged; check the job's schedule before treating either hour as a new incident.`

plus maintenance_window_moved = 1, maintenance_window_last_week_hour, maintenance_window_last_week_dow. Last week only, by design — the sentence compares two weeks. A pre-#3693 card with no name, a different job, a job in the SAME slot (that is recurring, the other arm's case), a quiet job, or a job fact with no name each label nothing rather than guessing.

MCP twins. get_analysis_findings on both SKUs publishes recurring_at_this_hour (bool), recurrence_weeks (int, null when not labelled) and maintenance_window_moved (bool) on every entry, read off the representative's frozen text with RecurrenceLabeler.TryReadLabel — the row has no metadata column and a three-week fact cannot be recomputed from a read capped at seven days. One sentence added to both tool descriptions (byte-identical across SKUs, pinned).

Evidence

• The engine re-persists every story every cycle (FindingOccurrences, measured 27.9× mean, 39× worst) — hence the per-hour collapse in SQL rather than counting rows.
• AnalysisFinding has no object-name column (InsertFindingSql, 21 columns) — hence the name read off the frozen job card, and the composer contract pin.
• Findings retention default is 30 days (AnalysisRetentionDefaults.FindingsRetentionDays), so the 21-day read is inside it; a shorter retention truncates the stated count honestly.
• Live PostgreSQL (timescaledb 2.28.1/pg18, migrated to V132) and DuckDB 1.5.5 return identical rows for the same fixture: per-hour collapse, Wednesday / wrong-hour / before-bound / after-reference rows excluded, job rows in other slots with their text, offset raw, UTC fallback null.

Why this shape

• A separate step rather than an arm of the reconciler or the scorer: it needs a STORE read the story pipeline does not have, and the scorer is precisely what the ruling said not to involve. Both are fenced for PostgreSQL-target analysis engine: v2/v3 slices and the v1 lanes' residue (from #3542) #3691 besides; neither is touched, nor any PgTarget*, BaselineMath, or CollectAndScoreFactsAsync.
• The flag reaches the read surfaces through the sentence, not a schema change: no rung (one un-landed rung at a time; PostgreSQL-target analysis engine: v2/v3 slices and the v1 lanes' residue (from #3542) #3691 holds V133), no {h,i,r} contract change (a fourth key would be dropped by the next with {} re-freeze; the sentence survives one).
• The store read degrades to an empty read on a fault (logged; not on an abandonment — AnalysisShutdown.IsExpectedAbandon / AnalysisAbandon.IsExpected are honoured), because a label is presentation and an unreadable history must not cost the pass its findings. This is Lite's one catch-and-degrade read, and its doc says why.

What it does NOT do

• Does not change severity, confidence, incident ids, clustering, or the fold. Bit-identical severity is pinned.
• Does not re-bucket baselines or fix DST (Q6/Q8's lanes). Uses a fixed offset and says so.
• Does not persist metadata (no rung). Does not touch deprecated/Dashboard (frozen twin; its pass has no labeler).
• Does not detect a job that moved two weeks ago, or a job the store cannot name.
• Does not label anchored (as_of) passes differently: they are labelled relative to the anchor and not persisted, as before.

Tests

• Lite.Tests/RecurrenceLabelerTests.cs (31, executed on this Mac): 3 weeks → label + slot + read-back; 2 weeks → none; gap week → none; 4 weeks → "4"; severity bit-identical; other chain in slot → none; week 0 ignored; offset puts the slot on the server's clock (03:30Z Tue → 23:00 Monday local); UTC fallback disclosed; idempotent; absolution/zero/legacy skipped; empty/null read no-op; metadata is a copy; moved-window on job/SCH_M/folded-anomaly and NOT the CPU story; same slot → recurring not moved; different/unnamed/quiet/no-facts/two-weeks-ago → none; case-insensitive names; TryReadJobName against the LIVE composer output; JobLinkMarker == AnomalyIncidentReconciler.JobSentenceMarker and present in a composed SCH_M card; WeeksAgo rounding theory; constants. Red-first: a no-op labeler reds 10; a Severity *= 0.9 reds the bit-identical pin.
• Darling.Tests/RecurrenceLabelStoreReadTests.cs (8; 7 ungated + 1 gated, all executed here against live PG): SQL shape; both stores one statement modulo the view token (from source, and the compiled const equals the source); both pass methods label after the fold and before the mute filter and never in CollectAndScoreFactsAsync; both MCP twins publish the three fields; descriptions byte-identical; live round-trip through to the sentences on both the offset and the no-offset server.
• Lite.Tests/FindingStoreRecurrenceReadTests.cs (3, DuckDB-backed via SharedDuckDbFixture): the same read fixture on DuckDB, the UTC fallback, an empty store. CI is first execution for this file (WPF-hosted; the Lite statement was executed here through DuckDB.NET 1.5.5 directly with identical results).
• Census updates: DarlingAnalysisStoreTests.AllFindingStoreSql gains the new const; AnalysisShutdownResidueTests counts the third pass-side catch in PgFindingStore; RepoFileAdoptionTests.s_lfReaders gains the new pin file.

[erikdarlingdata]

Retriggering CI: GitHub dropped the pull_request synchronize event for d0f93a5 (only the pull_request_target check ran); closing and reopening to fire the Build workflow on the current head. No code change.