Repository navigation
[FEATURE] Emit a monotonic per-fingerprint occurrence counter on alerts (the current count is a rolling-window gauge, so throttled consumers can't recover the true total) #2216
Description
Which component(s) does this affect?
- Lite
- Darling
- Documentation
Problem Statement
The deadlock/blocking alert value is a rolling 1-hour count (GetRecentDeadlocksAsync(hoursBack:1)), and webhook delivery is throttled per-fingerprint by the #1154 cooldown.
A downstream consumer therefore cannot reconstruct how many times an incident actually occurred: it's a gauge, not a counter, so summing delivered values over-counts overlapping windows and under-counts throttled/aged-out ones.
Real example: a deadlock fingerprint occurred 5×; webhooks delivered at 14:41 (Occurrences 2) and 15:56 (Occurrences 1) while the re-fires between (Value 4) went to the tray only and then aged out of the 1-hour window. Our Azure DevOps ticket reads 3, not 5. The severity scales with the cooldown: at a ≥1-hour delivery interval, essentially every occurrence between deliveries is lost.
Proposed Solution
Add a monotonic cumulative occurrence count per fingerprint to the alert payload (e.g. an Occurrences (total) fact / incident field) - total occurrences of that dedup key for the life of the active incident, independent of the rolling window and of how many deliveries the cooldown suppressed.
A consumer then SETs its count to that value on each delivered webhook and is exact regardless of throttling or window aging. The engine already keeps per-fingerprint state for the #1154 cooldown and the RollingCountAlertGate watermark, so the running total is already in reach. (Pairs with the dedup_key MCP filter — both surface per-fingerprint truth to downstream tooling.)
Use Case
We turn each alert into an ADO work item and need its occurrence count to be trustworthy for triage ("this deadlock has hit 5× since 14:41") - especially at longer cooldowns, where the rolling gauge otherwise drops most of the signal. A cumulative counter makes the ticket accurate on the next delivered webhook without shrinking the cooldown (which would just restore alert spam).
Alternatives Considered
No response
Additional Context
Darling (shared alert engine — Lite too) · Version: 3.4.0-nightly.20260812
Activity
[-][FEATURE] Title: Emit a monotonic per-fingerprint occurrence counter on alerts (the current count is a rolling-window gauge, so throttled consumers can't recover the true total)[/-][+][FEATURE] Emit a monotonic per-fingerprint occurrence counter on alerts (the current count is a rolling-window gauge, so throttled consumers can't recover the true total)[/+]erikdarlingdata commentedon Aug 12, 2026
Taking this. The diagnosis is right and the distinction you're drawing — gauge versus counter — is the part that matters, so let me restate it to be sure we're building the same thing.
GetRecentDeadlocksAsync(hoursBack: 1)is a gauge: it answers "how many in the last hour", which is a different question from "how many since this incident started". Those coincide only when the cooldown is short relative to the window and the incident is short relative to both. At longer cooldowns they diverge in exactly the direction that hurts you: the value ages out of the window between deliveries, so a consumer doing SET-on-delivery lands on a number that is not the total and is not even monotonic — it can go DOWN between two deliveries of the same fingerprint while the underlying problem is getting worse. For an ADO work item that reads as "this deadlock has hit 5x since 14:41", a non-monotonic source is worse than a missing one, because it looks authoritative.You're also right that the state is already in reach rather than needing a new store: the #1154 cooldown keeps per-fingerprint state and
RollingCountAlertGatekeeps a watermark. So this is a counter alongside them, not a new subsystem.Three things I want to pin down before it's built, because they decide whether the number is trustworthy in the cases that matter rather than just present:
1. What resets it. "For the life of the active incident" needs a definition that survives a service restart, since the interesting incidents outlive one. If the counter lives only in the engine's in-memory per-fingerprint state, a restart silently restarts the count and your ticket regresses — the same class of bug as #2166's alerted-state memory, which had to be persisted for precisely this reason. My inclination is that it persists beside the cooldown watermark and resets on the falling edge (fingerprint clears / recovery), not on process start, and that the payload names which incident the count belongs to so a consumer can tell "reset because recovered" from "reset because restarted".
2. Whether it counts OCCURRENCES or DELIVERIES. Your use case wants occurrences — including the ones the cooldown suppressed, which is the whole point. That means incrementing where the event is observed, not where the webhook fires. Worth stating explicitly in the payload's field name, because "Occurrences (total)" next to a throttled delivery invites the opposite reading.
3. Shared engine means both SKUs. You flagged it as Darling and Lite, and the engine is shared — so the counter should land in
PerformanceMonitor.Alertingwith both stores persisting it, or it becomes a Darling-only field that Lite's consumers see as always-zero. That's the shape #2166/#2203 ended up in, and doing it once is cheaper than the parity gap.One honest note on timing: I'm handing my current work off, so I'm scoping and specifying this rather than starting the code today. If you have a preference on the reset semantics in (1) — particularly whether a restart mid-incident should preserve the count or start a fresh incident — say so, because that's the one choice a consumer can actually feel and I'd rather match your triage workflow than guess at it.
erikdarlingdata commentedon Aug 12, 2026
Built and open as #2221 — sorry for the design-comment-then-nothing above; that was me deciding to defer work nobody asked me to defer.
Incidents now carry Total Occurrences (accumulated per dedup fingerprint over the life of the incident) alongside the existing window count, plus Incident Since so a total that restarts can be told from one that continues. Both reach every surface that iterates the alert details and persist in the history row.
Three things from your report that shaped it:
The gauge fact stays exactly as it is. Automation keys on fact names, so redefining
Occurrencesfrom a window count to a total would have silently changed the number for anyone already reading it. Two facts, two questions.Counted per fingerprint, not per (server, metric) — which is what your title said, and it matters: a deadlock over one object set and a deadlock over another are separate incidents, and pooling them would report each one's total as the sum of both.
Occurrences, not deliveries. An event that happens while the cooldown is suppressing delivery is still in the one-hour read window when the next delivery observes it, so it gets counted.
One limit worth being straight about, since you'd hit it before I would: the total is a lower bound. It is exact whenever the read window outlives the gap between deliveries, which covers the throttled case you described. What it cannot recover is an event that both arrives and ages out between two deliveries — that needs the cooldown configured near or above an hour, and no arrangement of a window gauge can see those. Counting them would take an event-identity watermark at the collector, which is a separate piece of work. If your cooldowns are long enough for that to bite, say so and I'll file it.
Also persisted in both stores, so a service restart doesn't reset the total — a counter that resets on restart is just a second gauge wearing a total's name.
Observe occurrences on every sweep, not only on deliveries (#2216)
Merge pull request #2221 from erikdarlingdata/feat/2216-incident-occu…
erikdarlingdata commentedon Aug 12, 2026
Shipped in 3.5.0: alerts now carry a monotonic Total Occurrences per fingerprint plus Incident Since, stored per (server, metric, dedup key) in both stores, surviving restarts and observed on every sweep whether or not anything is delivered - so a throttled consumer can recover the true total by subtracting, exactly what this asked for. The rolling-window count you see today is unchanged and still there. Landed on dev via #2221 (V61 store migration); merged after an adversarial gate review plus a fresh CI pass against the new migration-ladder pins. Accuracy note, stated plainly: the total is exact whenever the 1-hour read window outlives the gap between OBSERVATIONS (sweeps, not deliveries); an event aging out entirely between two sweeps under-counts by one, and a service outage longer than the window restarts the incident with a moved Incident Since so consumers can detect the discontinuity. Thanks for the well-specified request.
Merge dev: renumber the blocking rung V69 -> V71 after #2213's own re…
An incident can carry a total, not just a window count (erikdarlingda…
The occurrence accumulator (erikdarlingdata#2216)
Persist the counters in both stores (erikdarlingdata#2216): Darling V…
Wire both count gates to the accumulator (erikdarlingdata#2216)
Cover the cases where the gauge is uninformative (erikdarlingdata#2216)
CHANGELOG: the occurrence total and its honest bound (erikdarlingdata…
Accumulate occurrence totals for every fingerprinted alert, not just …