You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[BUG] A webhook batch posts when ANY incident is outside its #1154 cooldown, then re-renders EVERY incident on the card - already-delivered signatures repeat in Slack #3313
WebhookAlertService.PostAsync asks IncidentCooldown.EvaluateAsync whether to post, and that call is documented as "Post if any incident in this alert is outside its window (a distinct fingerprint is not throttled by an unrelated prior incident)". That half is correct and is what #1154 built.
The problem is what gets rendered once the decision is ShouldSend. The channel builders render every incident in context.Incidents, including the ones that are still inside their own cooldown window and were already delivered minutes earlier. A single new fingerprint therefore drags every co-resident stale fingerprint back into the channel with it.
So the per-fingerprint cooldown governs whether a card posts, but nothing governs which incidents that card contains. Raising the cooldown does not fix this, because the repeat is not a second firing of the old fingerprint — it is a passenger on the new one's card.
Observed
A 74-minute window on a 42-server SQL Server store delivered 18 Deadlocks Detected cards. Three of them re-delivered a fingerprint that had already been posted inside the same window.
The clearest instance, one server, Summary mode:
time
card
fingerprint
note
19:55:26
Deadlock (1 incident)
bf7e29b4…
first delivery
20:36:40
Deadlock 1 of 2
9d5ead0a…
new fingerprint — correctly posts
20:36:40
Deadlock 2 of 2
bf7e29b4…
re-delivered, 41 min after its own first delivery, with Total Occurrences: 1 — i.e. it had not recurred at all
Total Occurrences: 1 on the second rendering is the proof that this is a re-render and not a recurrence: the occurrence accumulator (#2216) saw no new events for that fingerprint between the two cards. The incident was simply still in the rolling 1-hour read window, so it rode along.
Expected
When a batch posts, render only the incidents that are actually outside their own cooldown window — the same set IncidentCooldown stamps on success. An incident inside its window has already been delivered and should be omitted from the card, not repeated.
If dropping it entirely loses useful context, a one-line "N other incidents still open (already reported)" footer preserves the signal without restating each one.
Notes
IncidentCooldown.EvaluateAsync already computes the per-fingerprint decision set, so the information needed to filter the render is present at the decision point and does not need to be recomputed.
The same path serves Teams, Slack, PagerDuty and the generic webhook, so the fix is one place for all four.
Lite shares PerformanceMonitor.Notifications, so it has the same behavior.
Fixed by #3324, merged to dev. Closing explicitly — closing keywords no-op on dev.
IncidentCooldown.Decision now reports the deliverable fingerprints alongside ShouldSend, and the rendered context is narrowed to them with a "N other incident(s) still open (already reported)" footer. Persistence is untouched, so config_alert_log still carries every incident.
Two findings from the lane worth keeping on the record:
Per-event mode never had this defect.PerEventNotification.Split gives each message a single-incident context, so an in-window fingerprint fails its own ShouldSend and never posts. It was not changed, and moving the filter ahead of the split — which this issue's notes suggested — would have cost the suppressed incident its history row.
PagerDuty's dedup_key now anchors on the first DELIVERABLE incident, which corrects a real correlation defect rather than merely changing behaviour: a {in-window, fresh} batch previously triggered under the in-window incident's key and folded into its existing PD alert, so the genuinely new incident never surfaced there at all.
One thing this made reachable rather than caused, now filed as #3330: AttachmentXml is per-alert while the cards are per-incident, so a filtered card can attach a graph belonging to a suppressed incident. Per-event mode has had the worse version of that since #1141 — it copies one graph onto every split message.
Activity
erikdarlingdata commentedon Sep 11, 2026
Claude posting for Erik Darling
Fixed by #3324, merged to
dev. Closing explicitly — closing keywords no-op ondev.IncidentCooldown.Decisionnow reports the deliverable fingerprints alongsideShouldSend, and the rendered context is narrowed to them with a "N other incident(s) still open (already reported)" footer. Persistence is untouched, soconfig_alert_logstill carries every incident.Two findings from the lane worth keeping on the record:
Per-event mode never had this defect.
PerEventNotification.Splitgives each message a single-incident context, so an in-window fingerprint fails its ownShouldSendand never posts. It was not changed, and moving the filter ahead of the split — which this issue's notes suggested — would have cost the suppressed incident its history row.PagerDuty's
dedup_keynow anchors on the first DELIVERABLE incident, which corrects a real correlation defect rather than merely changing behaviour: a{in-window, fresh}batch previously triggered under the in-window incident's key and folded into its existing PD alert, so the genuinely new incident never surfaced there at all.One thing this made reachable rather than caused, now filed as #3330:
AttachmentXmlis per-alert while the cards are per-incident, so a filtered card can attach a graph belonging to a suppressed incident. Per-event mode has had the worse version of that since #1141 — it copies one graph onto every split message.Merge pull request #3331 from erikdarlingdata/chore/changelog-3313-3315
Bound alert delivery per metric across the fleet, not only per finger…
Name the regression by the PR that shipped it, not by a guessed issue…
Render only the incidents outside their own cooldown window (erikdarl…
Assert both incident orders in the erikdarlingdata#3313 fixtures
Add the erikdarlingdata#3315 and erikdarlingdata#3313 CHANGELOG entries