[BUG] Email/webhook cooldown is keyed on (server, metric), not the #1140 fingerprint - distinct concurrent deadlock/blocking incidents are silently dropped (tray shows, Teams doesn't)

Component

Lite

Performance Monitor Version

Lite, v3.0.0 (dev build, with #1145)

SQL Server Version

SQL Server 2022

Windows Version

Windows Server 2019

Describe the Bug

The email and webhook cooldown is keyed on (serverId, metricName), not on the #1140 incident fingerprint. So once a deadlock (or blocking) alert is delivered, a genuinely different deadlock/blocking incident arriving within the cooldown window (EmailCooldownMinutes, default 15 min) is silently dropped from Teams/Slack/email — the tray notification still shows, but the distinct new incident never reaches the webhook, and therefore never reaches downstream ticketing. This partly defeats Per-event mode, whose purpose is one notification per distinct incident.

This is a pre-existing design limitation (not introduced by #1145), but #1145 slightly widens it by persisting the webhook cooldown across restart.

Steps to Reproduce

Generate the same deadlock twice (objects T1,T2 → fingerprint 37486…).
Within 15 minutes, generate a different deadlock once (objects T1,T3 → fingerprint 9c4b35…).

Expected Behavior

A distinct incident (different #1140 fingerprint) should not be throttled by an unrelated prior incident. Re-firing the same fingerprint should stay suppressed (that's #1091); a different fingerprint should be delivered.

Actual Behavior

Two alert cycles fire (the rolling 1-hour count climbs 2→3, so the edge-trigger gate correctly re-fires):

Error Messages / Log Output

Screenshots

Additional Context

Root cause

The cooldown key carries no fingerprint:

// WebhookAlertService.cs:82
var cooldownKey = $"webhook:{serverId}:{metricName}";   // metricName = "Deadlocks Detected"
// :97-101 → if (now - lastSent < EmailCooldownMinutes) return false;   // suppresses cycle B

Cycle A's delivery stamps that key (:115-118); cycle B (different fingerprint, ~1s later) hits the same key inside the window and returns false. The tray is shown synchronously before the send path (MainWindow.xaml.cs:1795-1804), so it's not gated by this cooldown — which is why the tray shows but Teams doesn't. The row logs tray because WebhookSent == false (EmailAlertService.cs:71-80).

Scope

Email is identical (EmailSendCore.cs:87 keys on {serverId}:{metricName}).
Blocking alerts share the same path (metricName = "Blocking Detected").
CPU / memory / poison-wait / low-disk / failed-job carry no fingerprintable incidents → unaffected (would fall back to today's behavior).
Summary mode has the same suppression (the second cycle's whole summary, including the new incident, is dropped); Per-event just makes the loss of one specific incident more visible.

@gotqn

Thanks @gotqn — the root cause is dead-on, and it's a bit worse than the title suggests. A few corrections so the fix lands in the right place.

Root cause confirmed, both channels

Webhook: WebhookAlertService.cs:82 → cooldownKey = $"webhook:{serverId}:{metricName}", gated at :97-101.
Email: EmailSendCore.cs:87 → cooldownKey = $"{serverId}:{metricName}", gated at :101-102.

Neither carries the #1140 fingerprint, and the tray fires synchronously ahead of the send path — exactly as you describe.

Scope is wider than deadlock/blocking

The #1140 fingerprint is actually attached (AlertContext.Incidents) by five live builders, in both Lite and Dashboard:

Alert	Fingerprinted	Key
Deadlock	yes	involved objects (IncidentGrouping)
Blocking	yes	involved objects (IncidentGrouping)
Long-running query	yes	`Query` / query_hash
Anomalous/long-running job	yes	`Job` / job name
Low disk	yes	`Disk` / mount point

All five are throttled by the per-(server, metric) cooldown the same way your second deadlock is: a distinct second query/job/volume arriving inside the EmailCooldownMinutes window is dropped from Teams/email. So the bug understates itself — it's five fingerprinted incident types, not two.

Two corrections to the "carry no fingerprintable incidents → unaffected" list:

Low disk is fingerprinted (BuildVolumeFreeSpaceContext, AlertFingerprint.Disk) — affected, not unaffected.
Failed job genuinely is not fingerprinted (BuildFailedJobContext applies none), even though the job name is a perfect natural key and its sibling BuildAnomalousJobContext fingerprints by job name right next to it. That's a real gap worth closing alongside this.

CPU / memory / poison-wait correctly carry no incident (continuous levels, not discrete events).

Fix shape

The cooldown can't just append the fingerprint to the key, because a single alert can carry multiple incidents (top-5 volumes, top-N queries). It needs to move to a per-incident-fingerprint cooldown: send if any incident in the alert has a fingerprint not currently in cooldown, while a re-fire of the same fingerprint stays suppressed (#1091). Metrics with no fingerprint keep today's per-metric cooldown. Both channels, both apps.

Merge pull request #1156 from erikdarlingdata/feature/1154-cooldown-f…

Fixed in #1156, now merged to dev — it'll be in the next nightly build, so you can verify there.

What changed: the email/webhook delivery cooldown is now keyed on the #1140 incident fingerprint instead of just (server, metric). A genuinely distinct incident arriving inside the EmailCooldownMinutes window is delivered; a re-fire of the same fingerprint stays suppressed (#1091). Per-event mode no longer collapses to one notification per cycle either. Both apps (Lite + Dashboard) and both channels (email + webhook) run the same shared decision, and the restart seed (#981/#1145) is now per-fingerprint, so a distinct incident isn't wrongly suppressed right after a restart.

Two notes on scope from digging into this:

The blast radius was wider than (deadlock, blocking): low-disk, long-running-query, and anomalous-job are also fingerprinted and were being throttled the same way — all five are fixed.
Failed-job is still not fingerprinted (it has no [FEATURE] Add a stable, machine‑readable dedup fingerprint (and involved‑object list) to alert payloads - esp. deadlocks & blocking #1140 incident yet), so two distinct failed jobs in the window still coalesce. Giving it a Job fingerprint by job name is a small separate follow-up I'll track; this change is forward-compatible with it.

One ask: I don't have email/Teams infra to live-smoke-test the end-to-end delivery. The change is covered by automated tests (the per-fingerprint decision, the seed filter on both stores, the null-context guard), but if you can, the decisive check on the nightly is: trigger two distinct deadlocks (different object sets) within ~15 minutes and confirm both reach Teams/email — previously the second was dropped. Thanks for the very precise root-cause writeup; it made this straightforward.

Merge pull request #1157 from erikdarlingdata/feature/failed-job-fing…

Merge pull request #1231 from erikdarlingdata/feature/changelog-backf…

Fix erikdarlingdata#1154: key the alert delivery cooldown on the erik…

Fingerprint failed-job alerts so distinct failures don't coalesce (er…

Backfill CHANGELOG for the shipped alert fingerprint cluster (erikdar…

Phase-5 D: the shared alert engine — Darling alerts

Fix erikdarlingdata#2716: none of the 5 Postgres alerts' restart wate…

Merge pull request #3324 from erikdarlingdata/fix/3313-cooldown-filte…

Bound alert delivery per metric across the fleet, not only per finger…

Never answer one cooldown key's question with another key's history (#…

Render only the incidents outside their own cooldown window (erikdarl…

Navigation Menu

Repository navigation

[BUG] Email/webhook cooldown is keyed on (server, metric), not the #1140 fingerprint - distinct concurrent deadlock/blocking incidents are silently dropped (tray shows, Teams doesn't) #1154

Component

Performance Monitor Version

SQL Server Version

Windows Version

Describe the Bug

Steps to Reproduce

Expected Behavior

Actual Behavior

Error Messages / Log Output

Screenshots

Additional Context

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[BUG] Email/webhook cooldown is keyed on (server, metric), not the #1140 fingerprint - distinct concurrent deadlock/blocking incidents are silently dropped (tray shows, Teams doesn't) #1154

Description

Component

Performance Monitor Version

SQL Server Version

Windows Version

Describe the Bug

Steps to Reproduce

Expected Behavior

Actual Behavior

Error Messages / Log Output

Screenshots

Additional Context

Activity

erikdarlingdata commented on Jun 19, 2026

erikdarlingdata commented on Jun 19, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions