notification_type 'undelivered' conflates a working cooldown with a failed webhook, both with a null send_error — so 'how many alerts actually reached Slack' is not answerable from the alert log #3427

notification_type = "undelivered" means two opposite things and nothing in a stored row tells them apart: a cooldown working correctly, and a webhook post that failed. Both carry alert_sent: false and send_error: null.

AlertDelivery.ChannelUndelivered's own summary says so:

At least one channel was configured and consulted, and none of them delivered — a cooldown-throttled send, or a webhook post that came back unsuccessful.

And FromFanout's remarks name the cause and call it a gap already:

EmailFanoutResult.SendError tracks the EMAIL channel only — WebhookAlertService.TrySendWebhookAlertsAsync collapses every per-channel outcome into one bool — so a webhook post that came back unsuccessful is reported here as "configured, nothing delivered" with a null error. That is honest but coarse, and it is a live delivery gap on any store that configures a webhook.

There is no distinct disposition for throttling: the vocabulary is none / unconfigured / muted / tray / undelivered / email / webhook / email+webhook, and a cooldown-suppressed send falls through every arm of FromFanout to ChannelUndelivered exactly as a failed webhook does.

Live evidence

One PostgreSQL store, a 24-hour get_alert_history read. Seven Poison Wait rows on two servers, threshold 600,000 ms:

13:30:08  value    701,165   alert_sent true   notification_type webhook       <- delivered
13:31:25  value    707,785   alert_sent true   notification_type webhook       <- delivered
13:35:37  value  1,302,701   alert_sent false  notification_type undelivered   send_error null
13:36:41  value  1,611,192   alert_sent false  notification_type undelivered   send_error null
13:41:01  value    644,779   alert_sent false  notification_type undelivered   send_error null
13:41:45  value    949,312   alert_sent false  notification_type undelivered   send_error null
13:12:07  value    831,295   alert_sent false  notification_type undelivered   send_error null

None muted. Two delivered, five did not, and the five carry no reason. The shape — deliveries first, then a run of undelivered — is consistent with the configured cooldown_minutes: 15 / per-fingerprint cooldown_minutes: 5 doing its job. I cannot demonstrate that, and that inability is the defect. The alternative reading, five silently failed webhook posts, fits the same rows identically. Note also that the largest values (1,611,192 and 1,302,701 — 2.7x and 2.2x threshold) are among the undelivered ones, so if it is a failure it is losing the worst instances.

Why this is worth fixing now rather than noting

Both blanket mute rules came off the two SQL Server stores tonight, and the immediate task is measuring alert chattiness: fires, dispositions, and actual webhook posts, against a muted-period baseline of 4,120 fires / 24 h on one store and 38 on the other. That measurement runs directly on this field, and with undelivered ambiguous:

• A throttle working perfectly and a webhook failing produce the same row, so "how many alerts actually reached Slack" cannot be answered from the log. The delivered count is trustworthy; the residual is not decomposable.
• The error is biased toward reassurance. A reader tuning cooldowns sees undelivered and credits the throttle, which is the likelier cause and therefore the one that will be assumed — while a genuine webhook outage hides inside the same value with no error text and no send_error.
• It gets worse as throttling gets more aggressive, i.e. exactly as someone tunes down the chattiness this measurement exists to reduce.

#3314 established that email_cooldown_minutes is the only throttle on Slack/Teams/PagerDuty delivery. That throttle's effect is now unobservable, distinct from a failure, in the log that records it.

What would fix it

In rough order of cost:

1. A distinct disposition for throttling — throttled (or cooldown) alongside undelivered, so undelivered narrows to "consulted and failed." One constant, one FromFanout arm, plus StateCarryingChannels and the three reading surfaces the list already pins (both WPF grids and the web dashboard's util.js). This is the change that makes the measurement possible.
2. Give the webhook fan-out an error channel. TrySendWebhookAlertsAsync returning one bool for four channels is what forces the null; a per-channel outcome (or even one first-error string) would let a failed post carry its reason the way an SMTP throw already does.
3. If neither is taken now, at minimum say this in get_alert_history's tool description, since the MCP read is how an agent measures delivery and nothing at that surface warns that undelivered is two conditions. The constant's own doc is honest; the read's is silent.

(1) and (3) together are small and would make tonight's measurement decomposable. (2) is the real repair.

Related: #3169 (established alert_sent's semantics and this taxonomy), #3314 (the cooldown this now hides), #3013 (the sibling case where an alerting failure was swallowed and reached no surface).