alerts_enabled is not the master switch its contract claims: two delivery paths paged through a fleet-wide mute, 38 and 60 minutes in #3464

alerts_enabled presents as the master switch — its own documentation says it governs all alerting — and eleven notification families honor that contract. Two do not, and both were measured delivering to a live paging channel while the switch was off. An operator who flips the master switch reasonably believes they are done; today that belief was falsified twice in one hour, in production, with an on-call audience.

The measured bypasses (2026-09-15, alerts_enabled: false on both SQL Server stores from 13:23:40Z, verified by read-back)

Bypass 1 — analysis notifications, delivered 38 minutes into the mute. An Analysis INFO card (severity 1.75, PLAN_REGRESSION story) reached the webhook channel at 14:01:00Z. The gate is structural: the analysis pass hands the notification service config.Analysis.NotificationsEnabled alone (DarlingWorker.cs:2328, :5574) — AlertsEnabled never participates on that path.

Bypass 2 — the collector-cost self-alert path, delivered 60 minutes into the mute. A Collector Cost Regression webhook fired at 14:23:00Z, confirmed present in the delivery channel itself, not just the store ledger. The counter evidence pins the mechanism: alert_read_health.server_alert_passes sat byte-identical across consecutive hourly reads on both SQL stores (1,285 and 1,461) while the PG store's counter kept growing — the master-gated per-server alert sweep was genuinely stopped, while the self-alert evaluator delivered through its own path regardless. The switch is honored by the sweep and bypassed by a sibling.

The census, which is the actual finding

Grepping the delivery-adjacent gates: the per-server alert engine, twelve self-alert sites, connection-change and AG-health notifications all consult AlertsEnabled — the latter two in the exact idiom the codebase established for families with their own toggles:

if (!_settings.AlertsEnabled || !_notifyConnectionChanges())

That AND'd-gate idiom is the fix shape, already precedented in-file. The two bypasses are the only families that reach a delivery channel without it. The defect is not that per-family toggles exist — they are useful narrower knobs — it is that the master switch's contract is only mostly true, and nothing enforces it.

Acceptance shape

1. Every path that can reach a delivery channel consults alerts_enabled first, AND'd with its own family toggle where one exists — the established idiom, applied uniformly.
2. A census test enumerates delivery call sites derived from source, not from a hand-maintained list (the same discipline as the existing read-failure surface tests), so a third bypass fails red in CI instead of shipping silently.
3. The operational mute-family interactions unwind cleanly once the gate is real: the stopgap fleet-wide mute rules and the multi-switch re-enable procedure that today's incident required should collapse back to a single-switch operation.

Operational context, for severity

Silencing the fleet today ultimately required three independent actions discovered serially — the master switch, the analysis toggle, and stopgap fleet-wide mute rules — each found only because the previous one demonstrably failed to silence a live channel. Un-muting now requires reversing all three in order. That is the cost of a master switch that is only mostly master.