Repository navigation
[BUG] create_mute_rule returns "created" but the mute does not take effect - config_mute_rules has no bump trigger, so the evaluator's cached rules are not reloaded #3315
Description
Summary
create_mute_rule returns {"status":"created", ...} and the row really is in config.config_mute_rules — but the running service keeps delivering matching alerts, for an unbounded period, because nothing invalidates the evaluator's in-memory mute cache.
The chain:
DarlingMcpAlertTools.CreateMuteRuleconstructs a fresh store and writes directly:await new PgMuteRuleStore(postgres).InsertAsync(rule);. It does not touch theMuteRuleServiceinstance the alert engine holds.PgMuteRuleStore.InsertAsyncis a bareINSERT INTO config_mute_rules. No version bump.- There is no bump trigger on
config_mute_rules. The schema has five:trg_bump_monitored_servers,trg_bump_alert_settings,trg_bump_notification,trg_bump_collector_schedules,trg_service_self_bump. MuteRuleServicecaches rules in memory.LoadAsync()has exactly two call sites:DarlingWorker.cs:1462(startup) andDarlingWorker.cs:2869(inside the control-plane reload, gated on aconfig_service.config_versionchange).
So a mute created through the MCP takes effect only when the service restarts, or when an unrelated write to one of the five bump-triggered tables happens to fire a reload. The comment at DarlingWorker.cs:1453 — "the mute service is hoisted too so a reload can re-LoadAsync() it (closes F16 …)" — is accurate about the reload path; the gap is that a mute write is not one of the things that causes a reload.
Observed
On a live 42-server store:
| time (UTC) | event |
|---|---|
| 20:42:39 | create_mute_rule with no scoping → status: created, summary: "(matches all alerts)" |
| 20:43:58 | a matching Deadlocks Detected fires → notification_type: webhook, muted: false — delivered |
| 20:46 | update_alert_settings {"cooldown_minutes": 120} → writes config_alert_settings, trg_bump_alert_settings fires |
| after the next sweep | matching alerts begin recording notification_type: muted, muted: true |
An all-metrics, all-servers mute rule did not suppress an alert that fired 79 seconds later. Activation required an unrelated setting change to force the version bump.
On a second store, mute rules created the same way hours earlier are active (notification_type: "muted"). That is the failure mode at its most misleading: it works eventually, so the tool looks correct, and the window in which it silently does not work has no upper bound and no signal.
Why this matters more than the delay
An operator mutes a noisy alert precisely when it is firing repeatedly. The success response tells them the mute is in place. It may not be, and the only way to find out is to notice that the pages keep coming and then doubt a tool that said created. get_mute_rules will confirm the rule exists, reinforcing the wrong conclusion — it reads the table, not the cache.
Suggested
Add a bump trigger on config.config_mute_rules matching the other four, so any write — MCP, Viewer, or hand-written SQL — causes the next sweep to reload. That is the smallest fix and it covers every writer rather than just this tool.
Alternatives, both narrower: have CreateMuteRule/DeleteMuteRule bump config_version explicitly after a successful write, or route the MCP through the live MuteRuleService so its persist-then-cache path updates the cache directly (the behavior MuteRuleServiceTests already pins for the Viewer path).
The same gap applies to delete_mute_rule, in the more dangerous direction: an operator un-mutes and believes alerting is restored while the stale cache keeps suppressing.
Test
A live-store test can assert the trigger directly: insert into config_mute_rules, then read config_service.config_version and assert it advanced. That pins the mechanism rather than the delivery outcome, and it fails if the trigger is ever dropped from the migration ladder.
Activity
erikdarlingdata commentedon Sep 11, 2026
Claude posting for Erik Darling
Correcting the blast radius I claimed, because the lane measured it and I had guessed.
I wrote that a bump trigger is the right fix because it "covers every writer — the MCP tool, the Viewer's Manage Mute Rules surface, and hand-written SQL — rather than just this tool." The Viewer half of that is wrong: the Viewer path was never broken. Every writer in
ViewerDataService.MuteRulescallsSignalConfigReloadAsyncexplicitly — five call sites, at lines 169, 193, 204, 211 and 245. The MCP tools are the ones that do not.That also explains the symptom I could not account for in the filing. I noted that mutes created hours earlier on another store WERE active while one created seconds ago was not, and called it an unbounded activation window with no signal. The mechanism is narrower than that: any Viewer action bumps the beacon, so an MCP-created mute activates on the next unrelated Viewer write. It reads as intermittent because it is downstream of whether a human happened to touch the desktop app.
The conclusion survives, for a better reason than the one I gave.
PgMuteRuleStorehas five mutating statements, not the two the filing implies —INSERT(:90),UPDATE ... SET(:118),UPDATE ... SET enabled(:139), andDELETEtwice (:148, :163). A per-call-site bump would have to find all five and stay found, while oneAFTER INSERT OR UPDATE OR DELETEstatement-level trigger covers them by construction and cannot be missed by a later writer. So: trigger, but because the write surface is five-wide, not because the Viewer needed rescuing.One design decision in the fix worth surfacing, since it is not obvious from the diff
The V117 probe sentinel reads
pg_trigger, notinformation_schema.triggers. That view only lists triggers on relations the current role owns or holds a non-SELECT privilege on, and the read-onlyviewerrole holds neither on this table — so the sentinel would report the rung absent on a fully-migrated store, and the viewer's connect gate would refuse a store that is actually current. A guard that fails toward "refuse a healthy store" is the expensive direction.Privileges were measured rather than reasoned: a role with SELECT on
config_serviceplus full CRUD onconfig_mute_rulesstill gets 42501 from insideconfig_bump_version(), and exactlyGRANT UPDATE (config_version, updated_at) ON config.config_serviceclears it. That is whatmcpalready holds, so no grant moves. The trap worth recording: a missing SELECT onconfig_serviceraises the identical error, so only isolating the grants separates the two causes.And a note on
delete_mute_ruleThe filing's warning stands and is now covered: the trigger is
AFTER INSERT OR UPDATE OR DELETE, so un-muting bumps the beacon too. That was the more dangerous direction — an operator un-mutes, believes alerting is restored, and a stale cache keeps suppressing.Merge pull request #3326 from erikdarlingdata/fix/3315-mute-rules-bum…
erikdarlingdata commentedon Sep 11, 2026
Claude posting for Erik Darling
Fixed by #3326, merged to
devasf87e3c3c.StorageVersion.SchemaVersionis now 117 andtrg_bump_mute_rulesis live onconfig.config_mute_rules. Closing explicitly — closing keywords no-op ondev.See the correction above: the Viewer path was never broken, so the trigger is the right fix because
PgMuteRuleStorehas five mutating statements rather than because a second writer needed rescuing.Merge origin/dev, renumbering this rung to V118
Merge pull request #3331 from erikdarlingdata/chore/changelog-3313-3315
Point every rung citation for alert_persistence_state at V118, and pi…
Bump the config_version beacon on every config_mute_rules write (erik…
Add the erikdarlingdata#3315 and erikdarlingdata#3313 CHANGELOG entries