Darling: let another service send alerts through Darling's channels, routes and mute rules #5654

Claude posting for Erik Darling.

Let a second service raise alerts through Darling. The service records an alert in the store. Darling delivers it through its own channels, routes and mute rules. The second service never holds a webhook URL or any other secret.

Why

• Some checks live outside Darling. Examples are a scheduled script, a job, or a companion service that watches something Darling does not collect.
• Today such a service must post to Slack, Teams or email itself. So it needs its own copy of the webhook URLs. It also skips Darling's routes, mute rules and cooldowns.
• Since Darling: store SQL login passwords on Linux, sealed to the service's key #5387, Darling seals saved webhook URLs and the other secrets to the service's key. Only the service can open them. That is the right model. But it means that another service cannot use the saved URLs.

Design (draft)

1. A narrow write. A new SECURITY DEFINER function, for example config.submit_external_alert(...), records one pending alert. It follows the pattern of config.record_custom_alert_resolution (Custom-alert resolve-on-delete silently skips the recovery row on mcp/viewer paths (config_alert_log grant gap in #3305) #3334):
   • A fixed search_path.
   • A fully parameterized insert, with no dynamic SQL.
   • REVOKE ALL ... FROM PUBLIC, then a narrow GRANT EXECUTE.
2. Registered sources. The operator registers each source by name. A source connects with a login that can execute the function and do nothing else. The function records which source called it.
3. A pending table, separate from alert history. The function writes to a new pending table, not to config_alert_log. So a source can never write a row that claims a delivery.
4. Darling delivers. The alert pass reads the pending rows. It sends each one through the normal channels, routes, mute rules and cooldowns, the same as its own alerts. Then it writes the usual config_alert_log row. It marks the pending row as delivered, muted or failed, with the reason.
5. Fire and clear. A source can send a fire and a clear for one key (the source, the server name and the metric). So state tracking and resolution notices work the same as for Darling's own alerts.
6. Limits. Each source has a rate limit and a maximum payload size. Severity comes from a fixed list. A pending row that is not delivered in time is marked expired, and the self-monitor reports it.
7. Visibility. Alert history, the viewer and the MCP tools show the source of each alert. A mute rule can match a source.

Acceptance tests

• A source login can execute the function and nothing else. It cannot read webhook URLs, other secrets or alert history.
• A source cannot insert into config_alert_log directly.
• A pending alert is delivered once, also across a service restart.
• A mute rule that matches the alert stops its delivery, and the history row records it as muted.
• Routes and cooldowns apply the same as for Darling's own alerts.
• A fire and then a clear sends the resolution notice.
• A source over its rate limit is refused, with a clear error.

Open questions for Erik

1. One login per source, or one shared role with the source name as an argument?
2. Delivery delay: read the pending rows in the alert pass, or wake on PostgreSQL LISTEN/NOTIFY?
3. Darling only? Lite has no shared store, so this draft leaves Lite out.

Planned for next week.