Skip to content

Lite connection alerts + generic webhook channel (#1506) - #1511

Merged
erikdarlingdata merged 4 commits into
devfrom
feature/1506-connection-alerts
Jul 13, 2026
Merged

erikdarlingdata merged 4 commits into
devfrom
feature/1506-connection-alerts

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Closes the reporter's ask in #1506. A user monitors an Azure SQL DB with Lite; his public IP rotates ~1-2x/day, so the Azure SQL firewall rule stops matching and his connection breaks. He asked for a way to run an app/ps1 on connection loss so he can notify himself and re-run a GitHub Action to re-authorize access. This delivers that need without an arbitrary-command runner, plus the missing connection alert.

Why a generic webhook instead of executing a script/exe

Running an operator-configured script or executable when an alert fires is an EDR-flagged attack path: a signed, trusted binary that spawns arbitrary processes is exactly what endpoint protection (and attackers) look for, and it turns any write to the settings store into code execution. A generic webhook reaches the same automation surface — PagerDuty, Opsgenie, n8n, or a GitHub repository_dispatch — with no process-execution surface at all. It is data on a socket, not a command.

The reporter's exact use case, solved

His "re-run the GitHub Action" is an HTTP POST. Configure the generic webhook (Lite: Settings → Webhooks → Generic Webhook):

  • URL: https://api.github.com/repos/OWNER/REPO/dispatches
  • Headers: {"Authorization": "Bearer ghp_YOUR_TOKEN", "Accept": "application/vnd.github+json"}
  • Body: {"event_type": "sql-monitor-alert", "client_payload": {"metric": "{{metric}}", "server": "{{server}}"}}

His repo's workflow triggers on: repository_dispatch, and the token needs Contents: write. When Lite loses the connection it fires Server Unreachable, the webhook POSTs, GitHub re-runs the workflow that re-adds his IP to the firewall rule.

What's in this PR

1. Lite connection-lost / connection-restored alerts (Dashboard/Darling parity)

Lite was the only one of the three apps that couldn't tell you your server went down unless you were watching the tray — Dashboard emails on that edge, Darling emails and webhooks it. Both edges in CheckConnectionsAndNotify now also route through Lite's alert path (email + webhook + config_alert_log row) as "Server Unreachable" / "Server Restored" — the metric names the shared AlertSeverity map and AlertMetricClassifier already recognize (CRITICAL-red; the restore classified as a green resolution), so no new naming.

  • The tray balloon is unchanged — this is purely additive.
  • Fires once per edge. The edge-trigger logic moved out of the inline WPF condition into a pure, unit-tested ConnectionEdgeDetector, so a server offline for 8 hours produces one "Server Unreachable", not one per poll.
  • Respects mute rules and per-server silencing exactly like Lite's other alerts; writes the alert-history row the same way.
  • Gated on the existing App.NotifyConnectionChanges pref (default on, already in Settings, same name Darling uses) — no duplicate pref added (the task suggested a new NotifyOnConnectionLost, but that already exists in Lite under this name; adding a second would drift).

2. Generic webhook channel (shared lib → Lite + Darling)

A third channel in WebhookAlertService beside Teams/Slack, mirroring their structure (health tracking, log throttle, proxy support). Settings: GenericWebhookEnabled/Url/HeadersJson/BodyTemplate/ProxyAddress on IAlertSettings; TrySendGenericAlertAsync / SendTestGenericAsync / GetGenericHealth.

  • JSON-injection safe. Placeholders substitute inside JSON string literals, so every value is JSON-escaped — a server name with " or \ (e.g. HOST\INSTANCE) can't break out. The substituted body is validated as JSON before it goes on the wire.
  • Robust. Substitution is single-pass (one field can't pull another's contents into itself); the escape tolerates SQL Server lone surrogates instead of throwing (which would drop the alert on this channel while the others deliver); malformed headers JSON and CR/LF in a header value are rejected with a clear error; a User-Agent is defaulted (GitHub 403s without one). Nothing throws into the alert loop.
  • Secrets. URL + headers JSON carry bearer tokens → Credential Manager (Lite); Darling control-plane V26 columns generic_url + generic_headers, column-REVOKEd from the read-only viewer role (the union-equals-table invariant is build-gated, so a leak would fail CI).
  • UI. Settings section + "Send Test" button beside Teams/Slack in both Lite and the Darling viewer, with the repository_dispatch example inline.

Parity / scope

  • Dashboard (deprecated, bug-fixes only) satisfies the shared interface with the channel off and gains no new UI.
  • Darling gets the channel end to end: config, DarlingAlertSettings, the Postgres control plane (migration V26, StoreConfigProvider read, viewer upsert/read + secret carve-out), and the viewer Settings UI.

Security review

An inline security review (JSON injection, header/CRLF smuggling, secret logging, SSRF, exceptions into the collection loop) verified the escaping is airtight and found four issues, all fixed in this PR: the lone-surrogate throw, CRLF header smuggling, an unbounded logged response body, and cross-field placeholder re-expansion. One optional item was not taken: blocking an http:// URL that also sets an Authorization header. Blocking it would break legitimate internal-LAN automation endpoints (the operator's own token, own endpoint, own trusted network — no privilege boundary crossed); flagging for a maintainer call rather than silently restricting.

Testing

Ran the sanctioned filtered suites (Installer.Tests not run):

  • Lite.Tests (filtered): 1011 passed, 0 failed — includes new GenericWebhookTests (escaping incl. quote/backslash/lone-surrogate, malformed + CRLF headers, single-pass substitution, the validator) and ConnectionEdgeDetectorTests (once-per-edge, incl. the reporter's daily-break pattern).
  • Darling.Tests: 2074 passed, 133 skipped, 0 failed (updated the schema-version, migration-count, viewer-probe, and viewer-role-secret-column guard tests for V26).
  • Dashboard.Tests: 732 passed, 0 failed.
  • Full solution dotnet build: 0 errors.

🤖 Generated with Claude Code

erikdarlingdata and others added 4 commits July 13, 2026 11:13
Close two gaps for the Azure SQL DB user whose connection breaks 1-2x/day
when his public IP rotates out of the firewall rule.

1. Lite connection-lost / connection-restored alerts. Lite was the only one
   of the three apps that fired a tray balloon and nothing else on the
   connection edge (Dashboard emails; Darling emails + webhooks). Both edges
   in CheckConnectionsAndNotify now also route through Lite's alert path
   (email + webhook + config_alert_log row) as "Server Unreachable" /
   "Server Restored" -- the metric names the shared AlertSeverity map and
   AlertMetricClassifier already understand. The tray balloon is unchanged.
   Edge-trigger logic is extracted into a pure, unit-tested
   ConnectionEdgeDetector so an 8-hour outage fires ONCE, not once per poll.
   Respects mute rules + per-server silencing; gated on the existing
   App.NotifyConnectionChanges pref (no duplicate pref added).

2. Generic webhook channel in the shared WebhookAlertService, for Lite AND
   Darling. POSTs an operator-authored JSON body (with {{metric}}/{{server}}/
   {{value}}/{{threshold}}/{{severity}}/{{context}}/{{timestamp}} placeholders)
   and arbitrary JSON headers to any URL -- covers PagerDuty/Opsgenie/n8n and,
   the motivating case, a GitHub repository_dispatch that re-runs a workflow.
   Deliberately NOT a script/exe runner (no process-exec surface in a signed
   binary). Every placeholder is JSON-escaped (a server name with a quote or
   backslash cannot break out of the template); substitution is single-pass
   (no cross-field re-expansion); the escape tolerates lone surrogates instead
   of throwing; malformed headers JSON / CR-LF in a header degrade to a clear
   error, never an exception in the alert loop. URL + headers JSON are secrets
   (Credential Manager in Lite; Darling V26 control-plane columns generic_url +
   generic_headers, column-REVOKEd from the read-only viewer role). Settings
   UI + Send Test button beside Teams/Slack in both apps. Dashboard (deprecated)
   satisfies the interface with the channel off and gains no new UI.

Tests: new GenericWebhookTests (escaping incl. quote/backslash/surrogate,
malformed + CRLF headers, single-pass substitution, validator) and
ConnectionEdgeDetectorTests (once-per-edge, incl. the reporter's daily-break
pattern). Lite 1011 pass, Darling 2074 pass, Dashboard 732 pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Applies the five review findings on the PR:

1. [security] Cleartext-http warning. A non-blocking Yes/No confirm at
   generic-webhook Save AND Test in both Lite and the Darling viewer when the
   URL is http:// (not https) and at least one header is present -- the
   Authorization token would otherwise go on the wire in cleartext. NOT
   blocked (a plaintext POST to a trusted LAN listener is legitimate). Shared
   WebhookAlertService.IsCleartextHttpWithHeaders drives both apps identically.

2. Extend the Darling ViewerNotificationSqlTests column parity test from 12 to
   all 16 config_notification columns (+ the four generic_* columns) and the
   bound-parameter loop to $16, matching the production "16 columns" contract.

3. Correct the BuildGenericPayload summary + CHANGELOG to cite JsonSerializer,
   not JsonEncodedText, as the escape mechanism -- the impl deliberately avoids
   JsonEncodedText.Encode because it throws on SQL Server lone surrogates.

4. Don't freeze the default body template on save. The settings box is
   pre-filled with DefaultGenericBodyTemplate, so saving would otherwise persist
   a frozen copy and lock the operator out of future default improvements. Both
   apps now persist the empty "use default" sentinel unless the box was actually
   edited (shared WebhookAlertService.IsDefaultBodyTemplate, newline-insensitive).

5. Add ApplyHeaders unit tests (User-Agent defaulted when absent, not overridden
   when set, Content-Type routed onto the content). Made ApplyHeaders internal.

Tests: Lite 1015 pass (4 new ApplyHeaders tests), Darling 2074 pass. Both apps
build clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant