Connection alerts: announce already-down servers and re-alert during standing outages (#1659)

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Connection alerts for servers that are already down, and re-alerts during a standing outage** ([#1674]) - closes #1659, the gap split out of #1535: connection alerts were pure edge detection, so an app or service that started while a server was already unreachable never announced the outage (no edge existed), and a standing outage produced exactly one alert however long it lasted - which silently broke the reporter's webhook-driven auto-heal loop the day the app restarted mid-outage. Two OPT-INS, both default-off so the classic one-alert-per-outage behavior is untouched: **alert at first sight** (announce a server already down on the first-ever observation) and **re-alert every N minutes while still down** (0 = off). Re-fires deliver under the SAME `Server Unreachable` metric name deliberately - webhook automation keyed on the metric re-triggers, which is the whole point - with the detail text marking the flavor (`Already unreachable when monitoring started` / `Still unreachable (re-alerting every N min)`). The decision is ONE shared definition (`ConnectionAlertPolicy` in PerformanceMonitor.Common, replacing Lite's `ConnectionEdgeDetector` and the inline machine in Darling's `DarlingSelfAlertEvaluator` - the `SqlErrorClassification` discipline, pinned from both test suites), and the two opt-ins interlock: even with the startup announcement off, re-fire alone re-announces after a mid-outage restart, because the re-baselined outage has no recorded down alert and is due immediately. The re-fire clock stamps on DELIVERY only, so an alert suppressed by the notify toggles never consumes the window. Lite: two settings beside the existing connection toggle (settings.json + Settings window). Darling: V33 store columns on `config_alert_settings` (read live like the V20 toggle; refire clamped 0-1440), editable from the viewer's Settings window; no ACL/provisioning change (the table carries table-level grants, no column carve).
 - **Darling: `--configure-network` can now expose the WEB DASHBOARD** ([#1617]) - the wizard offered Store and MCP but not the web dashboard, even though `--enable-web`'s own output told operators to run `--configure-network` to expose it on the LAN - a dead end that forced hand-editing `web.network` into darling.json. Web is now a first-class third surface, fully symmetric with Store/MCP: its own menu choice (plus comma combinations like `1,3`, and `4` = all three), a keep-or-generate DPAPI access token, listen/CIDR inputs validated by the SAME bind resolver the web host fail-closes on (extracted as `ResolveWebBind`, the web twin of `ResolveMcpBind` - never a reimplementation), the comment-preserving `web.network` write, a one-time token print, and next-steps text including the browser login URL (`http://<listen>:<port>/?token=...`, exchanged for a session cookie). Disable now removes all three network blocks. After the wizard, `--enable-web` opens the scoped firewall rule on the first try - no hand-editing required.
 - **Darling Web: adaptive `auto` time bucket for composed time-series panels** ([#1619]) - a compose custom-view time-series panel can set `timeBucket: "auto"`, and the compiler resolves it to a concrete grain from the panel's window (minute up to 2 days, hour up to 60 days, day beyond) so any range from 1h to 90d renders a readable line and never trips the 5,000-bucket ceiling. Previously a fixed `hour` bucket collapsed a sub-hour workload (e.g. a 30-minute HammerDB run) into a single invisible point, while a fixed `minute` bucket errored past ~3.5 days. The composer now defaults new time-series panels to `auto` and the MCP `describe_custom_view_catalog` recommends it; non-auto buckets compile byte-for-byte as before. Pinned by `DarlingComposeTests` (auto boundary resolution + compile-by-window + non-auto passthrough).
 - **Darling Web: custom time span on the view range picker** ([#1619]) - the rendered custom-view range picker gains a "Custom..." option (a number + a hours/days unit, up to the 90-day window ceiling) so a view is no longer limited to the six presets; a non-preset window stays selectable so it survives the 60s refresh.
@@ -568,6 +569,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 [#1636]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/1636
 [#1637]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1637
 [#1639]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1639
+[#1674]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1674
 [#1668]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1668
 [#1670]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1670
 [#1640]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1640

Darling/Darling.Tests/ConnectionAlertPolicyTests.cs

@@ -0,0 +1,99 @@
+/*
+ * Copyright (c) 2026 Erik Darling, Darling Data LLC
+ *
+ * This file is part of the SQL Server Performance Monitor.
+ *
+ * Licensed under the MIT License. See LICENSE file in the project root for full license information.
+ */
+
+using System;
+using PerformanceMonitor.Common;
+using Xunit;
+
+namespace Darling.Tests;
+
+/// <summary>
+/// Pins the SHARED connection-alert decision (<see cref="ConnectionAlertPolicy"/>, #1659) from Darling's suite —
+/// mirrored in Lite.Tests, the AzureMasterFallback discipline for classes both apps depend on. The rules
+/// replaced Lite's <c>ConnectionEdgeDetector</c> and the inline machine in Darling's
+/// <c>DarlingSelfAlertEvaluator</c>: identical edge-only behavior by default (a transition alerts exactly
+/// once; the first sighting is a silent baseline), plus the two opt-ins — announce a server already down at
+/// first sight, and re-announce a standing outage every N minutes.
+/// </summary>
+public sealed class ConnectionAlertPolicyTests
+{
+    private static readonly DateTime Now = new(2026, 7, 26, 12, 0, 0, DateTimeKind.Utc);
+
+    private static ConnectionAlertDecision Decide(
+        bool? previous, bool online,
+        bool startupOptIn = false, int refireMinutes = 0, DateTime? lastDownUtc = null) =>
+        ConnectionAlertPolicy.Decide(
+            previous, online, startupOptIn,
+            refireMinutes > 0 ? TimeSpan.FromMinutes(refireMinutes) : null,
+            lastDownUtc, Now);
+
+    /* ── the classic edge-only semantics, unchanged with both opt-ins off ── */
+
+    [Theory]
+    [InlineData(null, true, ConnectionAlertDecision.None)]    /* first sighting online: silent baseline */
+    [InlineData(null, false, ConnectionAlertDecision.None)]   /* first sighting down: silent baseline (the #1659 gap, by default) */
+    [InlineData(true, true, ConnectionAlertDecision.None)]    /* steady online */
+    [InlineData(true, false, ConnectionAlertDecision.Lost)]   /* the edge */
+    [InlineData(false, true, ConnectionAlertDecision.Restored)]
+    [InlineData(false, false, ConnectionAlertDecision.None)]  /* standing outage: one alert per outage, by default */
+    public void Defaults_AreExactlyTheOldEdgeOnlyBehavior(bool? previous, bool online, ConnectionAlertDecision expected) =>
+        Assert.Equal(expected, Decide(previous, online));
+
+    /* ── opt-in 1: already down at first sight ── */
+
+    [Fact]
+    public void StartupOptIn_AnnouncesAServerAlreadyDownOnItsFirstObservation()
+    {
+        Assert.Equal(ConnectionAlertDecision.AlreadyDownAtFirstSight, Decide(null, online: false, startupOptIn: true));
+
+        /* A first sighting that is ONLINE stays silent — the opt-in announces outages, not baselines. */
+        Assert.Equal(ConnectionAlertDecision.None, Decide(null, online: true, startupOptIn: true));
+    }
+
+    /* ── opt-in 2: standing-outage re-fire ── */
+
+    [Fact]
+    public void Refire_FiresWhenTheIntervalHasElapsed_AndNotBefore()
+    {
+        /* Inside the window: quiet. */
+        Assert.Equal(ConnectionAlertDecision.None,
+            Decide(false, online: false, refireMinutes: 10, lastDownUtc: Now.AddMinutes(-5)));
+
+        /* At/past the window: re-announce. */
+        Assert.Equal(ConnectionAlertDecision.StillDown,
+            Decide(false, online: false, refireMinutes: 10, lastDownUtc: Now.AddMinutes(-10)));
+        Assert.Equal(ConnectionAlertDecision.StillDown,
+            Decide(false, online: false, refireMinutes: 10, lastDownUtc: Now.AddMinutes(-60)));
+    }
+
+    /// <summary>
+    /// Down with NO recorded down alert and re-fire on = due immediately. This is what re-announces an
+    /// outage after a mid-outage restart even when the startup opt-in is off: the first sighting
+    /// re-baselines silently, and the next offline→offline poll lands here.
+    /// </summary>
+    [Fact]
+    public void Refire_WithNoRecordedDownAlert_IsDueImmediately() =>
+        Assert.Equal(ConnectionAlertDecision.StillDown,
+            Decide(false, online: false, refireMinutes: 10, lastDownUtc: null));
+
+    [Fact]
+    public void Refire_Off_NeverReannounces() =>
+        Assert.Equal(ConnectionAlertDecision.None,
+            Decide(false, online: false, refireMinutes: 0, lastDownUtc: Now.AddDays(-1)));
+
+    /* ── the opt-ins never disturb the edges themselves ── */
+
+    [Fact]
+    public void Restored_AndLost_AreUnchangedByTheOptIns()
+    {
+        Assert.Equal(ConnectionAlertDecision.Restored,
+            Decide(false, online: true, startupOptIn: true, refireMinutes: 10, lastDownUtc: Now.AddMinutes(-60)));
+        Assert.Equal(ConnectionAlertDecision.Lost,
+            Decide(true, online: false, startupOptIn: true, refireMinutes: 10));
+    }
+}

Darling/Darling.Tests/DarlingObservabilityTests.cs

@@ -32,9 +32,9 @@ public sealed class DarlingObservabilityTests
     private const int TestServerId = -424242;
 
     [Fact]
-    public void MigrationScripts_ThirtyTwoVersions_V31CustomViews_V32ServerTags()
+    public void MigrationScripts_ThirtyThreeVersions_V32ServerTags_V33ConnectionAlertOptIns()
     {
-        Assert.Equal(32, PgMigrations.Scripts.Count);
+        Assert.Equal(33, PgMigrations.Scripts.Count);
         Assert.Equal(1, PgMigrations.Scripts[0].Version);
         Assert.Equal(2, PgMigrations.Scripts[1].Version);
         Assert.Equal(3, PgMigrations.Scripts[2].Version);
@@ -67,7 +67,8 @@ public void MigrationScripts_ThirtyTwoVersions_V31CustomViews_V32ServerTags()
         Assert.Equal(30, PgMigrations.Scripts[29].Version);
         Assert.Equal(31, PgMigrations.Scripts[30].Version);
         Assert.Equal(32, PgMigrations.Scripts[31].Version);
-        Assert.Equal(32, StorageVersion.SchemaVersion);
+        Assert.Equal(33, PgMigrations.Scripts[32].Version);
+        Assert.Equal(33, StorageVersion.SchemaVersion);
 
         /* V26 (#1506) adds the generic webhook channel's four columns to the V17 control-plane table.
            Schema-qualified config.* and IF NOT EXISTS, per the file's additive-ALTER idiom. */

Darling/Darling.Tests/DarlingSelfAlertTests.cs

@@ -119,13 +119,19 @@ private sealed class Harness
         /// <summary>The V20 connection-change notify gate, read live by the evaluator's connect edge (default on).</summary>
         public bool NotifyConnectionChanges { get; set; } = true;
 
+        /// <summary>#1659 opt-ins (V33), read live like the V20 gate. Defaults off = classic edge-only.</summary>
+        public bool NotifyConnectionDownAtStartup { get; set; }
+        public int ConnectionRefireMinutes { get; set; }
+
         public DateTime Now { get; set; } = new(2026, 7, 1, 12, 0, 0, DateTimeKind.Utc);
 
         public DarlingSelfAlertEvaluator Build() => new(
             Settings, Deliverer, History,
             _ => MuteThrows ? throw new InvalidOperationException("mute check boom") : Muted,
             logger: null, utcNow: () => Now,
-            notifyConnectionChanges: () => NotifyConnectionChanges);
+            notifyConnectionChanges: () => NotifyConnectionChanges,
+            notifyConnectionDownAtStartup: () => NotifyConnectionDownAtStartup,
+            connectionRefireMinutes: () => ConnectionRefireMinutes);
     }
 
     /* ---------------- collection-stopped detection (pure) ---------------- */
@@ -377,6 +383,73 @@ public async Task AgentNotRunning_AlertsDisabled_DoesNotFire()
 
     /* ---------------- connection lost / restored edge ---------------- */
 
+    /* ---------------- #1659: already-down-at-first-sight + standing-outage re-fire ---------------- */
+
+    [Fact]
+    public async Task Connection_AlreadyDownAtFirstSight_OptIn_FiresOnTheFirstOutcome()
+    {
+        var h = new Harness { NotifyConnectionDownAtStartup = true };
+        var e = h.Build();
+
+        /* Unknown -> Offline with the opt-in: the outage is announced at first sight instead of being a
+           silent baseline — the case where the service starts mid-outage and would otherwise never alert. */
+        await e.ApplyConnectionOutcomeAsync(ServerId, Name, online: false, error: "no route", Ct);
+
+        var fired = Assert.Single(h.Deliverer.Outcomes);
+        Assert.Equal("Server Unreachable", fired.MetricName);
+        Assert.Contains("Already unreachable when monitoring started", fired.DetailText, StringComparison.Ordinal);
+    }
+
+    [Fact]
+    public async Task Connection_Refire_FiresAfterTheInterval_NotBefore_AndUnderTheSameMetricName()
+    {
+        var h = new Harness { ConnectionRefireMinutes = 10 };
+        var e = h.Build();
+
+        /* Establish online, then lose it: the classic edge fires once. */
+        await e.ApplyConnectionOutcomeAsync(ServerId, Name, online: true, error: null, Ct);
+        await e.ApplyConnectionOutcomeAsync(ServerId, Name, online: false, error: "no route", Ct);
+        Assert.Single(h.Deliverer.Outcomes);
+
+        /* Inside the window: offline->offline stays quiet. */
+        h.Now = h.Now.AddMinutes(5);
+        await e.ApplyConnectionOutcomeAsync(ServerId, Name, online: false, error: "no route", Ct);
+        Assert.Single(h.Deliverer.Outcomes);
+
+        /* Past the window: the standing outage re-announces — SAME metric name, so webhook automation
+           keyed on "Server Unreachable" re-triggers; the detail says it is a re-fire. */
+        h.Now = h.Now.AddMinutes(6);
+        await e.ApplyConnectionOutcomeAsync(ServerId, Name, online: false, error: "no route", Ct);
+        Assert.Equal(2, h.Deliverer.Outcomes.Count);
+        Assert.Equal("Server Unreachable", h.Deliverer.Outcomes[1].MetricName);
+        Assert.Contains("Still unreachable", h.Deliverer.Outcomes[1].DetailText, StringComparison.Ordinal);
+
+        /* Restore clears the re-fire clock and fires the classic restore. */
+        h.Now = h.Now.AddMinutes(1);
+        await e.ApplyConnectionOutcomeAsync(ServerId, Name, online: true, error: null, Ct);
+        Assert.Equal(3, h.Deliverer.Outcomes.Count);
+        Assert.Equal("Server Restored", h.Deliverer.Outcomes[2].MetricName);
+    }
+
+    [Fact]
+    public async Task Connection_Refire_ClockStampsOnDeliveryOnly_SoASuppressedDecisionDoesNotConsumeTheWindow()
+    {
+        var h = new Harness { ConnectionRefireMinutes = 10, NotifyConnectionChanges = false };
+        var e = h.Build();
+
+        /* Down while the notify toggle is OFF: state advances, nothing delivers, nothing stamps. */
+        await e.ApplyConnectionOutcomeAsync(ServerId, Name, online: true, error: null, Ct);
+        await e.ApplyConnectionOutcomeAsync(ServerId, Name, online: false, error: "no route", Ct);
+        Assert.Empty(h.Deliverer.Outcomes);
+
+        /* Toggle on mid-outage: the very next offline->offline poll is due immediately (no recorded down
+           alert to measure the window from), so the outage is announced rather than silently aged. */
+        h.NotifyConnectionChanges = true;
+        await e.ApplyConnectionOutcomeAsync(ServerId, Name, online: false, error: "no route", Ct);
+        var fired = Assert.Single(h.Deliverer.Outcomes);
+        Assert.Equal("Server Unreachable", fired.MetricName);
+    }
+
     [Fact]
     public async Task Connection_FirstConnect_IsSilentBaseline()
     {

Darling/Darling.Tests/DarlingServerTagsTests.cs

@@ -23,13 +23,21 @@ public sealed class DarlingServerTagsTests
         PgMigrations.Scripts.Single(s => s.Version == 32).Sql;
 
     [Fact]
-    public void V32_IsRegisteredLast_AndSchemaQualified_AndIdempotent()
+    public void V32_IsSchemaQualified_AndV33_IsRegisteredLast()
     {
         var v32 = PgMigrations.Scripts.Single(s => s.Version == 32);
 
         Assert.Equal("server-tags", v32.Name);
-        Assert.Equal(32, PgMigrations.Scripts[^1].Version);
-        Assert.Equal(StorageVersion.SchemaVersion, v32.Version);
+
+        /* V33 (#1659 connection-alert opt-ins) is the newest migration and tracks the build version;
+           its ALTERs are config.-qualified like every additive control-plane migration. */
+        var v33 = PgMigrations.Scripts.Single(s => s.Version == 33);
+        Assert.Equal("connection-alert-refire", v33.Name);
+        Assert.Equal(33, PgMigrations.Scripts[^1].Version);
+        Assert.Equal(StorageVersion.SchemaVersion, v33.Version);
+        Assert.Contains("ALTER TABLE config.config_alert_settings", v33.Sql, StringComparison.Ordinal);
+        Assert.Contains("notify_connection_down_at_startup boolean NOT NULL DEFAULT false", v33.Sql, StringComparison.Ordinal);
+        Assert.Contains("connection_refire_minutes integer NOT NULL DEFAULT 0", v33.Sql, StringComparison.Ordinal);
 
         /* config.-QUALIFIED: the migrate session runs under search_path = collect, config, public, so an
            unqualified CREATE TABLE would land in collect — the wrong schema AND the wrong ACL. */

Darling/Darling.Tests/ViewerDataServiceTests.cs

@@ -428,6 +428,12 @@ public void MapProbedSchemaVersion_TakesTheHighestSatisfiedSentinel(
             hasConfigControlPlane, hasAlertDeliveryOverride, hasAnalysisState, hasAlertTuningKnobs, hasDefaultTraceEvents, hasIndexObjectStatsLatestIndex, hasCollectionLogHypertableOrPlainPg, hasJobHistory, hasAgentStatus, hasGenericWebhook, hasDeadlocksDatabaseName, hasQueryStoreReplicaRole, hasLongQueryCompletions, hasWebDashboardConfig, hasCustomViews, hasServerTags));
     }
 
+    /// <summary>An upgraded store mid-state: fleet tags present (V32) but the #1659 knobs not yet — the
+    /// probe reports 32, not 33, so the gate correctly blocks a V33 viewer until the service migrates.</summary>
+    [Fact]
+    public void MapProbedSchemaVersion_V33KnobsAbsent_CapsAt32() =>
+        Assert.Equal(32, ViewerDataService.MapProbedSchemaVersion(true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true));
+
     [Fact]
     public void MapProbedSchemaVersion_V23CompositeIsGatedBehindV22_NotAStandaloneTopArm()
     {
@@ -450,7 +456,7 @@ public void RequiredStoreSchemaVersion_TracksTheBuildSchemaVersion_AndTheProbeCo
            the connect-time gate refuse to open the viewer against a perfectly healthy store. */
         Assert.Equal(
             ViewerDataService.RequiredStoreSchemaVersion,
-            ViewerDataService.MapProbedSchemaVersion(true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true));
+            ViewerDataService.MapProbedSchemaVersion(true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true));
     }
 }
 

Darling/PerformanceMonitor.Darling.Service/DarlingAlertSettings.cs

@@ -81,6 +81,15 @@ public DarlingAlertSettings(DarlingConfig config)
     /// </summary>
     public bool NotifyConnectionChanges => _config.Alerts.NotifyConnectionChanges;
 
+    /// <summary>#1659 opt-in (V33), read live like <see cref="NotifyConnectionChanges"/>: announce a server
+    /// already down on its first-ever connect attempt. Default false.</summary>
+    public bool NotifyConnectionDownAtStartup => _config.Alerts.NotifyConnectionDownAtStartup;
+
+    /// <summary>#1659 opt-in (V33), read live: re-announce a standing outage every N minutes (0 = off).
+    /// Clamped 0–1440 like the other store-fed numerics, so a hand-edited row can't drive a per-sweep spam
+    /// loop or a never-fires interval.</summary>
+    public int ConnectionRefireMinutes => Math.Clamp(_config.Alerts.ConnectionRefireMinutes, 0, 1440);
+
     /// <summary>"sql" → SqlProcess; anything else (incl. Lite's default "total") → TotalServer.</summary>
     public CpuAlertMode CpuAlertMode =>
         string.Equals(_config.Alerts.CpuMode, "sql", StringComparison.OrdinalIgnoreCase)