Skip to content

Lite has no AG disconnect re-fire, so a week-long replica outage is announced once #2426

Closed

Description

@erikdarlingdata

Split out of #2417. Not a regression — a gap #1726 left and #2417 made visible by forcing the two apps' MCP alert payloads to be compared key by key.

Darling has ag_disconnect_refire_minutes (#1696, store V37, clamped 0–1440, 0 = off): an AG Replica Disconnected alert is otherwise a pure edge, so a replica disconnected for a week announces itself exactly once. The re-fire re-announces under the same metric name so webhook automation keyed on it re-triggers, is stamped on DELIVERY only so a suppressed alert cannot consume the window, and is cleared on reconnect.

Lite has none of it. Not the static, not the settings.json key, not the Settings box, and — the part that makes this a feature rather than a payload key — no edge state that could re-announce. #1726 mirrored three of Darling's four AG knobs (notify_ag_health, ag_lag_alert_seconds, ag_redo_queue_alert_kb) and stopped there, which was correct at the time because Darling's fourth landed later, in #1696/#1734.

So a Lite user running Availability Groups gets one notification for an outage that may last days, and there is no setting that changes that. The asymmetry is invisible from either app on its own — which is exactly why it surfaced only when #2410's parity test started comparing the payloads.

Where it is currently exempted

#2417 reports the three knobs Lite really has rather than inventing a fourth, and exempts the fourth by name:

  • Lite.Tests/McpAlertSettingsKeyTests.cs → LiteOmittedMembers = { "ag.disconnect_refire_minutes" }, skipped in GetAlertSettings_ReportsEveryGroupDarlingDoes_SpelledDarlingsWay
  • GetAlertSettings_OmitsAgDisconnectRefire_WhichLiteHasNoEquivalentFor asserts both halves — Darling still emits it, and Lite still does not — so the exemption cannot outlive its reason in either direction

Emitting a constant 0 was rejected on the reasoning the same file already applies to self_alerts: reporting a constant under a name that reads as a knob tells an agent it can tune something the app cannot.

What closing it takes

  1. App.AgDisconnectRefireMinutes with Math.Clamp(v, 0, 1440) on load and on save, matching Darling's clamp exactly and the pattern the sibling AG knobs already use in App.LoadAlertSettings / SettingsWindow.SaveAlertSettings.
  2. The Settings box beside the existing AG rows, defaulting 0, gated with the rest of the AG family.
  3. The actual re-fire in Lite's AG edge state — the Connection alerts: announce already-down servers and re-alert during standing outages (#1659) #1674 pattern Darling follows: stamp on delivery only, clear on reconnect, same metric name.
  4. Delete the LiteOmittedMembers entry and the exemption test; the parity test then compares all four members with nothing skipped, and LiteOmittedMembers goes back to empty.

Worth checking while there: whether ConnectionRefireMinutes and the AG re-fire should share one re-fire helper, since they are the same #1674 shape and Lite now carries both.

The measurement that should decide priority: how many Lite installs run AGs at all. A fleet with no Availability Groups collects no AG rows and is unaffected, so this only bites the subset that runs them — but for that subset it is the difference between being told once and being told until it is fixed.

Activity

  1. erikdarlingdata commented on Aug 21, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed on dev by #2440 — and the answer to the question I put in the brief ("is the policy already shared, making this supply-a-setting?") is yes it is shared, and no that did not make it small.

    PerformanceMonitor.Common.AgAlertPolicy holds the pure decisions, but by explicit design each app owns its own edge STATE and its own delivery. A re-fire is made of exactly the two halves Lite lacked: a clock per ag+replica, and something that consumes it. The setting was the small part; this was implement-a-feature.

    On the one-helper question, folded — but not the way I framed it. The two re-fires are genuinely different mechanisms (per-server live probe vs per-ag+replica collected snapshot) and are not merged. What is shared is the window predicate: the same three lines carrying the same two easily-misremembered rules — non-positive is off, a missing stamp is due NOW — about to be written a third time. That is AlertRefireWindow, and ConnectionAlertPolicy calls it too. Lite also uses the shared policy through a DecideConnection overload returning StillDisconnected rather than re-deriving "None + IsDisconnected + due" by hand, which is what would have drifted.

    One deliberate departure is documented where the rule is stated: with re-fire on, an already-disconnected replica announces on first sighting. Both apps hold this state in memory, so under the silent-baseline rule alone a restart mid-outage would silence the alert permanently. Darling has behaved this way since #1696; stating it is the change, not the behaviour.

    The #2417 exemption is gone properly rather than deleted: LiteOmittedMembers is empty, the emptiness is asserted, and the exemption test is replaced by its inversion with a live-value check. That was the thing I was most worried about — a stale exemption quietly re-opening the drift the parity test exists to prevent.

    And doing the port found a real Darling bug, which is the argument for doing ports at all. Darling branched only shortMessage on the AG re-fire (DarlingSelfAlertEvaluator.cs:868), but ShortMessage reaches neither the history row nor the email — DarlingAlertDeliverer forwards DetailText and EmailTemplateBuilder ignores ShortMessage. So Darling's stored sixth re-announcement read byte-identical to its first: this issue's own complaint, standing in the view an incident is reconstructed from. Verified in source, fixed here, and worded to match Lite byte-for-byte.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions