Skip to content

Postgres alert watermarks (5 alerts) don't survive a service restart, unlike AlertEngine's #2716

Closed

Description

@erikdarlingdata

Follow-up from review on #2713 (Postgres Deadlocks/Blocking alerts, part of #2711).

The finding

`AlertEngine`'s SQL Server watermarks (`_lastAlertedDeadlockCount`/`_lastAlertedBlockingCount`) are seeded from `IAlertStateStore` on startup via `EnsureWatermarksSeededAsync`, so a host restart doesn't re-fire an event still sitting in the rolling window.

The Postgres side has no equivalent for ANY of its five alerts:

All of these are purely in-process `ConcurrentDictionary`s with no persistence or seeding. A Darling service restart (routine deployment, crash-and-recover, etc.) resets every watermark to 0. If a deadlock/blocking situation — or a wraparound/xmin/slot condition — is still inside its relevant window at that point, the very next sweep sees the condition still present with a zeroed watermark and fires again for something already reported before the restart. This is the #1091/#2704/#2708 re-fire class, re-opened at the restart boundary instead of the cooldown boundary.

Why this wasn't fixed in #2713

This isn't specific to the two alerts #2713 added — it's a property Postgres alerting has had since the three Tier 0 predictors shipped, and #2713 just matched that existing precedent rather than fixing it unilaterally for two of five alerts while leaving the other three inconsistent. Fixing the category (all five) properly means porting `AlertEngine`'s `IAlertStateStore`-seeding pattern to the Postgres side, which is a real, separate piece of work — not something to bolt onto a PR scoped to "add two alerts."

Scope, if picked up

Whatever the fix, it should cover all five Postgres alerts uniformly (the three existing predictors AND the two from #2713), not just a subset — this is exactly the "fix the category, not the instance" situation. Options, not yet decided:

  • Port `AlertEngine`'s `EnsureWatermarksSeededAsync`/`IAlertStateStore` seeding pattern to the Postgres alert path.
  • A lighter-weight Postgres-specific persistence shape, if the full `IAlertStateStore` contract is more than these five alerts need.

Where this plugs in

`Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs` — `_lastPostgresAlert`, `_lastPgDeadlockAlert`/`_activePgDeadlockAlert`/`_lastAlertedPgDeadlockCount`, `_lastPgBlockingAlert`/`_activePgBlockingAlert`/`_lastAlertedPgBlockingCount`, and `EvaluatePostgresAlertsAsync`/`EvaluatePgDeadlocksAsync`/`EvaluatePgBlockingAsync`. Reference implementation: `AlertEngine`'s own `EnsureWatermarksSeededAsync` and `IAlertStateStore`.

Activity

  1. added 2 commits that reference this issue on Aug 31, 2026
  2. erikdarlingdata commented on Sep 1, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed by #2723 (all 5 Postgres alert watermarks now survive a service restart). Merged to dev; ships in the next release.

  3. added 2 commits that reference this issue on Sep 10, 2026
    5d2f43c
    1ada0f5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions