You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Postgres alert watermarks (5 alerts) don't survive a service restart, unlike AlertEngine's #2716
Follow-up from review on #2713 (Postgres Deadlocks/Blocking alerts, part of #2711).
The finding
`AlertEngine`'s SQL Server watermarks (`_lastAlertedDeadlockCount`/`_lastAlertedBlockingCount`) are seeded from `IAlertStateStore` on startup via `EnsureWatermarksSeededAsync`, so a host restart doesn't re-fire an event still sitting in the rolling window.
The Postgres side has no equivalent for ANY of its five alerts:
The three existing Tier 0 predictors (`_lastPostgresAlert` — wraparound/xmin/replication slot)
All of these are purely in-process `ConcurrentDictionary`s with no persistence or seeding. A Darling service restart (routine deployment, crash-and-recover, etc.) resets every watermark to 0. If a deadlock/blocking situation — or a wraparound/xmin/slot condition — is still inside its relevant window at that point, the very next sweep sees the condition still present with a zeroed watermark and fires again for something already reported before the restart. This is the #1091/#2704/#2708 re-fire class, re-opened at the restart boundary instead of the cooldown boundary.
This isn't specific to the two alerts #2713 added — it's a property Postgres alerting has had since the three Tier 0 predictors shipped, and #2713 just matched that existing precedent rather than fixing it unilaterally for two of five alerts while leaving the other three inconsistent. Fixing the category (all five) properly means porting `AlertEngine`'s `IAlertStateStore`-seeding pattern to the Postgres side, which is a real, separate piece of work — not something to bolt onto a PR scoped to "add two alerts."
Scope, if picked up
Whatever the fix, it should cover all five Postgres alerts uniformly (the three existing predictors AND the two from #2713), not just a subset — this is exactly the "fix the category, not the instance" situation. Options, not yet decided:
Port `AlertEngine`'s `EnsureWatermarksSeededAsync`/`IAlertStateStore` seeding pattern to the Postgres alert path.
A lighter-weight Postgres-specific persistence shape, if the full `IAlertStateStore` contract is more than these five alerts need.
Where this plugs in
`Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs` — `_lastPostgresAlert`, `_lastPgDeadlockAlert`/`_activePgDeadlockAlert`/`_lastAlertedPgDeadlockCount`, `_lastPgBlockingAlert`/`_activePgBlockingAlert`/`_lastAlertedPgBlockingCount`, and `EvaluatePostgresAlertsAsync`/`EvaluatePgDeadlocksAsync`/`EvaluatePgBlockingAsync`. Reference implementation: `AlertEngine`'s own `EnsureWatermarksSeededAsync` and `IAlertStateStore`.
Activity
Fix dedupKey filter for non-hex values (review finding on #2723)
Merge pull request #2723 from erikdarlingdata/fix/2716-pg-alert-water…
erikdarlingdata commentedon Sep 1, 2026
Fixed by #2723 (all 5 Postgres alert watermarks now survive a service restart). Merged to
dev; ships in the next release.Fix erikdarlingdata#2716: none of the 5 Postgres alerts' restart wate…
Add Postgres Poison Wait analogue alert (erikdarlingdata#2711)