Fix #2716: none of the 5 Postgres alerts' restart watermarks survive a service restart

[erikdarlingdata]

Closes #2716.

Problem

AlertEngine seeds its SQL Server Deadlocks/Blocking watermarks from IAlertStateStore on each server's first post-restart sweep (EnsureWatermarksSeededAsync), so a restart doesn't re-fire an event still sitting in the rolling window (#1145). Nothing equivalent existed for any of Darling's five Postgres alerts — filed as a follow-up from review on #2713.

Fix

PgDeadlocks/PgBlocking (#2711/#2713) share AlertEngine's exact rolling-count watermark shape, and #2713 deliberately used the exact same metric names ("Deadlocks Detected"/"Blocking Detected") for parity — so they now seed from and persist to the same config_edge_trigger_watermarks row AlertEngine already reads/writes. No new schema. Seeded once per (server, metric) key on first evaluation, persisted on watermark change, mirroring AlertEngine's own shape exactly.

The three Tier 0 outage predictors (wraparound/xmin/replication slot) have no count watermark at all — just a plain last-alerted TIME per subject (database/slot), which IAlertStateStore's int-watermark contract doesn't fit. Rather than add new schema for a 3-part (server, metric, subject) key, this reuses IAlertHistoryStore.GetLastAlertTimeAsync's existing #1154 dedup-key filter: finding.Subject is the #1140 dedup fingerprint these alerts already fire with, so the filter already reconstructs exactly the per-subject seed needed.

GetLastAlertTimeAsync gained an optional dedupKey parameter (mirroring its GetLastEmailSentUtcAsync/GetLastWebhookSentUtcAsync siblings) across every IAlertHistoryStore implementer: Lite's DuckDB store, Darling's Postgres store, Dashboard's deprecated JSON store, and every test fake. Guarded to one history read per (server, metric, subject) for the life of the process, not one per sweep — most subjects (most databases) never alert at all, so an unconditional per-sweep read would be a real, avoidable cost against every server, every cycle.

Verification

Built the project and the full solution clean. Since macOS can't run Darling.Tests (net10.0-windows), wrote a throwaway console harness referencing the shipped PgAlertHistoryStore/PgAlertStateStore directly (not a retyped copy) against a real local Postgres 17:

• The dedupKey filter correctly isolates two subjects alerted at different times on the same (server, metric) — each seeds from its own row, not the newer one's.
• The unfiltered call still returns the pre-fix metric-level max (backward compatible).
• A subject with no history seeds nothing (no invented cooldown).
• The watermark store round-trips (confirming the PgDeadlocks/PgBlocking persistence primitive).

All 7 checks passed. Added PgAlertHistoryStore_GetLastAlertTimeAsync_FiltersByDedupKey_ForRestartSeeding to DarlingAlertingTests.cs, gated on DARLING_TEST_PG for CI to run for real.

Test plan

• dotnet build (project + full solution) clean, 0 warnings, 0 errors
• Verified the new capability against a real local Postgres 17 via a throwaway harness (see above)
• CI's Darling PostgreSQL tests job runs the new live test for real

🤖 Generated with Claude Code

[claude]

GetLastAlertTimeAsync's dedupKey filter was built (and is documented) as hex-fingerprint-only. AlertContext.cs's ContextJsonContainsDedupKey says the anchored match is "safe because the key is lowercase SHA-256 hex (no JSON escaping, no collision with any other serialized field)", and the DuckDB/Postgres stores' LIKE "%\"DedupKey\":\"" + dedupKey + "\"%" construction relies on the same assumption ("the hex key has no LIKE wildcards").

This call passes finding.Subject as dedupKey — but finding.Subject is a raw Postgres database name or replication slot name (db.DatabaseName/slot.SlotName in PostgresAlertEvaluator.cs), not a SHA-256 hash. That breaks both assumptions the matcher relies on:

• JSON-escaping mismatch (false negative): AlertContextSerializer.Serialize uses JsonSerializer.Serialize with no custom encoder, so the default JavaScriptEncoder escapes non-ASCII characters and JSON-special characters (", \) as \uXXXX/\\/\". A database named e.g. café or one containing a quote gets serialized as "DedupKey":"café", but the seed lookup builds its pattern from the raw, unescaped finding.Subject (café), so the match silently fails. The restart-seed this PR is adding then returns null for that subject, and the exact bug being fixed here (re-firing on restart) resurfaces for any non-ASCII/quoted database or slot name.
• LIKE-wildcard mismatch (potential false positive, Postgres/DuckDB stores only): _ and % in a database name are literal in the JSON but wildcards in the LIKE pattern, so a name like orders_db could coincidentally match a differently-named subject on the same server/metric, seeding from the wrong row.

Since this is the first caller to feed a non-hex value into this hex-only-safe matcher, worth either hashing finding.Subject the same way AlertFingerprint does before using it as DedupKey/dedupKey, or giving this call path an exact-match lookup instead of reusing the hex-oriented one.

[claude]

Reviewed. The watermark-seeding shape faithfully mirrors AlertEngine.EnsureWatermarksSeededAsync/_seededServerKeys (including the "seed failure logs and proceeds unseeded" tradeoff, which is pre-existing/intentional, not a regression), and the IAlertHistoryStore.GetLastAlertTimeAsync signature change is applied consistently across every implementer (Lite's DuckDB store, Darling's Postgres store, the deprecated Dashboard JSON store, and all test fakes) — no Lite/Darling parity drift on the interface itself.

One correctness issue worth a look: left an inline comment on DarlingWorker.cs — the new Tier-0-predictor seed call passes finding.Subject (a raw Postgres database/slot name) as the dedupKey into GetLastAlertTimeAsync, but that filter's anchored-substring/LIKE matching was built and documented as safe only for SHA-256 hex fingerprints (no JSON escaping, no LIKE wildcards). A non-ASCII, quoted, or underscore-containing database/slot name can silently fail to seed (JSON-escaping mismatch) or, in the Postgres/DuckDB stores, spuriously match a different subject (_/% as LIKE wildcards).

No SQL injection, secrets, or missing-index concerns; this is a pure C#/EF-style change with no T-SQL touched.

[claude]

Review

Went through the full diff (C# only — no T-SQL/install changes here, so the T-SQL style rules don't apply). Traced the fix end-to-end against the pieces it claims parity with:

Correctness

• EvaluatePgDeadlocksAsync/EvaluatePgBlockingAsync seed-then-evaluate-then-persist-on-change ordering is a byte-for-byte match of AlertEngine.EnsureWatermarksSeededAsync/CheckDeadlocksAsync (in-memory update before the store write, so a failed persist never desyncs the two watermarks).
• The _pgDeadlockWatermarkSeeded/_pgBlockingWatermarkSeeded/_postgresAlertHistorySeeded TryAdd-guarded seed-once pattern is safe against the one race that would matter (two concurrent evaluations of the same server key) — ServerLoopState.InFlightSweep gating (DarlingWorker.cs:1447) already guarantees a server's sweep body never overlaps itself, so there's no window where a second caller reads the dictionary before the first caller's seed read completes.
• BuildDedupKeyLikePattern's three-character escape order (backslash first, then %/_) is correct and necessary — escaping backslash after would double the backslashes the wildcard-escaping step just inserted. Traced the quote/backslash/underscore/percent/non-ASCII cases in Lite.Tests/StoreRoundTripTests.cs by hand against the algorithm; they check out.
• ESCAPE '\' in both the DuckDB and Postgres queries produces a literal single backslash under each engine's default (non-C-style) string literal handling — consistent with the parameterized LIKE value.
• Confirmed finding.Subject (Wraparound/Xmin/Slot) really is what ends up as AlertIncident.DedupKey in the persisted ContextJson (record's first positional field, constructed directly as new(finding.Subject, ...)), so the new seed read's dedupKey: finding.Subject actually matches what was written.

Lite/Darling/Dashboard parity

• IAlertHistoryStore.GetLastAlertTimeAsync's new optional dedupKey param is implemented consistently across all three production stores (PgAlertHistoryStore, DuckDbAlertHistoryStore, the deprecated JsonAlertHistoryStore) plus every test fake — no stale 2-arg holdouts found via grep.
• No leftover hand-built "%\"DedupKey\":\"" + dedupKey + "\"%" patterns anywhere; every LIKE-based store (email/webhook/analysis cooldowns, not just the new caller) now routes through the shared AlertContextSerializer.BuildDedupKeyLikePattern, so the escaping fix isn't scoped narrowly to the new caller.

Security

• All dedup-key values reach the database as bound parameters ($3), never string-concatenated into SQL text — no injection surface despite finding.Subject now being an arbitrary raw database/slot name rather than a hash.

Performance

• Both new seed paths are TryAdd-guarded to one extra read per (server, metric[, subject]) for the life of the process, not per sweep — matches the stated intent and the existing AlertEngine cost profile.

Nothing else stood out. This is a tightly-scoped, well-tested fix with the parity work (Lite/Darling/Dashboard signature + escaping) done consistently rather than just patched at the new call site.