Repository navigation
Fix Poison Wait alert re-firing on an unrefreshed wait_stats row - #2704
Conversation
CheckPoisonWaitsAsync gated re-fire purely on wall-clock cooldown, with no way to know whether the underlying wait_stats delta had actually changed since the last fire. When the collector's delivered cadence lagged the cooldown, the read adapter's "newest row in the last 10 minutes" query handed back the SAME row on consecutive sweeps, producing byte-identical duplicate alerts (observed live: two RESOURCE_SEMAPHORE_QUERY_COMPILE alerts 6.5 minutes apart with identical avg/delta/task values). PoisonWaitDelta now carries the row's own collection_time, threaded through both read adapters (Darling/Postgres and Lite/DuckDB). AlertEngine tracks the collection_time it last fired on per server and requires it to be newer than that, in addition to cooldown elapsed, before re-firing. Unlike CPU (legitimately level-triggered — resampled live every sweep), a wait_stats delta is one collector cycle's computation; reading it twice is the same event surfacing twice, not a fresh observation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Reviewed. This is a clean, well-scoped fix. Correctness: The gating logic is sound. Parity: Tests: Minor, pre-existing, not introduced by this PR: No correctness, parity, security, or performance concerns that block this. |
Never deduping RootBackendId == 0 rows (the fix for the earlier collision finding) traded one bug for another: the SAME persisting vanished-root block, sampled every sweep, added a new list entry every cycle, so RollingCountAlertGate's watermark kept climbing and "Blocking Detected" would re-fire every cooldown for one ongoing incident - exactly the #1091/#2704/#2708 class this design is supposed to be immune to. Fixed: sentinel rows now dedupe by RootPid instead of never deduping, narrowing the risk to pid reuse inside one rolling 1-hour window rather than either merging unrelated incidents (the original bug) or guaranteed re-alerting on a persisting one (this regression). RootPid was already the identity BuildPgBlockingIncident folds into that case's DedupKey, so this makes the count and the fingerprint agree on what identifies a sentinel incident. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ctor_cost row (erikdarlingdata#2707) DarlingSelfAlertEvaluator.ApplyCostRegressionsAsync gated re-firing on cooldown-elapsed alone, with no memory of which collect.collector_cost data point it last alerted on. If the hourly store-metrics tick's cadence ever outpaces the cooldown, or the flush itself lags a tick, re-asking GetCostRegressionsAsync hands back the identical latest_ms computed from the identical underlying rows - observed today as byte-identical repeat alerts on harvest-01/plan_correction and multi-24/ag_replica_states. Same shape as erikdarlingdata#2703/erikdarlingdata#2704 (Poison Wait). Added LatestMetricTime to CostRegression - the newest collect.collector_cost row actually folded into latest_ms - and require it to have advanced past what was last alerted on before re-firing, in addition to cooldown-elapsed. Verified the SQL change against a real local Postgres 17: latest_metric_time stays put with no new hourly row, advances only once one lands. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Postgres analogue of SQL Server's Poison Wait alert, watching the IPC:BtreePage / IPC:BufferIo pair identified by the issue's fleet research: near-zero on a healthy server, a structural inter-backend contention signal no other alert covers when it accumulates. Fires on total accumulated wait time normalized to a 10-minute window (Warning at an average of one backend continuously stuck, Critical at ten), per event, because an avg-ms-per-wait bar can never see 1-2 ms waits at six-figure volumes. Follows the engine-gated sibling path (PostgresAlertEvaluator + DarlingPostgresAlertReadAdapter, delivered from DarlingWorker), the shared PoisonWaitEnabled switch (erikdarlingdata#2722's precedent), per-subject cooldown keying (erikdarlingdata#1140), the unrefreshed-source-row guard (erikdarlingdata#2704), and history-seeded restart survival (erikdarlingdata#2716/erikdarlingdata#2723). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JgLaaxFenZFZs1SntKQDEg
Fixes #2703.
Problem
`AlertEngine.CheckPoisonWaitsAsync` gates re-firing purely on wall-clock cooldown (`CooldownElapsed`). `IAlertReadAdapter.GetPoisonWaitDeltasAsync` always asks "what's the newest poison-wait `wait_stats` row within the last 10 minutes" (`ORDER BY collection_time DESC LIMIT 3`) — independent of whether that row is NEW since the last ask. When the collector's delivered cadence lags the alert cooldown (documented fleet behavior at load), the same still-unrefreshed row answers the question twice, and the engine fires again on data it already reported. Observed live: byte-identical `RESOURCE_SEMAPHORE_QUERY_COMPILE` alerts on the same server 6.5 minutes apart, same avg/delta/task values to the decimal.
This is unlike CPU's alert, which is correctly level-triggered (resampled live every sweep — an identical reading really is a fresh observation of a standing condition). A wait_stats delta is one collector cycle's computation; reading it twice is the same event, not two observations.
Fix
Test plan
Darling.Testscan't run on macOS (net10.0-windows) — relying on CI's `Darling PostgreSQL tests`/`build` jobs to actually execute it.🤖 Generated with Claude Code