Deadlock re-mask keys a missed alert instead of leaving its raw hash, walks pages to its hourly budget, and a failing stage no longer blocks the rest (#4012)

[erikdarlingdata]

Closes #4012.
Closes #4035.

Follow-up to #4022 (already merged to dev): the fixes for its round-1 security review. The rulings are the coordinator's and are applied as given.

Why

The round-1 review of #4022 found:

1. High. An alert the pass missed kept its raw-graph hash permanently. Once the report stage rewrote a report, an alert still carrying that report's raw hash resolved as Missing on every later pass and kept the key, which sat for 90 days beside the normalized graph, occurred_at and victim: the offline oracle raw_line_hash and statement_fingerprint are unkeyed hashes a store reader can use to guess the literals they hide #4004 removed. Alerts got missed in two ways. A dismissal UPDATE could move an unvisited row behind the ctid cursor, and that wasn't counted as raced. The rescan cap also marked a stage done anyway.
2. Medium. One page per hourly tick: about 11 days for the reports alone at the review's 518k rows. A restart started from zero.
3. Medium. Findings ran before reports and gated them. The findings page was a Seq Scan plus a Sort over every detoasted drill_down_json, and a stage error retried hourly with no cap.
4. Medium. The README and code comment claimed more than the pass does: pg_log_events rows from before Log-event hashes are keyed with a per-store secret the service keeps outside the store, so a store reader can no longer test guesses at the literals they hide (#4004) #4020 still hold an unkeyed raw_line_hash.
5. Low. Old and new identities don't always match (tabs; a cut middle query).
6. Low. Raw text went back to the store as bind parameters, which lands in the store's own log under log_min_duration_statement.

What changes

1. A missed alert is keyed, never keeps its raw hash (PgLogHashKey, PgDeadlockRemask.RemaskAlert)

• PgLogHashKey derives a third subkey under a new domain label, darling-pg-deadlock-alert-key-v1, using Log-event hashes are keyed with a per-store secret the service keeps outside the store, so a store reader can no longer test guesses at the literals they hide (#4004) #4020's existing derivation. It is not the raw_line_hash subkey. The new method is DeadlockAlertKey(key) and returns hk- plus 32 hex characters.
• The hk- tag is never a report hash's shape, so a key that has been keyed once isn't resolved again.
• An alert key that can't be resolved becomes that HMAC, in the context, the detail item and detail_text. Two alerts that shared a key still share one, and there's no oracle.
• The key is consumed only through PgLogHashKey's public API. The worker reads it from runner.LogHashKey, which is a new internal getter on the runner, at the store-metrics tick. DarlingLogHashKeyFile.cs, DarlingManagedRoles.cs and the key load site are untouched (lane R).
• With no key, the re-mask doesn't run, and says so once.
• Design correction the live test caught. Keying Missing alerts during the alert walk is wrong. On the confirmation walk, an alert already rewritten to a raw report's NEW identity finds no report by it (the report is still raw), so it would be keyed and cut from its report. So the keying happens in a separate stage, alert keys, that runs only once the report stage is done. The order is alerts, reports, alert keys, findings. Until then an unresolvable key is left as it is, and only its statement is normalized.

Done only after a clean walk

• A stage is done only when a whole walk rewrote nothing and left nothing.
• A walk that rewrote rows is followed by another, which catches a row a dismissal moved behind the cursor.
• A walk that rewrote nothing but left rows (a report that fails every time) counts toward MaxRescansPerProcess.

2. Pages to the budget; the cursor survives a cancel (PgDeadlockRemask.RunAsync)

• Each tick reads page after page until its 60 s SliceBudget.
• Each page function reports every finished row through a callback. On a budget cancel, each walk's cursor therefore stands at the last finished row, and the next tick resumes there.
• RunAsync ends quietly on its own cancel.
• Per the ruling there is no store migration: the state is in memory (RemaskProgress), like The store's statement statistics survive their review: ALTER SYSTEM advice that bricked the store, a reader filter any grantee could switch off, secrets in slow_statement, a probe that failed for non-superuser owners, and 20 more (#3899) #3915's store-log cursor. The measured rescan cost is in the test plan.

3. Order, a findings filter, and a failure cap

• The order is alerts, reports, alert keys, findings, so findings can't block reports.
• The findings page reads only rows whose root_fact_key is PG_DEADLOCK_RATE or ANOMALY_PG_DEADLOCK_RATE, the only roots the deadlock exemplar section is attached to. That is a plain column, so no other finding's drill-down is detoasted. I filtered on root_fact_key rather than category because the section is attached by root fact, which is the exact discriminator.
• Consecutive stage failures are capped at MaxConsecutiveStageFailures = 4. Each failure is logged. The stage then skips 0, 1 and 3 ticks, and on the fourth failure stops for the process.
• Meanwhile the stages after it run: a failing alert stage no longer starves the reports.
• The alert-key stage waits for the report stage to be fully done, not merely backing off.

4. Narrowed claim (README pg_deadlocks row, PgDeadlockRemask class doc)

• The deadlock reports, alerts and findings are re-masked.
• The pg_log_events deadlock ERROR rows from before Log-event hashes are keyed with a per-store secret the service keeps outside the store, so a store reader can no longer test guesses at the literals they hide (#4004) #4020 keep their unkeyed raw_line_hash until the table's 30-day retention drops them. That is raw_line_hash and statement_fingerprint are unkeyed hashes a store reader can use to guess the literals they hide #4004's ruling, since re-keying them would split their dedupe identity.

5. Identity match: nothing more is reconstructible (comment in RemaskReport, pinned by a test)

• Tabs. The collector before pg_deadlocks keeps each deadlock query and the victim statement verbatim, literals included, and returns them to readers and alerts #4005 removed every tab from the DETAIL (93787e8c~1's FromEntry), including the query's own. So LIMIT<tab>50 is stored as LIMIT50, one identifier that keeps its digits, as every read already shows it. The live read gives LIMIT ?. Where the tab stood isn't in the stored text.
• Wholeness. The live collector passes DetailComplete. The old collector stored a report cut at a chunk boundary and kept no proof it was whole. The rewrite keeps complete = false, which fails closed: claiming complete for a cut report is Log messages show as PostgreSQL wrote them, the SQL in them stays normalized, and one statement's text never decides how another's is read (#3944) #3996's out-of-step read, where a literal is kept as a word.
• A report with neither issue gets the live collector's identity exactly. That is pinned.

6. No raw text as a parameter

• The alert and finding UPDATEs check the row by ctid plus xmin (an Npgsql Xid parameter).
• The report UPDATE drops the raw victim_statement comparison. The raw-hash predicate already proves the row still holds the graph the page read, and the victim statement is that graph's own victim query.
• Raw-graph HASHES travelled as parameters (the report cursor and key, and the alert lookup array), filed as Deadlock re-mask still sends raw-graph hashes as bind parameters, which log_min_duration_statement writes into the store's own log #4035; fixed in round 2 below.

Test plan

• PgDeadlockRemaskTests + PgLogHashKeyTests: 22/22 (live on PG 18.6 + TimescaleDB 2.30.1).
• New pure tests:
  • the HMAC rewrite of an unresolvable key: summary and per-event, detail_text included, stable, store-specific, not the raw-line subkey, not a report-hash shape;
  • a Missing key is left alone before the reports are done;
  • no update statement carries raw text;
  • the findings page is filtered by root;
  • the tab and cut-query fixtures.
• New live tests:
  • a second page of reports (2,150 rows) and of alerts (521);
  • the alert order: the named alert takes its report's new identity, not an hk- key;
  • a raw alert behind the cursor of a walk that rewrote rows, caught by the next walk (3 walks);
  • a restart with fresh state: one walk per stage, tables byte-identical;
  • a failing alert stage (42P01): reports and findings done on the first tick, failures by tick [2,2,3,3,3,3,4,4], capped, 8 warnings (alerts plus alert keys);
  • a budget cancel mid-page keeps the cursor at the 10th finished row, and the resumed page rewrites the other 30 and re-reads at most the 10th;
  • the driver canceled before it starts moves no cursor.
• Revert proof for the High: with the Missing arm reverted to keeping the raw key, 3 tests fail: OnlyAnAlertCarryingRawText_IsRewritten, RunAsync_WalksEveryPage_InOrder_AndIsDoneOnlyAfterACleanWalk and RawReportsAndAlerts_AreRewrittenOnce_.... Restored, all pass.
• Measured on the rig: PG 18.6 + TimescaleDB 2.30.1, 120,000 rows (20,000 reports x 6 sightings over 85 days of chunks), every chunk compressed.
  • Rescan after a restart (every row already rewritten): one clean walk of all 120,000 rows (60 pages) in 2.7 s, inside one tick.
  • Time per tick while raw rows remain: the 60 s budget, by design. One tick rewrote 32,969 rows (5,495 reports, about 10.9 ms per report including page reads), and the cursor was kept at the cancel.
  • Whole table: about 3.6 ticks of rewrites plus a 3 s confirming walk, so about 4 hourly ticks. At one page per tick this table would have taken 60 ticks.
  • At the review's 518k rows: about 16 ticks at 6 sightings per report, and at worst (every row its own report, about 5.5k rows per tick) about 95 ticks, against about 259 before.
• Full Darling suite, once, after merging origin/dev, on a fresh darlingtest: 13,040 total, 2 failed, 33 skipped. Both failures were mine, and both are fixed:
  • CollectorStateContractTests.EveryDeclaredStateKeyPrefixHasAPruneVerdict. My constant was named DeadlockAlertKeyPrefix, and that census treats every public *KeyPrefix const as a collector-state key prefix. It is an HMAC value's tag, not a state key, so it is renamed DeadlockAlertKeyTag; the census itself is not weakened.
  • RefreshCeilingStalenessTests.TheSpacingFinding_IsCalledPerLightRefresh_OnTheSameWatch. My sed edits left the working copy of DarlingWorker.cs with LF endings, and the test searches for CRLF. A checkout under .gitattributes gives CRLF, as CI's does. After re-checkout it passes.
  • After the fixes: those two classes plus the remask, key, AlertReadFailureSurfaceTests and StoreToastAndCheckpointerTests classes, 143/143. AlertReadFailureSurfaceTests' exemption key is updated to the worker catch's renamed message, with the same reason plus the stage cap.
• Build: 0 warnings, 0 errors.
• Lite parity: not applicable. Lite has no PostgreSQL deadlock store, alert history or findings of this shape.

For the coordinator to double-check

• The added alert-key stage (alerts, reports, alert keys, findings) is the one place this departs from the brief's literal three-stage order. The reason is in item 1. It keeps the ruled behavior: a missed alert is keyed, never kept raw.
• The findings filter is on root_fact_key, not category (item 3).
• With no log-hash key the re-mask doesn't run at all, and warns once. Keying requires the key, and running the report stage without it would recreate the High.
• runner.LogHashKey is read at the store-metrics tick, deliberately away from lane R's load site.
• Issues filed: Deadlock re-mask still sends raw-graph hashes as bind parameters, which log_min_duration_statement writes into the store's own log #4035 (raw-graph hashes as bind parameters), now fixed here in round 2.

Round 2: the six findings of review round 1

All six are fixed on this branch (commit 4fdaebd and the hygiene commit after it). Closes #4035 too.

1. HIGH: no log-hash key skipped the whole re-mask. RunAsync takes a nullable key. The alerts, reports, findings and finding-alerts stages always run. Only the alert-key stage waits for a key: it logs one warning and stays pending, never done and never given up. The worker hands its key over as it stands, with no gate of its own (_pgDeadlockRemaskNoKeyWarned is gone; the once-only flag lives in RemaskProgress).

2. MEDIUM: outcomes on a partly finished page were lost. Every slice reports each row's outcome (Unchanged, Rewritten, Left) through its per-row callback, and the stage counts it there. A walk is the clean one only when it counted no rewrite, left nothing, and no cancel or failure cut it short (WalkInterrupted). The hand-set WalkRewritten = 10 is gone from the test.

3. MEDIUM: finding alerts were never re-masked. A new last stage, finding alert, pages config_alert_log rows whose metric starts with Analysis: and whose context carries a Pg Deadlock Exemplars item flattened before #4005 (no Sql Normalized field and no withheld marker). The same statement finds the finding by server, category and the 8-character story-hash prefix the metric name carries: the newest one analysed within a day before the alert. Then:

• The finding is stored and every non-SQL field matches: the section's fields are rebuilt from the finding's normalized section by the live alert's own flattening (FindingMessageFormatter.DrillDownItem, now shared) and stamped Sql Normalized. The victim the advice prose names is replaced by the normalized fingerprint.
• Anything else: Exemplars and Note are withheld as (withheld: stored before #4005 with its SQL raw), and so is the victim in the prose. This fails closed.

Writes are guarded by ctid and xmin, and the stage follows the same clean-walk rule. The advice item's body (the frozen story prose) also named the raw fingerprint. The review did not list it, and it is covered here.

4. LOW: exemplars off the root. The findings page now filters on string_to_array(story_path, ' → ') && ARRAY[PG_DEADLOCK_RATE, its anomaly]. That test sits inside a CASE, so no other finding's drill-down is detoasted.

5. LOW: a stage that gave up stayed given up for the life of the process. A given-up stage is reset and walked again from its table's start once RetryGivenUpAfter (1 day) has passed.

6. LOW (#4035): raw hashes sent as bind parameters.

• The ruled ctid plus xmin cannot work for pg_deadlocks. On a compressed chunk, SELECT ctid, xmin fails with transparent decompression only supports tableoid system column (rig, TimescaleDB). So the report walk pages by (collection_time, server_id) with FETCH FIRST n ROWS WITH TIES. A page never splits a batch, and the cursor moves only when a batch is finished. Alerts, findings and finding alerts stay on ctid plus xmin.
• The report UPDATE takes the raw hash server-side. A scalar subquery finds the row the page read by server, collection_time, occurred_at, victim_pid and the raw test. If two raw reports share that place, the statement fails with 21000 and the report is counted Left, instead of guessing.
• The UPDATE keeps occurred_at = $3 AND victim_pid = $4 as plain equalities, on purpose. On the rig, an UPDATE keyed only by the subquery hit tuple decompression limit exceeded: TimescaleDB filters compressed batches only by constant or parameter quals. With the equalities, EXPLAIN ANALYZE showed 1 of 201 batches decompressed. A report with a NULL timestamp or pid is rewritten in its own batch (ReportUpdateOwnBatchSql).
• The alert page resolves its incidents' reports in the same statement. It pulls report-shaped DedupKeys out of context_json with regexp_matches. There is no JSON parse, so a malformed context cannot fail the page.

Measured (rig, TimescaleDB, 120,000 rows over 84 compressed chunks, 5 servers, every row its own raw report, the worst case)

• Raw: one 60-second tick rewrote 5,876 reports, about 10 ms each, so the whole table takes about 20 hourly ticks.
• Clean: a walk of the whole table with nothing to rewrite took 1.7 s, inside one tick.
• Report UPDATE on a compressed chunk: 1 of 201 batches decompressed.

Tests (all in PgDeadlockRemaskTests; every class in the file run)

• New:
  • NoStatementBindsARawReportHash (Lite overview: show Online/Offline status #6)
  • TheWorkerRunsTheRemaskWithOrWithoutAKey (Add check for updates feature #1, source pin)
  • WithoutAKey_EveryStageButTheAlertKeysRuns (Add check for updates feature #1)
  • ACancelMidPage_ThenACompletedWalk_NeverLeavesTheStageDoneOverARawRow (Lite overview: show Online/Offline status for servers #2)
  • AFindingAlert_IsRewrittenFromItsFinding_OrWithheld (Add check for updates feature #3)
  • AFindingWhoseDeadlockFactIsOffTheRoot_IsRewrittenToo (Add possibility to exclude WAITFOR() from Top Queries view #4)
• Extended: AFailingStage_... (Lite: collector health status bar shows global errors instead of per-server #5, the daily retry) and ABudgetCancel_... (the new cursor, no hand-set count).
• Each fix was reverted once, and its test failed:
  • Add check for updates feature #1: RunAsync throwing on a null key, plus the worker's gate, fails WithoutAKey_... and TheWorkerRuns....
  • Lite overview: show Online/Offline status for servers #2: page-level counting with no interrupted flag fails ACancelMidPage_... (expected 0 raw rows, found 1, with the stage done).
  • Add check for updates feature #3: fails AFindingAlert_....
  • Add possibility to exclude WAITFOR() from Top Queries view #4: the root filter fails AFindingWhoseDeadlockFactIsOffTheRoot_... and the SQL pin.
  • Lite: collector health status bar shows global errors instead of per-server #5: fails AFailingStage_....
  • Lite overview: show Online/Offline status #6: a static pin. The previous commit's SQL holds all three shapes it forbids.
• Builds: Darling.Tests and Lite.Tests both build with 0 warnings.
• Full suites: Darling 13056 total, 3 failed on the first full run: two hygiene pins my change tripped (DocCommentHygieneTests.NoMemberCarriesTwoStackedSummaryBlocks, TsqlConventionGuardTests.TheMemberScan_ReadsEveryDeclarationWhole), fixed in 2c83912 and re-run green with PgDeadlockRemaskTests, AlertReadFailureSurfaceTests, StoreLogSelfMonitoringStoreTests and StoreToastAndCheckpointerTests (215 tests, only the third below failing); and TrendPayloadBudgetLiveTests.EveryDefaultAnswer_StaysNearTheBudget_AndTheLargestAnswerStaysUnderTheCap, a client-side timeout on get_pg_io_trend over 72h that is this rig's, not the change's: origin/dev's own build fails it identically on a freshly created database (45.6 s, 'Exception while reading from stream') and both builds pass it on a warmed one (10.4 s dev, 10.6 s branch); dev CI passes it. Lite 5186 total, 0 failed.

For the coordinator to double-check

• Lite overview: show Online/Offline status #6 departs from the ruling. It said to page and update by ctid plus xmin, which a compressed chunk refuses (the error is quoted above). The replacement is a batch cursor plus a server-side anchor.
• Lite overview: show Online/Offline status for servers #2 is literal. Any cancel or failure during a walk marks it interrupted, so a stage whose clean walk needs more than one tick's budget can never finish. At the measured 1.7 s per 120k rows, that starts somewhere past 4M rows. Such a stage gives up after MaxRescansPerProcess walks and retries a day later.
• Add check for updates feature #3 matches an alert to its finding by metric identity plus the newest finding within a day before the alert, and checks the non-SQL fields. Any mismatch is withheld.
• Notifications: InternalsVisibleTo PerformanceMonitor.Darling.Service was added, so the stage shares DrillDownItem/DrillDownHeading. BuildContext now calls DrillDownItem, with no change in behaviour.
• CHANGELOG: the buffer entry 4036-reported.txt is rewritten for the new behaviour.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GdmA4ND1wLSqA91ax1m4xv

Round-2 fixes

Lane V's fix pass for review round 2, the last round. Commits 1642059..c81507c, plus a merge of dev (head 6576320). Every fix has a regression test, and each test was proven once by reverting its fix.

1 (MEDIUM): the day-later retry now runs. The worker gated on RemaskProgress.Done, which a stage that gave up also satisfies. Once every stage had finished or given up, RunAsync was never called again, so the daily reset never ran. The worker now gates on RemaskProgress.Pending (Stages.Any(s => !s.Done || s.GaveUp)). While no retry is due, every stage is skipped without a statement.

• AFailingStage_BacksOff_IsCapped_AndDoesNotStarveTheOthers now ticks through TickAsTheWorkerAsync, the worker's own gate.
• TheWorkerRunsTheRemaskWithOrWithoutAKey pins the worker to if (_pgDeadlockRemask.Pending).
• Red: with Pending => !Done the test fails.

2 (MEDIUM): an alert can no longer be cut from its report.

• (a) The alert-key stage waits for Reports.Done && !Reports.GaveUp. Test: TheAlertKeys_WaitForAReportStageThatGaveUp. Red on the old gate: the alert-key stage walked and keyed the rebuilt identity.
• (b) The reports wait for Alerts.Done (a clean walk or a give-up), never for an alert stage that is only waiting out a failure. Test: AnAlertStageFailure_DoesNotLetTheReportsRunAheadOfIt. Red on the old gate. The failing-stage test now shows the reports starting on the tick the alert stage gives up.
• (c) The marker. An incident rebuilt from its raw report gets "RemaskRebuilt4012":true appended to its incident object. Why this marker:
  • It is its own JSON property, not part of the key. It can't be taken for a raw hash (32 upper-case hex) or a keyed one (hk-...), and no key's value changes.
  • Dedupe is unchanged. The rebuilt key is still exactly the key the live alert gives the rewritten report. The cooldown seed's anchored "DedupKey":"..." LIKE match still finds it, because the marker follows the incident's other members; this is pinned with ContextJsonContainsDedupKey.
  • Every reader ignores it. AlertIncidentDto doesn't map it, and nothing in the product sets UnmappedMemberHandling (grepped), so System.Text.Json skips it. That means no store migration, and no change to a shared DTO (PerformanceMonitor.Notifications, which Lite also builds).
  • No walk resolves or keys a marked incident. Test: ARebuiltIncident_IsMarked_AndNoWalkKeysIt. Red with the marker check removed.
  • PostgreSQL deadlock reports, alerts and findings stored before #4005 are rewritten in place, so a direct SELECT finds no raw query or raw-graph hash (#4012) #4022 transition. Dev's PostgreSQL deadlock reports, alerts and findings stored before #4005 are rewritten in place, so a direct SELECT finds no raw query or raw-graph hash (#4012) #4022 rebuilt incidents without a marker, and a dev store's reports may still be raw. So while reports may be raw (the alert stage, reportsMayBeRaw: true), an unmarked key that finds no report is keyed only in a row no pass has rewritten: one with an incident that still resolves to a raw report, or that still carries a statement to normalize. A row PostgreSQL deadlock reports, alerts and findings stored before #4005 are rewritten in place, so a direct SELECT finds no raw query or raw-graph hash (#4012) #4022 rewrote has neither, so it isn't written at all, and it waits for the alert-key stage. Test: AnUnmarkedRebuiltRow_WaitsForTheAlertKeys. Red without the rule.

3 (LOW, #4035): with a key, no raw hash is bound as a value. The alert stage now receives the key. It keys an unmarked unresolvable incident in the same write that normalizes its row.

• NoRemaskWriteBindsARawReportHashAsAValue installs BEFORE UPDATE triggers on config_alert_log and pg_deadlocks. They record every value the re-mask's UPDATEs write, which are exactly the values bound to SET ... = $n. The test finds no raw hash and no literal among them. Red when the alert stage gets no key.
• Residual exposure, stated plainly: with NO log-hash key, a row the alert stage must rewrite (to rebuild an incident or normalize a statement) still binds an unresolvable incident's raw hash back in $1/$2, because there is nothing to key it with. I did not build an in-SQL rewrite for this case (a per-row placeholder-restore UPDATE). The service already logs the no-key state at start-up, and the hash stays in the row itself until a key exists either way.

4 (LOW): walks page by alert_time or finding_id, never by ctid.

• config_alert_log (the alert, alert-key and finding-alert walks) pages by alert_time, FETCH FIRST n ROWS WITH TIES. analysis_findings pages by finding_id WITH TIES. ctid+xmin remain only in the writes.
• Both tables are plain tables, never hypertables (see TimescaleSupport's scope comment), so ctid/xmin always exist on them. pg_deadlocks, which is compressed, was already paged by collection_time and is unchanged.
• Each row's callback moves the cursor only when its tie group closes, as the report walk already did.
• Test: AnAlertMovedBehindTheWalk_IsStillReached. CLUSTER, run after the walk's first row, moves a raw alert to (0,1), behind a full page. The test runs without a key, so the alert-key stage can't rescue the row. Red on 478b8fa (the commit before the paging change): Walks was 1, meaning the walk declared itself clean with the row still raw.

5 (LOW): a finding alert is rewritten only from the run it was sent for. The page also returns the finding's severity, confidence and time_range_start/time_range_end. The alert's Diagnosis item must match them:

• Severity and Confidence as F2. A decimal comma is accepted, since the service formatted them in its own culture.
• Window as the two u timestamps, or no Window when the finding has no range.
• Otherwise the SQL is withheld.
• Test: AFindingAlert_WhoseStoryRanAgain_IsWithheld_NotRewrittenFromTheOtherRun, with a different severity, a different window, and a matching control. Red with the check disabled.

6 (LOW): the finding-alert page skips rows older than #3721.

• The page filters alert_time >= '2026-09-18'. The same test sits inside the CASE that guards strpos, so an older row's context is never detoasted.
• PostgreSQL deadlock cards carry the captured exemplars, grouped by shape and ranked by recurrence, so an operator learns the access pattern and not just the count (#3691) #3721's first commit is 2026-09-19 11:11 UTC (merged 11:28 UTC). The bound is a day earlier because alert_time carries no zone, so it holds for stores recording local time from UTC-12 to UTC+14.
• The date is formatted with the invariant culture.

Measured on the rig (PostgreSQL 18, TimescaleDB 2.30.1), in the probe database:

• Finding-alert page: 243,200 alert rows over 90 days, 295 MB, including 43,200 Analysis: rows with ~6 KB contexts.
  • Before (ctid, no bound): 293-466 ms, 154k buffers (120k hit, 34k read).
  • After: 71-102 ms, 12.5k buffers.
  • Both plans are a Parallel Seq Scan of the heap, because no index leads with alert_time. The saving is the TOAST reads of old rows.
• Report UPDATE (ReportUpdateSql shape): a pg_deadlocks hypertable of 518,400 raw reports, 40 servers, 90 daily chunks, 87 of them compressed with segmentby server_id.
  • 18.7 ms. Batches scanned 86, filtered after decompression 85, decompressed 1, tuples decompressed 144.
  • TimescaleDB prunes by server_id (segmentby) and collection_time min/max, then filters that server's later batches in memory by occurred_at/victim_pid. It reads every later batch of that server, so the cost grows with one server's rows after the report, not with the table.
• TimescaleDB version: the product declares no floor. TimescaleSupport.cs says "Nothing in the product declares a TimescaleDB floor"; the stated target is 2.x. The plan shape depends on the version, and I measured only 2.30.1. The in-memory "filtered after decompression" step is a newer DML path. An older 2.x decompresses every batch the metadata can't exclude, bounded by timescaledb.max_tuples_decompressed_per_dml_transaction where that setting exists. Before 2.11 an UPDATE on a compressed chunk is refused; the report stage counts that report as left, gives up after its rescans, and retries daily.

Round 2's clean items, re-confirmed:

• Without a key, only the alert-key stage waits: WithoutAKey_EveryStageButTheAlertKeysRuns passes. The reports now wait for the alerts, but that is ordering and needs no key.
• Per-row counting and a cancel mid-page still leave no raw row behind a stage marked done: ACancelMidPage_ThenACompletedWalk... and ABudgetCancel... pass, and the new cursors move only at a closed tie group.
• Report paging (WITH TIES, the 21000 guard, the victim statement taken from the graph) is untouched.

Tests:

• PgDeadlockRemaskTests (the file's only test class) 24/24 on the rig, live.
• Full Darling suite once at the end, after merging dev, on a freshly created darlingtest: Total 13094, Failed 0, Errors 0, Skipped 36 (345 s).
• Build: 0 warnings.
• Lite: every file touched is under Darling/, and no shared project changed.

For the coordinator to double-check:

• The marker (an unmapped JSON property on the incident) and the PostgreSQL deadlock reports, alerts and findings stored before #4005 are rewritten in place, so a direct SELECT finds no raw query or raw-graph hash (#4012) #4022-transition rule (reportsMayBeRaw). Read the first loop in RemaskAlert.
• The no-key residual under finding 3.
• FindingAlertSectionSince = 2026-09-18.
• The changelog buffer entry 4036-reported.txt is updated: the retry and bound-value claims are now true, the no-bind claim is scoped to "with a log-hash key", and the two new outcomes are added.

[erikdarlingdata]

Round 2: the six findings of review round 1

All six are fixed on this branch (commit 4fdaebd and the hygiene commit after it). Closes #4035 too.

1. HIGH: no log-hash key skipped the whole re-mask. RunAsync takes a nullable key. The alerts, reports, findings and finding-alerts stages always run. Only the alert-key stage waits for a key: it logs one warning and stays pending, never done and never given up. The worker hands its key over as it stands, with no gate of its own (_pgDeadlockRemaskNoKeyWarned is gone; the once-only flag lives in RemaskProgress).

2. MEDIUM: outcomes on a partly finished page were lost. Every slice reports each row's outcome (Unchanged, Rewritten, Left) through its per-row callback, and the stage counts it there. A walk is the clean one only when it counted no rewrite, left nothing, and no cancel or failure cut it short (WalkInterrupted). The hand-set WalkRewritten = 10 is gone from the test.

3. MEDIUM: finding alerts were never re-masked. A new last stage, finding alert, pages config_alert_log rows whose metric starts with Analysis: and whose context carries a Pg Deadlock Exemplars item flattened before #4005 (no Sql Normalized field and no withheld marker). The same statement finds the finding by server, category and the 8-character story-hash prefix the metric name carries: the newest one analysed within a day before the alert. Then:

• The finding is stored and every non-SQL field matches: the section's fields are rebuilt from the finding's normalized section by the live alert's own flattening (FindingMessageFormatter.DrillDownItem, now shared) and stamped Sql Normalized. The victim the advice prose names is replaced by the normalized fingerprint.
• Anything else: Exemplars and Note are withheld as (withheld: stored before #4005 with its SQL raw), and so is the victim in the prose. This fails closed.

Writes are guarded by ctid and xmin, and the stage follows the same clean-walk rule. The advice item's body (the frozen story prose) also named the raw fingerprint. The review did not list it, and it is covered here.

4. LOW: exemplars off the root. The findings page now filters on string_to_array(story_path, ' → ') && ARRAY[PG_DEADLOCK_RATE, its anomaly]. That test sits inside a CASE, so no other finding's drill-down is detoasted.

5. LOW: a stage that gave up stayed given up for the life of the process. A given-up stage is reset and walked again from its table's start once RetryGivenUpAfter (1 day) has passed.

6. LOW (#4035): raw hashes sent as bind parameters.

• The ruled ctid plus xmin cannot work for pg_deadlocks. On a compressed chunk, SELECT ctid, xmin fails with transparent decompression only supports tableoid system column (rig, TimescaleDB). So the report walk pages by (collection_time, server_id) with FETCH FIRST n ROWS WITH TIES. A page never splits a batch, and the cursor moves only when a batch is finished. Alerts, findings and finding alerts stay on ctid plus xmin.
• The report UPDATE takes the raw hash server-side. A scalar subquery finds the row the page read by server, collection_time, occurred_at, victim_pid and the raw test. If two raw reports share that place, the statement fails with 21000 and the report is counted Left, instead of guessing.
• The UPDATE keeps occurred_at = $3 AND victim_pid = $4 as plain equalities, on purpose. On the rig, an UPDATE keyed only by the subquery hit tuple decompression limit exceeded: TimescaleDB filters compressed batches only by constant or parameter quals. With the equalities, EXPLAIN ANALYZE showed 1 of 201 batches decompressed. A report with a NULL timestamp or pid is rewritten in its own batch (ReportUpdateOwnBatchSql).
• The alert page resolves its incidents' reports in the same statement. It pulls report-shaped DedupKeys out of context_json with regexp_matches. There is no JSON parse, so a malformed context cannot fail the page.

Measured (rig, TimescaleDB, 120,000 rows over 84 compressed chunks, 5 servers, every row its own raw report, the worst case)

• Raw: one 60-second tick rewrote 5,876 reports, about 10 ms each, so the whole table takes about 20 hourly ticks.
• Clean: a walk of the whole table with nothing to rewrite took 1.7 s, inside one tick.
• Report UPDATE on a compressed chunk: 1 of 201 batches decompressed.

Tests (all in PgDeadlockRemaskTests; every class in the file run)

• New:
  • NoStatementBindsARawReportHash (Lite overview: show Online/Offline status #6)
  • TheWorkerRunsTheRemaskWithOrWithoutAKey (Add check for updates feature #1, source pin)
  • WithoutAKey_EveryStageButTheAlertKeysRuns (Add check for updates feature #1)
  • ACancelMidPage_ThenACompletedWalk_NeverLeavesTheStageDoneOverARawRow (Lite overview: show Online/Offline status for servers #2)
  • AFindingAlert_IsRewrittenFromItsFinding_OrWithheld (Add check for updates feature #3)
  • AFindingWhoseDeadlockFactIsOffTheRoot_IsRewrittenToo (Add possibility to exclude WAITFOR() from Top Queries view #4)
• Extended: AFailingStage_... (Lite: collector health status bar shows global errors instead of per-server #5, the daily retry) and ABudgetCancel_... (the new cursor, no hand-set count).
• Each fix was reverted once, and its test failed:
  • Add check for updates feature #1: RunAsync throwing on a null key, plus the worker's gate, fails WithoutAKey_... and TheWorkerRuns....
  • Lite overview: show Online/Offline status for servers #2: page-level counting with no interrupted flag fails ACancelMidPage_... (expected 0 raw rows, found 1, with the stage done).
  • Add check for updates feature #3: fails AFindingAlert_....
  • Add possibility to exclude WAITFOR() from Top Queries view #4: the root filter fails AFindingWhoseDeadlockFactIsOffTheRoot_... and the SQL pin.
  • Lite: collector health status bar shows global errors instead of per-server #5: fails AFailingStage_....
  • Lite overview: show Online/Offline status #6: a static pin. The previous commit's SQL holds all three shapes it forbids.
• Builds: Darling.Tests and Lite.Tests both build with 0 warnings.
• Full suites: Darling 13056 total, 3 failed on the first full run: two hygiene pins my change tripped (DocCommentHygieneTests.NoMemberCarriesTwoStackedSummaryBlocks, TsqlConventionGuardTests.TheMemberScan_ReadsEveryDeclarationWhole), fixed in 2c83912 and re-run green with PgDeadlockRemaskTests, AlertReadFailureSurfaceTests, StoreLogSelfMonitoringStoreTests and StoreToastAndCheckpointerTests (215 tests, only the third below failing); and TrendPayloadBudgetLiveTests.EveryDefaultAnswer_StaysNearTheBudget_AndTheLargestAnswerStaysUnderTheCap, a client-side timeout on get_pg_io_trend over 72h that is this rig's, not the change's: origin/dev's own build fails it identically on a freshly created database (45.6 s, 'Exception while reading from stream') and both builds pass it on a warmed one (10.4 s dev, 10.6 s branch); dev CI passes it. Lite 5186 total, 0 failed.

For the coordinator to double-check

• Lite overview: show Online/Offline status #6 departs from the ruling. It said to page and update by ctid plus xmin, which a compressed chunk refuses (the error is quoted above). The replacement is a batch cursor plus a server-side anchor.
• Lite overview: show Online/Offline status for servers #2 is literal. Any cancel or failure during a walk marks it interrupted, so a stage whose clean walk needs more than one tick's budget can never finish. At the measured 1.7 s per 120k rows, that starts somewhere past 4M rows. Such a stage gives up after MaxRescansPerProcess walks and retries a day later.
• Add check for updates feature #3 matches an alert to its finding by metric identity plus the newest finding within a day before the alert, and checks the non-SQL fields. Any mismatch is withheld.
• Notifications: InternalsVisibleTo PerformanceMonitor.Darling.Service was added, so the stage shares DrillDownItem/DrillDownHeading. BuildContext now calls DrillDownItem, with no change in behaviour.
• CHANGELOG: the buffer entry 4036-reported.txt is rewritten for the new behaviour.