PostgreSQL deadlock reports, alerts and findings stored before #4005 are rewritten in place, so a direct SELECT finds no raw query or raw-graph hash (#4012)

[erikdarlingdata]

Closes #4012.

Builds on #4013 (the #4005 normalizer), which is already on dev.

Why

Since #4005 the deadlock collector stores each report's SQL normalized, and every product read normalizes a row stored before it. The rows themselves were never rewritten:

• collect.pg_deadlocks: pre-pg_deadlocks keeps each deadlock query and the victim statement verbatim, literals included, and returns them to readers and alerts #4005 rows keep victim_statement and graph_text raw, with a deadlock_hash over the raw graph (a test for the literals, raw_line_hash and statement_fingerprint are unkeyed hashes a store reader can use to guess the literals they hide #4004), for the 90-day retention.
• Alert history (config_alert_log): a deadlock alert fired before pg_deadlocks keeps each deadlock query and the victim statement verbatim, literals included, and returns them to readers and alerts #4005 keeps that raw hash and the raw victim statement in context_json. A per-event row also keeps them in its detail item and in detail_text.
• Analysis findings (analysis_findings): a finding written before pg_deadlocks keeps each deadlock query and the victim statement verbatim, literals included, and returns them to readers and alerts #4005 keeps its deadlock exemplars' victim statement, fingerprint and graph raw, with a hash that may be over the raw graph, in drill_down_json and its prose, for 30 days. PostgreSQL deadlock reports store and show their SQL normalized, and no read returns a hash of the raw report (#4005) #4013 normalizes these on read only. This is the same exposure in a third table, so this PR covers it too.

A role with a direct SELECT (#4004 names mcp's grant on every table in collect and config) reads all of them as stored. Erik ruled on the issue: rewrite them with #3915's proven pattern.

What changes

PgDeadlockRemask (new, in Darling.Service) runs one bounded slice per hourly store-maintenance tick. It uses the #3915 store-log remask shape and sits right after that pass in SweepStoreSelfMetricsAsync. It has its own 60 s slice cap and its own catch, so neither a failure nor a slow slice can cost the collector-cost flush.

• The alerts go first. An incident is found through its report's raw hash, and the report pass replaces that hash, so the report pass starts only after the alert pass reaches the end of the log.
• Reports. The pager walks (collection_time, server_id, deadlock_hash), the hypertable's partition order, 2,000 rows per slice.
  • Each raw report is rewritten with all its sightings. Its graph becomes NormalizeGraph, its victim statement becomes NormalizeStatement, and it takes pg_deadlocks keeps each deadlock query and the victim statement verbatim, literals included, and returns them to readers and alerts #4005's identity (IdentityOf over its timestamp and normalized graph).
  • That is exactly what every read already showed, so nothing a reader sees changes except the identity. The report's sightings now group with any the new collector stored of the same report.
  • RawGraphHashSql is part of the UPDATE, so only rows still raw are touched. The pass is safe to repeat and safe beside the collector.
• Alerts. An incident whose key names a raw report takes the incident the live alert builds for the rewritten report (BuildPgDeadlockIncident), key and object both. This applies in the context, in a per-event row's detail item, and in detail_text.
  • Every other member round-trips through AlertContextDto unchanged.
  • An incident whose report retention already dropped keeps its key, because nothing is left to test that hash against. Its statement is normalized as it stands.
  • Left alone: alerts built from normalized text (report current, or named by a ReportIdentity), SQL Server alerts (64-character lower-case fingerprints), and contexts that do not parse.
  • The UPDATE matches on the row's exact old content, so a row changed in between (a dismissal) is left for a re-read.
• Findings. These run between the alerts and the reports. The pass uses the read's own normalization (NormalizeStoredDeadlockExemplars, which now reports whether it rewrote the section and can stamp it) and stamps the section sql_normalized, so the read and any later pass leave it as it is.
  • Only a section without the stamp is read.
  • A row is written only while it still holds what the page read.
• Bounds. Each report, alert and finding write is its own transaction under SET LOCAL statement_timeout = '15s', with the client CommandTimeout 5 s above it.
  • A report whose rewrite fails is counted and left raw. It does not hold the rest of the table back.
  • A scan that left rows raw reads its table through again, at most 3 times per process.
• No store migration.
• The AlertReadFailureSurfaceTests census gets the new catch as its 29th stated exemption, the store-log remask's twin.
• The README's pg_deadlocks row says the older rows are rewritten.

Test plan

• PgDeadlockRemaskTests (new, 6 tests):

  • Pure: every slice statement carries the parser's raw-hash test.
  • Pure: a raw report becomes what every read shows, under the identity the collector gives the same captured report, and is idempotent.
  • Pure: a legacy alert, Summary and Per-event, becomes byte for byte what the live alert persists for the rewritten report, context and detail_text, and a second pass returns nothing.
  • Pure: only alerts carrying raw text change (report gone, report current, ReportIdentity, SQL Server key, bad JSON).
  • Pure: a pre-pg_deadlocks keeps each deadlock query and the victim statement verbatim, literals included, and returns them to readers and alerts #4005 finding becomes what every read showed and is stamped, so the read leaves it alone and a second pass returns nothing. A finding with no deadlock section is not touched.

• Live (own scratch store, TimescaleDB hypertable with every chunk compressed). The test plants pre-pg_deadlocks keeps each deadlock query and the victim statement verbatim, literals included, and returns them to readers and alerts #4005 reports: one seen twice across two chunks, one with no victim statement, and one seen raw and again by this build. It also plants a current report and five alerts (summary, per-event, fired-since, report-gone, SQL Server). After the pass:

  • No literal and no raw hash remains in any of the three tables.
  • get_pg_deadlocks/get_pg_deadlock_detail (the reader behind the MCP tools, web dispatch, viewer and alert) and the drill-down's own SQL read every report exactly as before, identity aside. The old at-... name still finds each one.
  • The report seen by both builds reads as one report, times_seen 2.
  • A pre-pg_deadlocks keeps each deadlock query and the victim statement verbatim, literals included, and returns them to readers and alerts #4005 finding reads as it did.
  • The current row, the fired-since alert, the SQL Server alert and a finding with no deadlock section are byte-identical.
  • A second pass rewrites 0 and leaves all three tables byte-identical.

• Red-watch 1: with the per-event detail-item rewrite removed, the pure alert test and the live test both fail. Restored.

• Red-watch 2: with the findings stamp removed, the pure findings test and the live test both fail. Restored.

• Slice cost, measured on PostgreSQL 18.6 + TimescaleDB 2.30.1 with 61 compressed daily chunks and 20,000 raw rows (2,000 reports x 10 sightings, 5 servers):

  Pass	Slice	Result
  Alerts	500 rows	0.70 s, 500 rewritten
  Reports	2,000 rows	first 1.36 s, max 1.88 s; 11 slices, 9.2 s in all, 20,000 rewritten, 0 failed, 0 raw left
  Reports, second pass	2,000 rows	43 ms, 0 rewritten

• Gates: AlertReadFailureSurfaceTests, TimescaleEnableConnectionSafetyTests, DocCommentHygiene, StartupCommandTimeoutTests, LivePostgresCollectionHygieneTests, LiveCleanupConversionRatchetTests, McpPayloadContractCensusTests, plus PgDeadlockNormalizationTests, StoreLogRemask* and PgTargetDeadlockDrillDown*: all green.

• Full Darling.Tests with DARLING_TEST_PG, final run: 13,020 run, 32 skipped, 2 failed.

  • Both failures are FleetIdentifierScrubTests, and they are environmental. That test skips every path containing \.claude\, and this lane's worktree sits under .claude\worktrees, so it scanned 0 files.
  • This PR's added lines were checked by hand against both of its patterns: no match.
  • The first full run also caught this PR's live test reusing the connection it handed to TryEnableAsync (TimescaleEnableConnectionSafetyTests). It now goes through LiveTimescaleProbe.

• Lite untouched (no shared project changed).

🤖 Generated with Claude Code

https://claude.ai/code/session_01GdmA4ND1wLSqA91ax1m4xv