PostgreSQL deadlock reports store and show their SQL normalized, and no read returns a hash of the raw report (#4005)

[erikdarlingdata]

Closes #4005.

Why

collect.pg_deadlocks stored each deadlock report's DETAIL block verbatim (tabs stripped) and the victim's statement verbatim. Every read returned them as stored: get_pg_deadlocks (victim statement), get_pg_deadlock_detail (graph), the web dispatch on both, the desktop viewer's grid, the deadlock alert (email, webhook {{incidents_json}}, Teams and Slack cards, the persisted alert context), and the analysis drill-down (pg_deadlock_exemplars), whose finding persists the statement, its fingerprint in the advice prose, and the graph for 30 days. That is the exposure #3944 ruled on, through a different table.

deadlock_hash was SHA-256 over the raw graph. Readers got it in get_pg_deadlocks, get_pg_deadlock_detail, the drill-down, and the alert's dedup key (rendered as "Dedup Key" on cards and persisted in the context JSON). get_pg_deadlock_detail also looked rows up by it. Once the graph is shown normalized, that hash is the #4004 guessing oracle: rebuild the graph, try values for each ?, and compare, offline or by lookup.

The block pattern also found ERROR: deadlock detected and DETAIL: anywhere on their lines, so a logged statement carrying those words was stored as a deadlock. Measured: the old pattern matches a LOG: statement: SELECT 'x ERROR: deadlock detected line followed by a forged DETAIL, as a report from pid 1600.

What changes

Write: one reader, one lexer. PgDeadlockLogParser no longer parses a report with its own pattern.

• The block pattern (C# on the RDS route, SQL on the pg_read_file route) now only finds candidate text. It returns the ERROR line, the DETAIL block, and the line after it, which is where DeadLockReport always writes its HINT.
• FromReport reads each candidate with PgLogEntryAssembler, the reader Log messages show as PostgreSQL wrote them, the SQL in them stays normalized, and one statement's text never decides how another's is read (#3944) #3996 hardened. The label has to be the line's own, the DETAIL has to come from the same backend, and each continuation line loses exactly the tab PostgreSQL added. A query's own tabs stay, so the lexer reads the tokens PostgreSQL ran. The HINT sets DetailComplete.
• PgLogTextRedactor.RedactDetail(detail, DetailComplete) normalizes the queries before anything is read out of the block. It is Log messages show as PostgreSQL wrote them, the SQL in them stays normalized, and one statement's text never decides how another's is read (#3944) #3996's own code (per-waiter heads, fail-closed without the HINT proof), not a second masker. The wait-for lines stay as written.
• The victim statement is taken from the normalized graph.
• deadlock_hash is IdentityOf = SHA-256 over (timestamp, normalized graph). It covers only what a reader sees. The timestamp is there because normalizing can make two different reports' graphs match. It also means a new hash can never equal the old HashOf(graph), which is how a pre-pg_deadlocks keeps each deadlock query and the victim statement verbatim, literals included, and returns them to readers and alerts #4005 row is told apart.

Read: every path normalizes, idempotently, and no raw hash leaves.

• DarlingPgDeadlockReader (MCP, web dispatch, WPF viewer, deadlock alert) normalizes the victim statement and graph of every row. A row this build wrote comes back unchanged.
• A row stored before pg_deadlocks keeps each deadlock query and the victim statement verbatim, literals included, and returns them to readers and alerts #4005 is recognised in SQL (PgDeadlockLogParser.RawGraphHashSql: its hash equals SHA-256 of its graph). Its hash is never returned; it is named at-<occurred_at>-<victim_pid> instead, from values every read already shows.
• get_pg_deadlock_detail finds a report by either kind of identity. A hash lookup never matches a row whose hash is over its raw graph, so a guessed raw hash finds nothing.
• The drill-down (PgTargetDrillDownCollector.Deadlocks) now reads as much text as normalizing needs, normalizes it, then cuts to its 2000/4000-character caps in memory. Cutting first could land inside a literal, and the lexer would then withhold the whole statement. The exemplar's identity follows the same rule as the reader.
• Stored findings: PgFindingStore.ReadFinding normalizes a pg_deadlock_exemplars section written before this change. It replaces the fingerprint in the stored advice prose and names each exemplar's report by time and pid. Every stored-finding read (get_analysis_findings, the viewers) gets that. A section this build writes carries sql_normalized: true and is left as written, because a statement already normalized and cut to its cap can end inside a '?'.

Found while here, fixed in lane. Without an identity, get_pg_deadlock_detail returned the reports whose hashes sort first, not the newest. Its LIMIT sat on the DISTINCT ON sort, which leads with the hash. It now sorts newest first in an outer query.

Wording. Tool descriptions, the detail note, the web panel note, the README collector row and the runbook now say the SQL is normalized.

No store migration. No parity item: Lite and the deprecated Dashboard do not monitor PostgreSQL.

Test plan

• Lite.Tests PgDeadlockLogParserTests (54): three new regression tests.
  • Literals in both queries, with the exact normalized graph and victim statement, idempotence, and the identity over the normalized graph.
  • A query cut mid-literal is withheld. With the HINT the next query is read from its own head; without it, it is withheld too.
  • A statement literal holding a report is not read as one, and a real report right after it still is.
  • Existing tests updated deliberately: victim attribution is pinned by pid now that both statements normalize alike; the collector seam now carries one column.
• Darling.Tests PgDeadlockNormalizationTests (6, live PG):
  • A pre-fix row reads normalized under at-..., and its raw hash is neither returned nor finds it.
  • A new row reads back unchanged under its own hash.
  • The detail read without an identity returns the newest reports.
  • Notification body: the incident built from the read has a normalized statement and no raw hash, checked in both the context JSON and {{incidents_json}}.
  • Every read's inline raw-hash SQL matches the parser's.
• PgTargetDeadlockDrillDownTests (9, live):
  • The e2e plants a pre-fix exemplar row with a literal and a raw hash; the drill-down carries neither.
  • A pre-fix stored finding read back through get_analysis_findings is normalized.
  • A unit test pins the stored-finding normalization and its idempotence.
• RdsDeadlockIngestorTests, PgDeadlockLogTimezoneTests: identity and zone pins updated deliberately (the zone is now read out of the returned report text).
• Proved against the old behaviour by mutation (each reverted after):
  • Normalization off: 9 tests fail (4 Darling, 5 Lite).
  • Raw hash returned and looked up: 3 fail.
  • The stored-finding hook removed: the e2e fails.
• The pg_read_file pattern on PostgreSQL 18.6, over a 4.78 MB synthetic tail with 8 reports: old 20-26 ms, new 20 ms, same 8 matches. The HINT line is taken, and a report with no HINT does not take the next report's first line.
• Full Darling.Tests and Lite.Tests: see below.
• 0 warnings on both builds.

Full runs, on the rig (PostgreSQL 18.6 + TimescaleDB 2.30.1), DARLING_TEST_PG set:

• Darling.Tests: 13011 total, 3 failed, 31 skipped. All three are accounted for:
  • PgLogEventsPipelineTests.TheTailerExtraction_LeftBothSiblingsSqlByteIdentical: the deadlock sibling's SQL pin. Updated deliberately, with the reason in the test.
  • LivePostgresCollectionHygieneTests: the new class had not joined the live-postgres collection yet. Fixed.
  • TrendPayloadBudgetLiveTests: get_file_io_trend hit a transient Exception while reading from stream under full-suite load. This class is untouched here; it passes alone (1/1).
• After those fixes, every touched or affected class re-run green: PgLogEventsPipelineTests 38, LivePostgresCollectionHygieneTests 11, PgTargetDeadlockDrillDownTests 9, PgDeadlockNormalizationTests 6, RdsDeadlockIngestorTests 11, PgDeadlockLogTimezoneTests 2, DocCommentHygiene 77, McpPayloadContractCensusTests 68, DarlingPgOperationalAlertTests 30.
• Lite.Tests: 5171 total, 1 failed: PgLoggingCollectorGateTests.AnOrdinaryDeadlockRowStillParses, the collector seam's old four-column shape, since updated. The affected classes re-ran green (73).

For the reviewer

• Dedupe continuity, one time. On the pg_read_file route, a report still inside the 4 MB tail at upgrade is stored again under its new identity. It shows as two rows (one at-..., one hash) until it ages out of the window. The alert's dedup key for such an in-window report changes once.
• Residue, filed as pg_deadlocks rows stored before #4005 keep their raw SQL and raw hash for the rest of their retention, readable by a direct SELECT #4012. Rows stored before this still hold raw text and raw hashes in collect.pg_deadlocks until the 90-day retention. Every product read normalizes them, but a role with a direct SELECT on the table (raw_line_hash and statement_fingerprint are unkeyed hashes a store reader can use to guess the literals they hide #4004 names the mcp grant) reads them as stored. Closing that needs a one-time rewrite of those rows, which is out of scope under no-migration.
• Pre-fix rows lost their tabs. The old parser stripped every tab, a query's own included. On read, a value the stripping glued to a word (LIMIT\t5 stored as LIMIT5) lexes as an identifier and is not masked. New rows keep the query's own tabs.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GdmA4ND1wLSqA91ax1m4xv