Log masking fails closed: a DETAIL or CONTEXT is read as SQL only where PostgreSQL writes SQL, a value cut before its close is masked to the end, and get_store_query_stats masks a raw text pg_stat_statements kept (#3920 follow-up)

[erikdarlingdata]

Follow-up to #3920 (merged as bbf3e73). A fourth security review of #3920's last round found no Critical or High issues, five Medium and three Low. Every Medium and two of the Lows are fixed here; the remaining design questions are in #3944.

CHANGELOG: this PR carries no CHANGELOG edit. Its entry is in the batched splice.

Why

Threat model: a viewer or mcp role user, or an LLM agent calling get_store_log, get_pg_log_events or get_store_query_stats, reads literal values out of other sessions' SQL. The store log keeps them 400 days, pg_log_events 30.

Each finding below was reproduced as a failing test before it was fixed: all 7 new tests failed on dev.

What changes

M1: SQL is read only where PostgreSQL writes SQL (a regression from round 3).

• Round 3 split a DETAIL at any line shaped like Process N: or Failed process was running:, even inside a value. A literal holding numbered lines ("Process 1: call Jane Roe at jane@...") kept its middle as identifiers. A key tuple or failing row cut at such a line lost the close its prose shape needs, and kept the SSN.
• RedactDetail now reads SQL only in three shapes:
  • A deadlock report, whose wait-for lines come first. A Process N: line starts the next query only when N is one of the deadlock's own processes AND the query before it reads to its end.
  • A crash's Failed process was running:, one query to the end of the field.
  • A logged EXECUTE's prepare: (L3), one query to the end of the field.
• Every other DETAIL is prose, read whole.
• In RedactContext, an SQL frame is read as SQL only in three places, and never while an earlier value's quote is open:
  • the field's first line;
  • after a closed SQL frame;
  • after a one-line frame (PL/pgSQL function ..., while ... in relation ..., JSON data, line N: ...).
• Other frames (a COPY row) are values that can run onto the next line. Their lines are masked as one prose block, so a COPY value holding SQL statement "..." is its own content, not SQL.

M2: a non-ASCII non-letter no longer keeps a value (a regression from round 3).

• To PostgreSQL, =<NBSP>4111... is one identifier, and full-width digits and zero-width marks behave the same. All three are how a value pasted from a web page or typed through an IME reads.
• The lexer masks such a token. A non-ASCII letter (café, 名前) is still a name.
• The prose allowlist keeps a quoted run after a noun only when it reads as a name: closed, with a letter in it, and no non-letter non-ASCII character. column "4111..." is masked now.

M3: masking a hostile field is bounded.

• Round 3 re-lexed a CONTEXT frame at every candidate close: 90 KB took 6-8 s, 180 KB took 21 s, and the self-hosted tail re-reads its overlap window every cycle.
• Now, a field over 64 KB is withheld, and a frame or split tests at most 32 candidate boundaries.
• A pre-existing ReDoS the new test surfaced: the exclusion-pair key-tuple pattern re-tried every longer key against every value, taking 7.2 s on 27 KB of Key (a)=( and 3.7 minutes on 90 KB. Its key is now an atomic group: 19 ms. The key tuple and partition key got the same change.
• Every prose pattern now carries a 500 ms match timeout, and a text that runs one out comes back as <text withheld: it could not be masked in time>.

M4: prose value shapes fail closed (predates #3920).

• Each shape needed its closing character. A value cut before it (a read boundary, a newline inside it, M1's splits) kept its opening half.
• : "…" / at or near "…" now closes only at a quote that ends the text, or that the log's at character N follows. Before, json: "{"card": 4111 closed at the key's quote and kept the number.
• The key tuple closes only before one of PostgreSQL's own sentences (already exists, is duplicated, is still referenced, is not present, conflicts with) or the end. It matches [Kk]ey, so the exclusion violation's second tuple is covered too.
• The failing row and the partition key close only at a ) that ends the text.
• Every shape otherwise masks to the end, and an unclosed double-quoted run is masked too.

M5: get_store_query_stats masks what it shows (predates #3920).

• Confirmed in PostgreSQL 18's pg_stat_statements.c: when a statement's entry is gone at executor end, pgss_store runs with no jumble state and stores norm_query ? norm_query : query, the raw text. That happens when the entry was evicted during a long execution, or reset between a protocol Parse and its Execute.
• Reproduced live: Parse, then pg_stat_statements_reset(), then Execute stores SELECT 'hunter2-m5b'::text AS pw, $1::int AS n. That passes the reader's allowlist.
• The tool now passes every text except its two sentinels through the SQL lexer. A normalized text comes through as it was, including PostgreSQL 18's IN-list marker, which the lexer now keeps. A raw text is masked, or withheld when it cannot be read to its end.
• Residual: a direct SQL caller of config.store_statement_stats(), the viewer or mcp role with its own credentials, still reads the raw text in that race. It already reads every collected table.

L1: LOCATION. Its own line is kept as written only in PostgreSQL's shape (function, file.c:N), and is masked as prose otherwise. The head rule for foreign lines is in #3944.

L3: prepare: is a query head (see M1).

Not fixed here, needs a decision: #3944. Prose keeps values PostgreSQL didn't quote: RAISE text (P0001 / class P0), libxml DETAIL source lines, a typed value after a noun, and a foreign stderr line's head. Withholding RAISE text on monitored targets trades the application's own error messages for privacy.

Test plan

• M1: ADetailIsSqlOnlyInTheShapesPostgresWritesSqlInto and AContextFrameIsSqlOnlyWhereAFrameCanStart.
  • The reviewer's inputs: a numbered line inside a deadlock literal, including one naming a real PID; a failing row and a key tuple holding head-shaped lines; a COPY value holding SQL statement "..." or a fake PL/pgSQL frame.
  • prepare: and crash heads.
  • Nested and JSON frames still read as SQL, and every shape is idempotent.
• M2: ANonAsciiSpaceOrDigit_NeverKeepsAValue: NBSP, full-width digits and zero-width mark in statements; column "<NBSP>4111" and column "4111" in prose. Non-ASCII letters are kept.
• M3: MaskingAHostileField_StaysLinear, each under 2 s:
  • the reviewer's 90 KB and 180 KB CONTEXT shapes, and a 63 KB one under the cap;
  • a 60 KB and a 300 KB deadlock DETAIL;
  • 90-180 KB of Key (, Key (a)=(, partition keys, a JSON value and column ".
• M4: AValueShapeWithoutItsClose_IsMaskedToTheEnd (10 shapes, each idempotent), and through the store log's own capture in AnUnclosedValueAndAForeignLocation_AreMaskedWithTheEntry.
• M5:
  • Live, in OnTheProductsOwnConf_...: a prepared Npgsql command, pg_stat_statements_reset(), then Execute. The view holds RawLiteral3920 verbatim; the tool's full text and preview do not, and show '?'::text AS pgss_raw_marker.
  • Unit test: TheToolMasksEveryTextItShows_AndLeavesNormalizedTextAsItWas.
• L1: a PostgreSQL-shaped LOCATION is kept; a foreign one is masked.
• Full suites. Full Darling.Tests ran on a pgss-preloaded rig with DARLING_TEST_PG and DARLING_TEST_PGRUNTIME; full Lite.Tests too (numbers in the thread).

🤖 Generated with Claude Code

https://claude.ai/code/session_018Z4T51J4E6L8fjriPHpsif