Skip to content

raw_line_hash and statement_fingerprint are unkeyed hashes a store reader can use to guess the literals they hide #4004

Closed

Description

@erikdarlingdata

Found by the round-1 security review of #3996 (#3944). This needs a design decision, so it is its own issue.

What is wrong

collect.pg_log_events.raw_line_hash is SHA-256 (first 16 bytes, hex) over the raw log entry, literals included (PgLogTextRedactor.RawLineHash). A reader who can see the stored row can rebuild almost all of that entry: the prose is stored as PostgreSQL wrote it (#3944), the SQL in it is normalized with ? where each literal stood, and the prefix is the target's own log_line_prefix. The one part the reader cannot see is the literal values. Because the hash is unkeyed, the reader can enumerate candidate values and test each one against the hash offline.

The review measured it: a 4-digit PIN in a failed UPDATE was recovered from the stored raw_line_hash in 6 ms.

statement_fingerprint has the same weakness for one statement shape. It hashes RedactStatement's output, and that function masks quoted literals and bare numbers but not dollar-quoted bodies. So a secret inside a $$...$$ literal or a DO block is hashed as written, and can be guessed the same way.

What #3996 already does

  • get_pg_log_events (MCP, and the web dispatch built on it) no longer returns raw_line_hash. The column stays, because every read dedupes on it.
  • The doc comments that said the hash "discloses nothing" are corrected: PgLogTextRedactor.RawLineHash, PgLogTextRedactor.Fingerprint and V129's remarks. No migration SQL was touched.

What is still exposed

  • Any role with SELECT on collect.pg_log_events can still read raw_line_hash. The mcp role gets GRANT SELECT ON ALL TABLES IN SCHEMA collect. Any surface that runs a reader's own SELECT over collect (custom views) reaches the column too.
  • statement_fingerprint is still returned by get_pg_log_events and shown in the web viewer's log tab.

Proposed fix (needs a decision)

  1. Key both hashes: use HMAC-SHA-256 with a per-install key that the service holds and the store does not (for example, beside the store credentials in darling.json, DPAPI-protected). A reader of the store then has nothing to test guesses against.
  2. Plan the dedupe continuity for raw_line_hash. Stored rows cannot be re-keyed, because the raw text is never stored. The reads dedupe inside the self-hosted tail's overlap window, so switching keys at upgrade stores each in-window entry once more, one time. Either accept that and say so, or dedupe on (old hash OR new hash) for one retention period (30 days).
  3. For statement_fingerprint, first check who joins on it: PgTargetDrillDownCollector.Deadlocks, PgTargetFactCollector.Blocking, PgTargetAdvice.Database and the web log tab. Then either key it the same way (with continuity handled for those joins) or mask dollar-quoted bodies in RedactStatement. The second option breaks fingerprint continuity with every stored row, which Ingest target PostgreSQL logs as telemetry — errors, spills, autovacuum runs, and connection churn are invisible between counter samples #3601 deliberately avoided.

Activity

  1. erikdarlingdata commented on Sep 23, 2026

    @erikdarlingdata
    OwnerAuthor

    Ruled (Erik, 2026-09-23): HMAC-SHA256 with a per-store key held outside the database. The service generates the key and keeps it in DPAPI on a managed Windows store, in an owner-only file in the darling-credentials volume on compose, and in an owner-only file in the service's config directory on bring-your-own. New rows use it. Old rows keep their unkeyed hash until retention drops them, which means a one-time dedupe discontinuity, as #4013 accepted. No tool or viewer returns any hash.

  2. added
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Sep 23, 2026
  3. erikdarlingdata commented on Sep 23, 2026

    @erikdarlingdata
    OwnerAuthor

    Closed by the watcher: delivered in PR #4020, merged to dev.

  4. removed
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions