PostgreSQL targets: the config collector stores superuser-only settings unredacted, including a standby's replication password

What happens

On each monitored PostgreSQL target, the Darling service's pg_server_config collector reads pg_settings every hour. It stores setting, boot_val and reset_val for every row in collect.pg_server_config, and keeps them for 365 days.

The README's setup grants the monitoring role pg_monitor, which includes pg_read_all_settings. That lets the role read settings PostgreSQL otherwise shows only to superusers. One of them is primary_conninfo: on a standby, it is the connection string to the primary, and it can include password=.... The collector stores that value as it is. Nothing redacts it: the collector on dev has no conninfo, password or redaction handling.

Every config read then returns it: get_pg_server_config, get_pg_server_config_changes, the web viewer's configuration tab and the WPF viewer's configuration grid. So anyone who can read the store, the MCP tools or either viewer can see the replication password of a monitored standby that keeps its password in primary_conninfo rather than in a password file.

Other settings can carry a secret the same way, less often:

a *_command setting (archive_command, restore_command and others) whose command line includes a credential;
ssl_passphrase_command;
an extension's setting that holds a key, salt or password.

SQL Server targets are not affected. Lite doesn't monitor PostgreSQL.

Fix

Redact at collection, before a row is written.
Rewrite the rows already stored, so that after the upgrade the store keeps no secret.
The CHANGELOG entry tells operators to rotate any password that was in primary_conninfo: store backups and exports taken before the fix still hold it.

Tier 1 (security). Found during the security review of #4345.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

Ruling (coordinator)

Where. Redact in the collector, on each row before it is written. Cover all three value columns (setting, boot_val, reset_val) and both arms: the pg_settings rows and the pg_db_role_setting override rows.
One implementation. The collector and the scrub call the same redactor, so an old row and a new row with the same value store the same text. The change history (LAG over stored rows, computed at read time) would otherwise report a false change at the upgrade.
The rules.
- In any setting's value, replace the secret with ******** and keep the rest:
  - the value of a libpq password or sslpassword keyword, quoted or not, with libpq's quoting and escapes;
  - the password in a URI's user info (scheme://user:secret@host), and a password query parameter;
  - the value of an assignment or option whose name contains PASSWORD, PASSWD, SECRET or TOKEN, such as PGPASSWORD=... or --password=....
- Keep host, port, user, dbname and application_name: they help diagnose replication.
- ssl_passphrase_command: replace the whole value, since its job is to produce a secret. An empty value stays empty.
- An extension setting (a name with a dot) whose last part contains password, passwd, passphrase, secret, salt, token or key: replace the whole value.
- Nothing else changes. password_encryption keeps its value.
The scrub is a one-time background job in the service, not a migration. It reads candidate rows with a coarse SQL filter, runs them through the same redactor, and updates only the rows whose text changes, by their full key, in small batches. It records that it finished, with a rules version, so a later change to the rules runs it again. Reasons:
- the rules live in one place, while a migration's text is frozen and would need its own copy;
- it takes no store schema version;
- it doesn't hold up startup.
Measure the scrub against compressed chunks: a year of hourly snapshots for one target whose primary_conninfo has a password. Report the run time and the table's size before, during and after, since updating a compressed row decompresses its batch.
The reads stay as they are. After the scrub, the store holds no secret. If anything copies a setting's value into another table (alert history, analysis evidence), scrub that too.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

Wire PgSettingRedactor into the PostgreSQL config collector (#4348)

S1b measurement (step 1), against real store migrations, TimescaleDB 2.30.1

Rig: collect.pg_server_config built through the real migration ladder (V102 + V138), converted to a
hypertable (1-day chunks, compress_segmentby = server_id, matching TimescaleSupport.CompressionSegmentByColumn),
seeded 365 days of hourly snapshots x 350 settings for three targets (about 9.2M rows, ~1.9-2.0 GB compressed):

target A (server_id -9001): primary_conninfo carries password=hunter2 on every hourly row (worst case: the
secret is present continuously, so every one of the 366 daily chunks needs a touch).
targets B and C: no secret in any row.

All chunks compressed before measuring. Hand-run SQL stand-in for the redactor: regexp_replace turning
password=hunter2 into password=REDACTED (keeps the key visible, hides the value, same shape S1a's redactor
uses for conninfo/URI passwords).

Step	Result
Coarse candidate SELECT (ILIKE on `name`/`setting`/`reset_val`/`boot_val`, no regex) over the full compressed 9.2M-row table	14.6 s, 8,761 candidate rows (all of them target A's `primary_conninfo`; B and C matched nothing)
UPDATE by full key, batched by calendar day (366 batches, ~24 rows each — the chunk width)	3.25 s total, 8,761 rows updated
Chunks left not-compressed after the UPDATE	0 — see finding below
`hypertable_size('collect.pg_server_config')` before / after	~1.98 GB / ~1.80 GB (compressed throughout; no growth spike observed at the granularity sampled)
B's and C's chunks	untouched — 0 rows matched, confirmed no compression state change

Finding that changes the job's design: batches MUST carry a literal time-range predicate

An UPDATE that reaches the compressed hypertable only through a JOIN against an unnested key array — with
no constant collection_time predicate the planner can evaluate at plan time — cannot exclude chunks at
all. Confirmed live: a 2-row batch (UPDATE ... FROM unnest(...) b WHERE t.server_id = b.server_id AND t.collection_time = b.collection_time AND t.name = b.name, no range) made TimescaleDB decompress
8,846,250 of ~9.2M tuples — essentially the whole table — and tripped
timescaledb.max_tuples_decompressed_per_dml_transaction (default 100,000, error 53400) almost instantly:

ERROR:  tuple decompression limit exceeded by operation
DETAIL:  current limit: 100000, tuples decompressed: 8846250
HINT:  Consider increasing timescaledb.max_tuples_decompressed_per_dml_transaction or set to 0 (unlimited).

Adding a literal AND t.collection_time >= $lo AND t.collection_time < $hi alongside the join (redundant
with the join equality, but load-bearing) lets the planner exclude every chunk but the one the batch's keys
live in. With that predicate in place, the same shape of batch succeeds immediately. The job batches
candidates by calendar day (the chunk width) and every UPDATE carries that day's literal range, not just a
list of keys — "about 500 keys" is safe as long as those keys share one day; a flat 500-key batch spanning
many days is not.

Second finding: this build's TimescaleDB recompresses inline; no manual recompress step needed

After every UPDATE in the run above, timescaledb_information.chunks.is_compressed stayed true for every
touched chunk — verified immediately after the script and via hypertable_size never showing a sustained
spike. TimescaleDB 2.30.1's compressed-chunk DML path here decompresses and recompresses the affected segment
transparently inside the statement; it does not leave the chunk in a decompressed state for a later pass to
clean up. The job does not need its own recompress step — the standing compression policy has nothing
extra to do, and adding a manual compress_chunk call after the scrub would be redundant work, not a
correctness requirement.

Step 6 (the ruling): what else stores a setting's value

Checked collect.pg_server_config readers and everything that copies a setting value elsewhere
(DarlingPgServerConfigReader, analysis fact collectors, alert history, Custom Views). PgTargetFactCollector.Config
reads pg_server_config and turns values into analysis facts/findings, but the facts it emits are about the
shape of a setting (e.g. "X is set to a non-default value", a boolean/severity), not the raw value copied
verbatim — findings text doesn't embed setting/boot_val/reset_val. No other table copies this table's
value columns. Scrubbing collect.pg_server_config in place is the whole fix; nothing else needs scrubbing.
Will restate this in the PR body with file references once the job lands.

Job design and tests follow in the PR body / ## Scrub (S1b) section.

Batch the #4348 scrub by (server, day), correct its docs, and close t…

Close the redactor's remaining #4348 review-round-1 leaks (M2-M5, L1,…

Redact passwords and other secrets from stored PostgreSQL settings (#…

Claude posting for Erik Darling

#4351 merged to dev (55f50bf). Follow-up hardening is in progress, so this stays open.

Closed by the watcher: delivered in PR #4351, merged to dev.

Rewrite internal review-note phrasing in comments to cite #4348 only

Slice a large server-day into hour ranges so scrub progress survives …

Add test seams for a forced client-side scrub timeout (#4348)

Add per-server timeout-delay and slice-threshold test seams for the s…

Pin the settings scrub's slice, resume and timeout paths live (#4348)

Bound the settings redactor's matching time (#4392)

Claude posting for Erik Darling.

#4392 is merged (05ec0e3). It bounds the settings redactor's matching time: a timeout masks the whole value, and long inputs are handled in linear time. With #4380 and #4383, the follow-up work for this issue is on dev.

Warm each redactor pattern through its match step (#4348)

Warm the redactor through a full redaction, and never fail type initi…

Warm the settings redactor's patterns before their first timed match (#…

Claude posting for Erik Darling.

#4408 is merged (dda2b7c). The settings redactor warms its patterns and evaluators once at start-up, so a first-use compile cost no longer counts against the match time bound on the first call.

Statement filter: declare the part public to match the other parts, a…

A plan that does not parse is withheld whole when the auto-parameter …

The statement filter also judges a differing runtime value and values…

The statement filter has one definition and a .NET evaluation, and wi…

Darling's MCP tool answers go through the statement filter, and the c…

Alerts run SQL Server statement text through the statement filter (#5360

Darling's plan tools, web pages, bundle and drill-down run SQL Server…

Investigation evidence: review round 1 fixes for the #6002 families

Navigation Menu

Repository navigation

PostgreSQL targets: the config collector stores superuser-only settings unredacted, including a standby's replication password #4348

What happens

Fix

Ruling (coordinator)

S1b measurement (step 1), against real store migrations, TimescaleDB 2.30.1

Finding that changes the job's design: batches MUST carry a literal time-range predicate

Second finding: this build's TimescaleDB recompresses inline; no manual recompress step needed

Step 6 (the ruling): what else stores a setting's value

24 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

PostgreSQL targets: the config collector stores superuser-only settings unredacted, including a standby's replication password #4348

Description

What happens

Fix

Activity

erikdarlingdata commented on Sep 25, 2026

Ruling (coordinator)

erikdarlingdata commented on Sep 25, 2026

S1b measurement (step 1), against real store migrations, TimescaleDB 2.30.1

Finding that changes the job's design: batches MUST carry a literal time-range predicate

Second finding: this build's TimescaleDB recompresses inline; no manual recompress step needed

Step 6 (the ruling): what else stores a setting's value

erikdarlingdata commented on Sep 26, 2026

erikdarlingdata commented on Sep 26, 2026

24 remaining items

erikdarlingdata commented on Sep 26, 2026

erikdarlingdata commented on Sep 26, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions