Repository navigation
Both RDS log readers carry a standing denial on every pgmon target while rolling up HEALTHY, errors 0, "no new deadlocks/plans" #2994
Description
Found by the first full 50-target sweep of the pgmon store (2026-09-05). #2633 fixed this exact dishonesty for one path — a denied AWS call now records PERMISSIONS with a message that ends "Nothing was read this cycle — this is NOT 'no plans were captured'." The message is there. The rollup ignores it.
What the surface says today, fleet-wide (all 50 targets, none excepted)
pg_plan_capture: rolls up HEALTHY, errors 0, note no new auto_explain plans in the RDS log window on ~136 of 167 runs — and last_error on every one of the 50 targets is an IAM denial on the monitoring host's role:
- 49 (cluster-endpoint targets): not authorized to perform
rds:DescribeDBClusters - 1 (target-a, the only instance-endpoint target): not authorized to perform
rds:DescribeDBLogFiles
pg_deadlocks: rolls up HEALTHY, errors 0 on 48 of 50, note no new deadlocks in the RDS log window on ~1,590 of ~2,000 runs — and last_error on those 48 is permission denied for function pg_read_file (SQLSTATE 42501). (The other 2 carry a different defect that overwrites the denial — filed separately.)
get_fleet_overview on this store: total_deadlocks: 0.
Why this is a defect and not just missing grants
The denials are deterministic — the role has the action or it doesn't; the login can execute pg_read_file or it can't — so the read path fails the same way on every target. Yet no surface a reader actually consults says so:
errorsis 0. The denial is a classified non-fatal skip, which is right for the cycle and wrong for the rollup: a skip that means "I read nothing" is indistinguishable, in every counter, from a read that found nothing.- The reassuring note fires anyway on the majority of runs. Whatever sub-path emits "no new deadlocks/plans in the RDS log window" reports emptiness on targets where the log demonstrably cannot be opened.
- The fleet answer — zero deadlocks across 50 production Aurora clusters, in every window anyone has looked at — is exactly what a healthy quiet fleet would show and exactly what total blindness would show. The product's own conventions (pg_blocking's capture-count denominator, A per-database collector that fails in SOME databases reports SUCCESS with no note #2623's survivors note) exist to keep those distinguishable; the two log readers don't follow them.
What the MCP surface cannot establish, and the store query that settles it
Whether ANY cycle has ever read a log successfully. last_error proves denials happen; it cannot prove they happen every cycle. Settle it with per-status histograms and rows-ever:
SELECT collector_name, status, COUNT(*) AS n
FROM collect.collection_log
WHERE collector_name IN ('pg_deadlocks', 'pg_plan_capture')
GROUP BY collector_name, status;plus row counts in the deadlock/plan tables. If PERMISSIONS accounts for every attempted read, this fleet has been blind since enrollment and "0 deadlocks" has never been evidence of anything.
Fix, in two halves
Product half — the rollup must be able to show blindness:
- A collector whose most recent read-path outcome is a denial must not band HEALTHY with
errors 0. Either count classified skips distinctly (adeniedcounter beside errors/yields/abandoned) or band it NO_PERMISSIONS the way an all-runs-denied collector already does. - The "no new X in the RDS log window" note must be unreachable on a cycle that did not open the log — that note is a claim about the log's contents.
get_fleet_overview's deadlock total should carry the denominator when the underlying reader is denied: "0 (not collected on N servers)" is the honest number.
Ops half — the actual grants, recorded here so the issue closes on both:
- The monitoring host's IAM role needs
rds:DescribeDBClusters,rds:DescribeDBLogFiles,rds:DownloadDBLogFilePortionon the pgmon targets. pg_read_fileneeds an explicit grantdarling_monitordoes not have — or the deadlock reader should take the same RDS API path as plan capture on managed targets and stop attemptingpg_read_filethere at all. The hint text the denial currently carries ("pg_monitor covers every collector here") is wrong for this function: pg_monitor does not grantpg_read_file.
Related: #2633 (the single-path fix whose vocabulary this should reuse), #2661 (where the log reader came from), #2623 (the convention violated).
Activity
erikdarlingdata commentedon Sep 5, 2026
get_collector_costputs a number on what these denied readers cost, and it is not small. Seven days, 50 targets:pg_deadlockspg_plan_capturepg_deadlocksis the single most expensive collector on this store by wall time — 49.3M ms ≈ 13.7 hours per week, ~621 ms per run — and it has produced zero rows in the entire window. Withpg_plan_capturethat is ~15 hours a week, ~2.1 hours a day, spent on two collectors whose reads are denied on all 50 targets.The
sql_ms: 0is not a measurement gap. Both read the RDS log through the AWS API rather than by querying the target, so their work lands instorage_ms;pg_cpu_utilizationhas the same shape (sql_ms: 0, 14.7Mstorage_ms) and does return rows, which is the control that says the field is real work and not an artifact.Two things this sharpens in the issue above:
errors: 0, status HEALTHY, a reassuring note, and a cost that only shows up if you specifically askget_collector_cost. Nothing correlates the cost against the output.last_erroron 48 of 50, the balance is clear.Worth considering as a general guard rather than a one-off: a collector with a nonzero cost and a persistently zero row count is a cheap, engine-agnostic signal that something is running without collecting — it would have surfaced this weeks ago without anyone knowing to look.
Date the last_error slot, and correct this branch's own retracted evi…
erikdarlingdata commentedon Sep 5, 2026
Retracting this issue. Both halves are false, and the ops half asks for production IAM changes that are not needed — do not action it. Measured against the store rather than reasoned.
The evidence
Both readers cut over on 2026-08-30 and neither has produced a single
PERMISSIONSrow since — six days and ~14,270 successful deadlock reads a day, across all 50 targets.What that means for each claim
no new deadlocks in the RDS log windowis an honest empty, not blindness.rds:DescribeDBClusters/DescribeDBLogFiles/DownloadDBLogFilePortionon the host role, plus apg_read_filegrant) — unnecessary. No IAM denial of any kind is current on that store. The only ongoingPERMISSIONSrows are missing extensions (pg_wait_sampling,pg_buffer_usage,pg_predicate_stats,pg_kernel_stats,pg_index_bloat) and an Aurora-unsupported source (pg_write_stats/pg_stat_get_wal, 0A000), all correctly classified non-fatal skips.pg_deadlockstoday and be wrong, flagging an honest empty as a fault. That is the failuretarget_has_user_databasesexists to prevent elsewhere in this codebase. Adding a third term (a denial actually present in the window) makes it safe and also makes it redundant with reading the errors.How I got it wrong, because the cause is the one thing worth keeping
I read
last_erroras a current condition. It is a single retained slot with no timestamp, and it was holding a six-day-oldpermission denied for function pg_read_file (SQLSTATE 42501)from the in-database deadlock route — a code path the collector no longer takes.The two routes are mutually exclusive:
no new deadlocks in the RDS log windowis emitted only atDarlingCollectorRunner.cs:638(the RDS-API route), whilepg_read_filelives inPgDeadlocksCollector.cs:83(the in-database route). So on any run where that note fires,pg_read_filewas never called — the 42501 beside it can only be a fossil.Nothing in
get_collection_healthdistinguishes that from a live fault: no timestamp, no indication the emitting path is retired, sitting next to a HEALTHY status and tens of thousands of successful runs.That is the real defect, and it is filed separately. It is also what #2996 scoped out to this issue, so the ground is covered rather than lost.
Closing as not-a-defect. Credit for the route-exclusivity catch to the #2996 lane; the dilution explanation for the near-identical apparent denial rates (15.9% vs 14.3% — one cause, one window, diluted differently because
pg_deadlocksruns ~12x more often) came from the reviewer on #3002.