Postgres High CPU alert isn't a light port — there's no instance-level CPU collector to alert on

Summary

Part of #2711 (Postgres alert-type parity). #2711's own table classified High CPU as "needs light design" — this issue corrects that: it's not a design question, it's a missing data source. There is currently no collector anywhere in this codebase that captures instance-level CPU utilization for a Postgres/Aurora target. Building a real "High CPU"/"CPU Resolved" pair requires new collector infrastructure, not just an alert-wiring change alongside EvaluatePostgresAlertsAsync.

What SQL Server's High CPU actually reads

AlertEngine.CheckCpuAsync (PerformanceMonitor.Alerting/AlertEngine.cs:334-388) thresholds on AlertServerSnapshot.SqlCpuPercent/TotalCpuPercent, populated for SQL Server by ReadLatestCpuAsync (Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs:3246):

SELECT sqlserver_cpu_utilization, other_process_cpu_utilization
FROM cpu_utilization_stats
WHERE server_id = $1
ORDER BY sample_time DESC
LIMIT 1

This is an instance-level percentage-of-capacity gauge, sourced from a SQL Server collector that reads sys.dm_os_ring_buffers' RING_BUFFER_SCHEDULER_MONITOR entries (SQL Server tracks its own and the OS's CPU% internally). The fleet overview's cpu_percent/total_cpu_percent fields (DarlingFleetReader.FleetCpuSql) read the exact same table — confirming this is a SQL-Server-only signal end to end, not just in the alert path. The fleet overview shows no CPU number for any pgmon server today either — this isn't a gap unique to alerting, it's a gap in what's collected at all.

What Postgres actually has

The only CPU-shaped Postgres collector is PgKernelStatsCollector (pg_stat_kcache, PerformanceMonitor.Collectors/PgKernelStatsCollector.cs) — and it measures something categorically different: per-query kernel CPU time (plan_user_time + plan_system_time per queryid, summed since the extension's last reset), not instance-level percent-of-capacity. It answers "which queries burn CPU," not "is this instance CPU-saturated right now." Summing it as a proxy for instance utilization would be wrong in both directions: it excludes background/non-query CPU (autovacuum, WAL, checkpointer, the postmaster itself), and pg_stat_kcache's counters are cumulative-since-reset, not a live percentage — extension resets, coverage gaps, or an idle instance would all produce numbers that don't mean "CPU is high" even when correct.

There is also no CloudWatch integration in this codebase at all — grepped for Amazon.CloudWatch, none found. The only AWS RDS SDK usage (AWSSDK.RDS, Amazon.RDS) is RdsLogSource.cs, scoped to DescribeDBLogFiles/DownloadDBLogFilePortion for the plan-capture/deadlock log-reading collectors (#2698's IAM doc covers exactly this grant) — nothing touches GetMetricData/GetMetricStatistics for AWS/RDS's CPUUtilization metric, which is the actual live source of instance-level CPU for an Aurora/RDS target.

What real parity would require

A new collector calling CloudWatch GetMetricData for the CPUUtilization metric (namespace AWS/RDS, dimension DBInstanceIdentifier) against each monitored Aurora/RDS Postgres instance — the AWSSDK.CloudWatch package isn't referenced anywhere yet.
A new IAM grant: cloudwatch:GetMetricData (or GetMetricStatistics) scoped to the monitored instances, on the monitoring hosts' SSM role — the box currently has no CloudWatch permissions at all (its only RDS-adjacent grant is the log-API one from docs: document IAM permissions plan capture/deadlocks need on Aurora/RDS #2698).
Only once that collector exists does a High CPU alert for Postgres become the same shape of work Add Postgres Deadlocks and Blocking alerts (part of #2711) #2713 did for Deadlocks/Blocking — an EvaluatePgXAsync method reusing the existing simple active-flag+cooldown state machine (AlertEngine.CheckCpuAsync's shape, not RollingCountAlertGate — CPU is a continuous gauge, not an event count).

Scope note

Not fixing this in the current #2711 pass — it's a new collector + new IAM grant, not an alert-wiring task, and shouldn't be forced into the same shape as Deadlocks/Blocking just because the parent issue's table said "light design." Filing this to correct that estimate and give whoever picks it up the actual starting point.

Follow-up research on the AWS-side options, since Erik's hunch was that Aurora likely reads CPU from an AWS source rather than a Postgres one. Grounded in this codebase's actual code and docs — no implementation, no decision made.

Target-type scope — the gap is real, and there's a data-model wrinkle

Self-hosted/self-managed PostgreSQL 13+ is a genuine, documented, first-class target type here, not a theoretical corner case: docs/postgres-first-target-runbook.md frames onboarding as "Aurora PostgreSQL, or self-managed PostgreSQL 13+" (line 25) and its coverage table shows "Self-managed 16+ writer | 7 of 9 collectors apply." So any AWS-API-only CPU source leaves a real population uncovered, by design of the product, not by oversight.

There's also a wrinkle worth knowing before building anything: unlike the SQL Server side (CollectorTargetInfo.IsAwsRds, explicitly documented as "meaningful only when Engine is SqlServer"), PostgreSQL targets have no equivalent AWS-hosting flag at all — the only PG-side hosting fact is IsAurora. So IsAurora == false today means "not Aurora," full stop; it can't distinguish "self-hosted" from "RDS for PostgreSQL, non-Aurora" if that population exists in this fleet. The product's own docs treat it as a binary (Aurora vs. self-managed), suggesting RDS-non-Aurora may not be a real deployed case here — but the code has no fact that could confirm or refute that. A CloudWatch/PI-based collector would need a new target fact to know which servers it even applies to, not just IsAurora.

CloudWatch — the simpler AWS-side option

AWS/RDS namespace, CPUUtilization metric, DBInstanceIdentifier dimension — standard, and Aurora instances publish to this the same as plain RDS (instance-level, not cluster-level; a cluster's writer and each reader are separate DBInstanceIdentifiers, matching how this product already treats Aurora writer/reader as separate monitoring identities per CollectorTargetInfo.IsInRecovery's own doc comment).

Package: AWSSDK.CloudWatch is not yet referenced anywhere — confirmed via Directory.Packages.props, which only has AWSSDK.RDS (4.0.104.4) and AWSSDK.Core. New dependency.
IAM: the monitoring hosts' SSM role has zero CloudWatch permissions today (confirmed in Postgres High CPU alert isn't a light port — there's no instance-level CPU collector to alert on #2719's own investigation). Would need cloudwatch:GetMetricData (or GetMetricStatistics) added, scoped to the RDS/Aurora instance ARNs — same shape as the existing pg-plan-capture-log-access inline policy already granted for RdsLogSource's DownloadDBLogFilePortion use.
Cost/throttle: GetMetricData has real per-request cost and default rate limits. At 50 pgmon clusters this needs batching (one GetMetricData call can request many metrics/instances at once) rather than one call per cluster per cycle — worth designing for from the start, not retrofitting.
Always-on: CPUUtilization is a default RDS/Aurora metric with no per-instance enablement step — every instance already publishes it.

Performance Insights — richer, but has an operational precondition

pi:GetResourceMetrics/pi:DescribeDimensionKeys returns DB Load already decomposed by wait type (a CPU component plus everything else), which could double as a foundation for other alerts later (e.g., a real Postgres analogue for "poison waits," separately being scoped in the comment above this one). But Performance Insights must be explicitly enabled per-instance — it is not on by default on RDS/Aurora. I did not verify PI's actual enablement status across the 52 pgmon clusters in this pass; that's a real prerequisite to check before assuming this option is available fleet-wide, not something to build against blind.

IAM and package needs mirror CloudWatch's (new AWSSDK.PI reference, new IAM grant), but it's a second, separate AWS API surface on top of whatever CloudWatch work happens anyway.

Postgres-side, for completeness

Confirms #2719's finding: pg_stat_kcache is the only CPU-adjacent extension in current use, and it's per-query cumulative kernel time since extension reset — not an instance-level "is the box CPU-bound right now" signal. Found no evidence anywhere in this codebase of pg_proctab, a file_fdw reading /proc/stat, or any other OS-exposure extension — and Aurora's restricted extension allowlist (already a live constraint elsewhere in this product, e.g. pg_stat_statements needing a specific enablement path per the runbook) would likely block most of those on Aurora specifically even if built.

Existing precedent to reuse

RdsLogSource.cs already solves a problem a CloudWatch collector would hit too: Aurora cluster endpoints don't map 1:1 to an instance identifier (a cluster reader endpoint round-robins across replicas). RdsLogSource already refuses cluster-reader/cluster-custom endpoints and resolves to a specific instance via RdsEndpoint.TryParse — that resolution logic should be reused rather than rebuilt for a CloudWatch/PI collector, which needs the same "which specific DBInstanceIdentifier is this" answer. It also sets the credential-handling precedent worth following: default AWS SDK credential chain (the EC2 instance profile), nothing stored, degrade gracefully if the role lacks the grant.

Recommendation (not a decision)

CloudWatch CPUUtilization looks like the lower-risk first step: standard always-on metric, smaller IAM ask, and it follows RdsLogSource's existing pattern closely. Performance Insights is more capable but carries a real unverified precondition (per-instance enablement) and a second AWS API surface — better suited as a later enhancement once basic parity ships than as the first cut. Either way, this only closes the gap for Aurora/RDS-hosted Postgres — self-hosted Postgres targets (real and documented, see above) have no AWS-side answer and would either stay permanently uncovered for this alert or need an entirely different mechanism.

Fix CI: dense-ladder version pin, orphaned doc comment, WPF/web viewe…

Fix remaining #2719 CI gaps: MCP host registration, capability phrasi…

Carry the #2719 PG instance-CPU read on the High CPU triage page

Carry the #2711 PG wait read on the Poison Wait triage page

Fixed by #2727 (Postgres/Aurora instance CPU collector via Performance Insights, wired to the High CPU alert). Merged to dev; ships in the next release.

Add Postgres/Aurora instance CPU collector via Performance Insights, …

V107: the plan_force_actions journal and the per-server force opt-in (e…

Fleet cards read a PostgreSQL target's instance CPU (erikdarlingdata#…

Navigation Menu

Repository navigation

Postgres High CPU alert isn't a light port — there's no instance-level CPU collector to alert on #2719

Summary

What SQL Server's High CPU actually reads

What Postgres actually has

What real parity would require

Scope note

Target-type scope — the gap is real, and there's a data-model wrinkle

CloudWatch — the simpler AWS-side option

Performance Insights — richer, but has an operational precondition

Postgres-side, for completeness

Existing precedent to reuse

Recommendation (not a decision)

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Postgres High CPU alert isn't a light port — there's no instance-level CPU collector to alert on #2719

Description

Summary

What SQL Server's High CPU actually reads

What Postgres actually has

What real parity would require

Scope note

Activity

erikdarlingdata commented on Aug 31, 2026

Target-type scope — the gap is real, and there's a data-model wrinkle

CloudWatch — the simpler AWS-side option

Performance Insights — richer, but has an operational precondition

Postgres-side, for completeness

Existing precedent to reuse

Recommendation (not a decision)

erikdarlingdata commented on Sep 1, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions