[erikdarlingdata]

Fixes #2719.

Problem

PostgreSQL has no instance-level CPU signal anywhere in this codebase. pg_stat_kcache measures per-QUERY kernel time, not "is this instance CPU-saturated right now", and core PostgreSQL exposes nothing else. The fleet overview shows no CPU number for any pgmon server today, and #2711's own table incorrectly classified this as a "light design" alert-wiring task — it's actually a missing-data-source problem.

Decision: Performance Insights over CloudWatch

Live-tested both AWS/RDS/CPUUtilization (CloudWatch) and os.cpuUtilization.total.avg (Performance Insights) against the same instance/window: CloudWatch read ~6.8% (capacity-relative, roomy on Serverless v2), PI read ~16.8% (true OS-level utilization). PI is the honest signal for this fleet, and all 153/153 monitored Aurora Postgres instances already have it enabled — no enablement gap.

What's here

• PgCpuUtilizationCollector (PerformanceMonitor.Collectors) — the schema/column definition. BuildQuery/ReadAsync are unreachable by design (throw NotSupportedException): this collector has no SQL route at all, ever — see its doc comment.
• RdsCpuIngestor (Darling.Service.Targets) — reads PI via AWSSDK.PI, mirroring RdsDeadlockIngestor/RdsPlanIngestor's "reach the target through the AWS API" shape from PostgreSQL has no execution plan capture, which is where a DBM comparison is lost #2538. Resolves the Aurora cluster writer and the instance's DbiResourceId, reads its own resume watermark from the store (not in-memory — see its doc comment for why), and writes via the same binary-COPY path every collector uses.
• PiMetricsUnavailableException — a distinct type from RdsLogUnavailableException so the PERMISSIONS-degradation handler in DarlingWorker names the correct IAM grants (pi:GetResourceMetrics, rds:DescribeDBInstances, rds:DescribeDBClusters) rather than the log-capture ones.
• EvaluatePgCpuAsync (DarlingWorker) — High CPU / CPU Resolved alert pair, wired into EvaluatePostgresAlertsAsync. Reuses SQL Server's own CpuEnabled/CpuThresholdPercent settings and exact metric-name strings ("High CPU", "CPU Resolved") for Datadog/mute-rule/AlertMetricClassifier parity — one threshold means the same thing on both engines. Mirrors EvaluatePgLongRunningQueryAsync's active-flag+cooldown shape (a continuous gauge, not RollingCountAlertGate's rolling-window-count shape).
• DarlingPgCpuUtilizationReader (Darling.Storage) — the alert's read side, latest non-null reading within a 15-minute freshness bound.
• Migration V106 (pg-cpu-utilization), column-for-column verified against PgSchemaGenerator's output (PgSchemaGeneratorTests.EveryPostgresRung_IsIdenticalToTheGeneratedSchema).
• CI's throwaway cluster worker-sizing bumped 71/82 → 72/83 in both build.yml/nightly.yml for the new hypertable (CiClusterWorkerSizingTests derives these from the live collector count).

Aurora/RDS only. Self-hosted PostgreSQL has no equivalent CPU route, so RdsCpuIngestor no-ops silently for a non-RDS host (RdsEndpoint.TryParse returns null) — the same answer RdsLogSource gives for the same case — rather than fake a signal.

Verified

• Full solution build clean (dotnet build PerformanceMonitor.sln).
• V106 DDL and the alert's LatestCpuSql read query, both run against a real local PostgreSQL 17 (throwaway docker container): confirmed the freshest non-null, non-stale reading wins, a stale row is excluded, a null-cpu_percent row is excluded even when it's the freshest sample, and a different server's row never leaks in.
• AWSSDK.PI's actual API surface (types, method signatures, ServiceType.RDS) verified via reflection against the restored package before writing RdsCpuIngestor, rather than guessed from memory.

Test plan

• dotnet build clean, 0 errors
• New: DarlingPgCpuUtilizationReaderTests (structural SQL-text pins, matching DarlingPgSessionStatesReaderTests' own pattern)
• New: RdsCpuIngestorTests (the two decision points reachable without a store connection — non-RDS host no-op, reader/custom endpoint refusal — matching RdsLogSourceTests' scope for the same shared decisions)
• PgSchemaGeneratorTests/CollectorCatalog/CollectorScheduleDefaults/DarlingWorkerTests dispatch-completeness all updated for the new collector

Known gap, not fixed here

Like EvaluatePgLongRunningQueryAsync (added after #2716 shipped), this alert's cooldown/active-flag watermarks are in-memory only and do not survive a service restart. This matches existing shipped precedent, not a regression — filing separately if it's worth fixing across both alerts uniformly.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01Ddt5CYTp8CD2wreF6sunxj

[claude]

Reviewed the new Postgres/Aurora CPU collector + High CPU alert (#2719). Overall this is well-structured and follows the codebase's established patterns closely (PERMISSIONS degradation mirroring #2633, watermark-from-store reasoning, dispatch-table shape, parameterized SQL throughout — no injection concerns). Lite has no Postgres collectors at all today, so there's no Lite/Darling parity gap here; this is correctly Darling-only.

Two concrete issues below as inline comments:

1. Missing index for the sample_time-keyed access pattern on collect.pg_cpu_utilization — the only index created covers (server_id, collection_time), but both the ingestor's watermark query and the alert reader's "latest reading" query key off sample_time, which runs every 30-second alert sweep (s_alertSweepInterval).
2. PgCpuUtilizationCollector doesn't override AppliesTo to restrict itself to Aurora/RDS targets, unlike PgWaitStatsCollector (AppliesTo => target.IsAurora) which gates the identical "no self-hosted route" shape. Without it, self-hosted Postgres targets get this collector scheduled and dispatched forever, permanently producing a no-op SUCCESS/0-row collection_log entry.

Nothing else stood out as a correctness, security, or Lite/Darling-parity problem.

[claude]

The only index here is (server_id, collection_time), but neither hot-path query for this table filters/sorts by collection_time — both key off sample_time:

• RdsCpuIngestor.GetWatermarkAsync: SELECT MAX(sample_time) FROM collect.pg_cpu_utilization WHERE server_id = $1 — once per ingest cycle (5 min/server)
• DarlingPgCpuUtilizationReader.LatestCpuSql: ... WHERE server_id = $1 AND sample_time >= $2 ... ORDER BY sample_time DESC LIMIT 1 — once per alert sweep, i.e. every 30s per server (s_alertSweepInterval)

Without sample_time in an index, Postgres can only use (server_id, collection_time) to narrow by server_id, then has to filter/sort the full matching row set (up to the full 30-day retention window per server, ~8,640 rows at the 5-min cadence) on every single sweep instead of an efficient index scan/limit.

Worth adding (or swapping to) an index on (server_id, sample_time DESC) so both the MAX() watermark read and the ORDER BY sample_time DESC LIMIT 1 alert read can use it directly. The ingestor's own doc comment ("costs one indexed query per cycle" — RdsCpuIngestor.cs:40) assumes this index exists, which it doesn't.

[claude]

This collector doesn't override AppliesTo, so it inherits the base's => true and applies to every Postgres target, including self-hosted. But per this class's own doc comment, self-hosted has no possible route (unlike pg_deadlocks/pg_plan_capture, which fall back to a real pg_read_file-based SQL query for self-hosted — see the dispatch table comment in DarlingWorker.cs). The dispatch entry for pg_cpu_utilization is unconditional and always calls IngestPgCpuAsync, which for a self-hosted host resolves nothing in RdsEndpoint.TryParse and silently no-ops.

Net effect: every self-hosted Postgres target gets this collector scheduled and dispatched every 5 minutes forever, permanently logging a SUCCESS/0-row "no new Performance Insights CPU samples this cycle" collection_log entry for a target class that can structurally never produce one.

There's already a direct precedent for gating exactly this shape — PgWaitStatsCollector.AppliesTo => target.IsAurora (Aurora-only, no self-hosted equivalent). This collector should probably do the same (target.IsAurora || target.IsAwsRds), which would also let the dispatch-table entry drop its explanatory no-op comment since the scheduler would simply never invoke it for a target where it can't do anything.

[claude]

Minor doc/behavior mismatch: this tool description (and the matching DarlingWebEndpoints.cs:1140 catalog description, and the CHANGELOG entry) say "Aurora and RDS only", but PgCpuUtilizationCollector.AppliesTo gates on target.IsAurora alone (PerformanceMonitor.Collectors/PgCpuUtilizationCollector.cs:61). A plain (non-Aurora) RDS Postgres target is filtered out entirely upstream in DarlingWorker's CollectorCatalog.AppliesTo(name, target) pre-dispatch gate, so pg_cpu_utilization never runs for it — even though RdsCpuIngestor.IngestAsync itself would happily handle that host via RdsEndpoint.TryParse. The collector's own doc comment acknowledges this ("nothing in the fleet is that shape" today), but the three user/LLM-facing description strings advertise plain-RDS support the code doesn't currently deliver. Worth tightening the wording to "Aurora only" (or wiring the gate to also allow non-Aurora RDS) so an operator/LLM client isn't misled if a non-Aurora RDS Postgres server is ever added to the fleet.

[claude]

The running tally in this comment doesn't add up: "...= 66, plus pg_cpu_utilization (...) = 69" adds one collector but jumps the total by 3. The Assert.Equal(69, ...) below is correct (matches CollectorCatalog.All.Count), but the count was already drifted (68 actual vs. "= 66" in the old comment, a pre-existing gap this PR didn't introduce) before this edit. Since this line was touched anyway, it'd be worth fixing the arithmetic so the tally is self-consistent again — CONTRIBUTING.md calls out that these comments are relied on ("a comment restating the code is noise... this stops the next person undoing it"), and a visibly-wrong running sum undermines that for the next contributor who adds a collector here.

[claude]

Reviewed the diff (RDS/Aurora Postgres CPU collector via Performance Insights, High CPU alert wiring, viewer/MCP/web-dashboard surfaces, V106 migration, CI worker-sizing bump).

Overall: solid, well-precedented change. RdsCpuIngestor correctly mirrors RdsDeadlockIngestor/RdsPlanIngestor's shape (own AWS clients, store-backed watermark since this ingestor isn't held across the process lifetime, cluster-reader/custom-endpoint refusal, PiMetricsUnavailableException → PERMISSIONS degradation with the right IAM grants named). The alert (EvaluatePgCpuAsync) correctly reuses SQL Server's CpuEnabled/CpuThresholdPercent settings and exact "High CPU"/"CPU Resolved" metric strings, which are already wired into AlertMetricClassifier — genuine parity, not just a claim. V106's DDL matches PgCpuUtilizationCollector.PayloadColumns and the schema-generator test. The WPF grid row indices in ViewerServerTab.xaml line up correctly with the two new rows. CI's worker-sizing bump (71/82 → 72/83) matches the new hypertable count.

Lite/Darling parity: no concern here — PostgreSQL/Aurora monitoring (and all RDS-API-based ingestors: plan capture, deadlocks, and now CPU) is Darling-only today; Lite has no Postgres collector dispatch path at all, so there's nothing for this PR to keep in parity with on the Lite side.

Left two minor inline notes:

1. The MCP tool / web-dashboard-catalog / CHANGELOG descriptions all say "Aurora and RDS" for this collector, but PgCpuUtilizationCollector.AppliesTo gates on IsAurora only, so a non-Aurora RDS Postgres target would never actually get this collector dispatched (filtered out before RdsCpuIngestor is ever reached) despite the ingestor itself supporting that host shape.
2. A comment in PgSchemaGeneratorTests.cs that this PR touched has a running collector-count tally that doesn't add up (66 + 1 new collector ≠ 69) — pre-existing drift, but worth squaring away since the line was already being edited.

Neither is a functional bug; both are doc/comment accuracy nits.

[claude]

Correctness: AppliesTo => target.IsAurora silently excludes plain (non-Aurora) RDS Postgres targets from ever running this collector — contradicting this file's own doc comment and the PR's stated "Aurora/RDS only" scope.

RunDueCollectorsAsync in DarlingWorker.cs gates dispatch with:

if (!CollectorCatalog.AppliesTo(name, runtime.Target)) continue;

and CollectorCatalog.AppliesTo(name, target) = EngineMatches(...) && definition.AppliesTo(target) (CollectorCatalog.cs:140-141). Because this collector's AppliesTo returns target.IsAurora, a plain RDS-Postgres target (not Aurora) never reaches the ["pg_cpu_utilization"] = (r, s, ct) => r.IngestPgCpuAsync(s, ct) dispatch lambda at all — the loop continues before IngestPgCpuAsync/RdsCpuIngestor.IngestAsync is ever invoked.

This directly contradicts:

• This method's own doc comment: "A plain (non-Aurora) RDS Postgres instance would still be tried at runtime by RdsCpuIngestor (it dispatches on RdsEndpoint.TryParse succeeding, not on this flag)" — it is never reached, so it is never tried.
• The dispatch-table comment in DarlingWorker.cs (["pg_cpu_utilization"] entry): "A self-hosted host resolves to nothing in IngestPgCpuAsync's RdsEndpoint.TryParse and the ingestor no-ops" — same problem, self-hosted also fails IsAurora and never reaches the ingestor to no-op.
• RdsCpuIngestorTests.ReaderAndCustomEndpointsAreRefused / ANonRdsHostReturnsZero, which unit-test IngestAsync directly but don't reflect a path the production scheduler will ever actually take for a non-Aurora host.

This differs from the precedent it says it follows (PgWaitStatsCollector.AppliesTo => target.IsAurora), where gating on Aurora is correct because aurora_stat_system_waits() genuinely doesn't exist outside Aurora. AWS Performance Insights, by contrast, is generally available on standalone RDS Postgres instances too — not Aurora-specific — so gating it away here is a real capability gap, not a faithful mirror of the wait-stats precedent.

Compare with PgPlanCaptureCollector.AppliesTo => true (PgPlanCaptureCollector.cs:136), which defers the Aurora/RDS-vs-self-hosted routing decision entirely to the dispatch lambda (s.Target.IsAurora || s.Target.IsAwsRds ? r.IngestRdsPlansAsync(...) : ...). That's the shape this collector needs too: AppliesTo should return true (letting IngestPgCpuAsync/RdsCpuIngestor.IngestAsync's own RdsEndpoint.TryParse decide per-target at runtime, as the comments already claim it does), not target.IsAurora.

Concrete impact: if a plain RDS Postgres server is ever added to the fleet (the PR body says "nothing in the fleet is that shape" today, so this is currently dormant), it will silently never collect pg_cpu_utilization or fire the High CPU alert for that server — no error, no PERMISSIONS row, nothing; just a collector that never gets scheduled.

[claude]

Review summary

Reviewed the Postgres/Aurora CPU-via-Performance-Insights collector and High CPU alert wiring.

Correctness (posted inline): PgCpuUtilizationCollector.AppliesTo gates on target.IsAurora, but the pre-dispatch check in DarlingWorker.RunDueCollectorsAsync (CollectorCatalog.AppliesTo(name, target)) uses that same flag to decide whether IngestPgCpuAsync/RdsCpuIngestor ever gets invoked at all. That means a plain (non-Aurora) RDS Postgres target — which Performance Insights does support — will never have this collector scheduled, contradicting both this file's own doc comment and the dispatch-table comment, which both claim the ingestor's RdsEndpoint.TryParse decides this at runtime. See the inline comment on PerformanceMonitor.Collectors/PgCpuUtilizationCollector.cs:68 for the full trace and the suggested fix (mirror PgPlanCaptureCollector.AppliesTo => true and let the ingestor route per-target).

Lite/Darling parity: No drift — Lite has no PostgreSQL support at all, and the new get_pg_cpu_utilization MCP tool was correctly added to Lite.Tests/CrossAppMcpToolInventoryPinTests.cs's exclusion ratchet alongside the other Postgres-only tools.

Security: No issues found. All new SQL is parameterized ($1/$2/$3), the migration schema-qualifies collect.pg_cpu_utilization per CONTRIBUTING.md, and AWS calls go through the SDK rather than hand-built requests.

Style/other: T-SQL conventions don't really apply here (this collector has no SQL route by design — BuildQuery/ReadAsync correctly throw NotSupportedException, all data arrives via the RDS/PI API). The rest — migration versioning, schema-generator test pin, CI hypertable-count bump, CHANGELOG entry — all line up correctly with the new collector.

No other correctness, performance, or security issues found.