The wait profile of a PostgreSQL analysis pass: Lock, LWLock, IO and IPC rollups with their named standouts as fractions of the wait source's own observed time — Aurora's measured deltas under the three-state interval, or stock's sampling estimate marked is_sampled with its resolution and read only when the Aurora table is empty — so a PostgreSQL operator learns which wait class a backend-equivalent of time is going to, at which evidence grade, and what PostgreSQL's own levers for it are (#3542 lane 5)

[erikdarlingdata]

What does this PR do?

Headline mechanism. A PostgreSQL target's analysis pass now has its wait profile — the pg_waits family (#3542 lane 5, design §2a / §5): four type rollups (Lock, LWLock, IO, IPC) and four named standouts (Lock:relation, LWLock:WALWrite, IO:DataFileRead, IO:WALSync), each keyed through PgTargetFactKeys.WaitKey (case-insensitive, punctuation-collapsed — Aurora re-cases event names across majors and a key that differed by case would split one wait's history), each with Value = wait time as a fraction of the wait source's own observed time: backend-seconds waiting per wall second, time summed over tasks, the axis the SQL Server wait facts grade on (it exceeds 1.0 with concurrent waiters, as theirs does).

Two evidence grades under one key, told apart by metadata, never blended. The design's hardest point (adversarial item A): Aurora's pg_wait_stats is the engine's MEASURED cumulative wait time differenced at collect (the dm_os_wait_stats model); stock's pg_wait_sampling is SAMPLED sightings — Δsample_count × profile_period_ms, an estimate that cannot see a wait shorter than the period, quantises a continuous wait to ±1 period per backend-sample, and counts per backend-sample (ten backends waiting one second is "10 s of waiting"). So:

• PgWaitStatsSql (Aurora) sums GREATEST(delta_wait_time_us, 0) per (type, event) over the collections the window can honestly count, with the three-state interval — the stored sample_interval_seconds V128 gave the table → NULLIF(…, 0); a pre-V128 NULL → LAG(collection_time); never ELSE 0 — so a restart collection (every row's interval 0) contributes no time and no deltas (its rows are excluded by the JOIN, not just its seconds), exactly PgAnomalyDetector.WaitRateWindowSql's shape, not re-derived. The header aggregate is LEFT JOINed so the read always returns one row: an EMPTY window (collection_count = 0) is distinguishable from an UNCOUNTABLE one (rows, but every collection a restart) — the former falls back, the latter emits nothing and does not.
• PgWaitSamplingSql (stock) differences each (event_type, event, query_id) series per consecutive collection with the reader's reset rule (DarlingPgWaitSamplingReader: a count that went backwards is taken whole — everything since the reset — never GREATEST's silent zero), makes the estimate once beside the count (Σ Δsamples × profile_period_ms, per row, so a period that changed mid-window is honoured per interval), and derives its own observed time from LAG over distinct collection times (the table is not a delta family and stores no interval). A series' first in-window sighting contributes nothing — the count may predate the window, or the key may have fallen out of the collector's top-500 and returned with history the window did not see.
• Every fact from the sampled read carries is_sampled = 1 and estimate_resolution_ms = profile_period_ms (plus delta_samples, peak_backends, counter_resets); no fact from the Aurora read carries either (it carries restart_collections). The scorer grades both on the same fraction bars with the estimate stated; a sampled fact under three Δsamples scores 0 (below that the estimate has no resolution). The advice's evidence sentence differs by grade: "the engine measured N s of Lock:relation waiting across M completed waits" vs "seen in N backend-samples at a 10 ms sampling period — about T of waiting, estimated from sampling" plus the resolution and the per-backend-sample caveat. No wait block of either grade says the server "spent" time — the fraction is time-over-tasks and can exceed one.

Routing is READ-then-fallback, on the data, never on the registry kind. pg_wait_stats is read first; only when it has no rows in the window is pg_wait_sampling read. The two collectors are platform-exclusive by construction, so the data says which flavour this is; servers.engine_kind is NULL until a connect stamps it and would blind an Aurora target's waits on a stale row (adversarial item B). AnalysisContext is shared with Lite and did not grow an engine-kind field. A source pin holds that the wait partial's code names neither engine_kind, IsAurora nor EngineKind, and reads the Aurora const before the sampling const.

The denominator is the wait source's OWN observed time, and both are stated. The pass's coverage witness is pg_database_stats (D3, one-minute); on stock the wait series is five-minute and a gap in one is not a gap in the other. Every fact carries wait_source_observed_ms (the divisor), observed_ms (the witness's), witness_fraction (the same wait over the witness), coverage_fraction, sample_count (countable collections) and collection_count; the advice states which the fraction is over and, when the two differ by more than 5 %, says so in a sentence.

One wait is graded once — a rollup is the UNNAMED story. Executing the first draft's chain pins showed the shape the design did not: a standout's seconds are also inside its type's rollup and a standout's bar is tighter, so a rollup graded on its whole fraction rooted a second story about the same seconds right after the standout rooted the first (the engine walks roots in severity order along one linear path; a rollup cannot lead to a consumed standout). So the collector stamps max_standout_fraction on each rollup and the scorer yields the rollup (0, lineage still stated) whenever a named standout of its type is itself a finding; the rollup grades on its whole fraction only when the type's waiting is in events the vocabulary does not name (Lock:transactionid, LWLock:BufferMapping, …) — exactly when "the Lock class" is the most specific true thing to say. Value stays the whole fraction; the profile is not edited, only the grade.

Lineage of every bar (PgTargetThresholdLineageTests executed green)

Bar	Shape
Rollups Lock / LWLock / IPC: 0.20 → 1.0 (WaitRollupConcerning / Critical)	unmeasured — a fifth of a backend continuously; one whole backend
Rollup IO: 0.40 → 2.0 (WaitIoConcerning / Critical)	unmeasured — reading is what a database does; twice the others
Standouts: 0.15 → 1.0 (WaitStandoutConcerning / Critical)	unmeasured — one event holding 0.15 of a backend is a named cause
WaitSampledMinimumSamples = 3	unmeasured — the resolution floor
WaitCoFireBoost = 0.3 (every amplifier)	unmeasured

Every graded fact carries threshold_lineage = 0. The calibrating read is named once in the file: the per-server per-type fraction distribution over pg_wait_stats, 4-hour buckets — the pass #3616 ran over SQL Server wait_stats. No SQL Server wait constant reused by value: the SQL Server table's pairs are (0.01, 0.05, 0.10, 0.25, 0.30, 0.50, 0.75) and a pin asserts none of the PG bars is one of them. Gates are fractions of observed time; no absolute-ms total anywhere. "Fired" = at or past concerning (0 below it, the convention lane 2 set), so every edge and amplifier reading BaseSeverity > 0 on a wait means "a finding".

Co-fire gating and the chains

Amplifiers (WaitAmplifiers, all reading the sibling's BaseSeverity > 0): Lock and Lock:relation ← PG_CONNECTION_SATURATION (lane 3, landed) and PG_IDLE_IN_TRANSACTION (v2, declared, inert); IO:WALSync and LWLock:WALWrite ← PG_CHECKPOINT_PRESSURE; IO:DataFileRead ← PG_BUFFER_CACHE_PRESSURE. Each lifts a concerning wait from 0.5 to 0.65 — above the story line, well short of the notify line; a wait becomes a page on its own fraction, never on a co-fire.

Edges, AddEdge calls added to the three chain files (every predicate reads the destination's verdict):

• .Write.cs: IO:WALSync → PG_CHECKPOINT_PRESSURE, LWLock:WALWrite → PG_CHECKPOINT_PRESSURE → (lane 2's) CONFIG_PG_MAX_WAL_SIZE. Symptom → cause only, and the split when the CAUSE outranks (near-total requested share ×1.3) is pinned as the honest shape: a reverse pressure → wait edge would make the walk prefer the higher-severity wait over the knob and leave the knob to root a third card.
• .Memory.cs: IO:DataFileRead → PG_BUFFER_CACHE_PRESSURE → CONFIG_PG_SHARED_BUFFERS.
• .Saturation.cs (lane 3 landed while this was building; its commented-out lane-5 edge is now live code): a mesh, both directions, Lock:relation ↔ PG_CONNECTION_SATURATION and Lock ↔ PG_CONNECTION_SATURATION — safe here because saturation has no other leaf in v1, so whichever fires harder roots and the other is consumed; pinned wait-led AND saturation-led.

Advice (OtterTune doctrine)

Value-stated from the fact: the wait seconds, the observed seconds and collection count, the backend-equivalents, the share of all waiting (CPU excluded) and of its own type, the peak backends and resets on a sampled fact, the restart count on an Aurora one. Each key's block says what the wait IS in PostgreSQL and names its levers with counter-objectives. The SQL Server lock advice does not transfer: the Lock / Lock:relation blocks say MVCC means readers and writers never queue on each other, that a relation queue is DDL / LOCK TABLE / TRUNCATE / a conflicting mode held by a parked transaction, and name get_pg_blocking for the holder, idle_in_transaction_session_timeout, lock_timeout, CONCURRENTLY — "RCSI" appears nowhere (pinned). Durability settings are not a lever (D6): the WAL-wait blocks name the WAL device, commit batching, max_wal_size, wal_compression, wal_buffers and say in as many words that a commit waiting for its WAL is doing what it promised; synchronous_commit / fsync are not named in any wait block (pinned). No CREATE INDEX (D8; pinned). Static (no-fact) blocks exist for all eight keys and point at get_pg_wait_stats / get_pg_wait_sampling.

What it does NOT do

• The Service-side wait sampler for stock PostgreSQL — targets without pg_wait_sampling get no wait accumulation at all #3604 service-sampler arm's duty cycle is not corrected. When a stock target lacks the extension, the collector polls pg_stat_activity at 1 s for 30 s of every 5-minute cycle and stores profile_period_ms = 1000; Δsamples × 1000 ms then covers only the sampled 30 s, so the fraction over the interval is a ~10× floor. The arm cannot be told apart honestly from this read: the instrument token lives in collector_state, which is outside the analysis read's table whitelist, and the reader's own note says guessing from the period is wrong the day an operator sets the extension to a second. The facts state estimate_resolution_ms = 1000 and the advice calls the estimate a floor; the fix is an out-of-lane item below.
• No query attribution (the sampling table's query_id dimension is a later slice); no ANOMALY_PG_WAIT_PROFILE (lane 9); no wait-profile drill-down tool; no new key, source, schema or knob. No IPC standout — the design names none.
• Both sources are not read on every pass, so "both non-empty" (the adversarial pass's ERROR-worthy finding) is not detected — the plan's READ-then-fallback was chosen for cost; noted.

Blast radius

Four lane-5 partials filled (PgTargetFactCollector.Waits.cs, PgTargetScorer.Waits.cs, PgTargetAdvice.Wait.cs), additive AddEdge calls in the three chain files (lane 3's saturation file: its commented edge made live, its header brought current), two new test files, and one deliberate pin move in PgTargetSharedSwitchRoutingTests (the wait representative flips from Null to NotNull; a key outside the vocabulary still answers null). Zero shared switches, keys or the census touched. Every other family's facts are unaffected; a store with no wait rows emits nothing (the plumbing e2e's Assert.Single(facts) still holds — executed).

Test plan

New (compile-verified AND executed here through a net10.0 xunit.v3 harness, AssemblyName=Darling.Tests, against a throwaway timescale/timescaledb:2.28.1-pg18 on desktop-linux migrated to V131 — 177/178 green across this lane's two files plus PgTargetFactCollectorTests, PgTargetSharedSwitchRoutingTests, PgTargetThresholdLineageTests, PgTargetPostureIsolationTests, PgTargetPostureTests, StoreSqlClockDisciplineTests, LivePostgresCollectionHygieneTests, LiveCleanupConversionRatchetTests, AnalysisPassCommandTimeoutTests, AnalysisPassTokenThreadingTests, RepoFileAdoptionTests, FactSourceRegistryTests, CommentFilterAdoptionTests, AsOfWindowAnchorTests; the one red is LivePostgresCollectionHygieneTests flagging LivePostgresStoreFixture, a harness artifact — that fixture is not compiled into the harness so its type-name exemption cannot resolve; this lane's live class carries [Collection("live-postgres")]. First CI execution is the Windows job.)

• PgTargetWaitTests (14): both reads' shape (three-state interval, NULLIF, LAG, no ELSE 0, restart exclusion, the reader's reset rule, the estimate beside the count, server_id = $1, membership in AllSql, data-routing source pin); the grouping and estimate arithmetic executed on arranged rows through the collector's internal seam (rollups, standouts, shares with CPU excluded and BufferPin included, both denominators, max_standout_fraction, is_sampled + resolution on EVERY sampled fact and on NO Aurora fact, peak_backends per key); empty / uncountable / unknown-type emission; the bar table (every v1 key, unknown → null, re-cased key → same entry, no SQL Server value); the ramp (0 below, 0.5 at, 0.75 mid, 1.0 at critical and past it, lineage stamped, unobserved → 0); sampled = measured on the same bars and the three-sample floor; the rollup yield rule; every amplifier fires only on the sibling's verdict; the saturation mesh both ways; the write chain wait-led and the pinned split when the cause leads; the memory chain; advice pins over all 8 keys × 3 shapes (no "spent" / "RCSI" / CREATE INDEX / synchronous_commit; "estimated from sampling" + resolution + per-backend-sample caveat on sampled; "the engine measured" on Aurora; value-stated numbers; static blocks name the PG reads); the relation-lock block's levers and co-fire sentence; the witness-divergence sentence only when the denominators differ; the shared entry points answer this family's block.
• PgTargetWaitLiveTests (gated, [Collection("live-postgres")], LiveStoreCleanup): plants an Aurora-stamped server with 241 one-minute pg_wait_stats collections carrying all three interval states (10 pre-V128 NULL rows → LAG, a restart collection at minute 120, stored 60 elsewhere) and six events incl. CPU, and a stock server with 49 five-minute pg_wait_sampling collections (10 ms period, cumulative, a second Lock:relation series that resets mid-window, a CPU/Running series), plus the 25 h witness series for both; asserts the collector alone — Aurora Lock:relation = 0.20 over 14,340 s (239 countable of 241; 1 restart), no sampled keys, shares 12/57 and 12/15, rollups and the other standouts; stock Lock:relation = 302,200 Δsamples × 10 ms over 14,400 s through the fallback, is_sampled = 1, estimate_resolution_ms = 10, 1 reset taken whole, peak_backends = 3 per key — then drives the REAL analyze_server on both: Aurora roots IO:DataFileRead and Lock:relation (the rollups, whose standouts fired, root nothing; IO:WALSync at 0.10 under its bar roots nothing), the relation card's prose says "the engine measured … 0.2 of one backend … 80 % of its own wait type … restart or first sighting", never "estimated" / "spent" / "RCSI", next_tools = get_pg_wait*; stock roots Lock:relation with "estimated from sampling … backend-samples at a 10 ms sampling period … cannot see a wait shorter than 10 ms … reset 1 time(s)"; get_analysis_facts source=pg_waits shows 7 Aurora facts all with threshold_lineage = 0 and none with is_sampled, 4 stock facts all with is_sampled = 1 and resolution 10.

Two defects the execution caught that compile-only would have shipped: (1) a rollup and its standout rooting two stories about the same seconds — now the yield rule; (2) peak_backends taken as the window-wide max, so the CPU/Running series' backend count leaked into every wait fact — now per key.

Builds, zero warnings of ours: PerformanceMonitor.Analysis, PerformanceMonitor.Darling.Analysis, Darling.Tests, Lite.Tests (the one xUnit2000 in LogTailOverlapThresholdPinTests.cs:95 is pre-existing on dev), deprecated/Dashboard.Tests, with -p:EnableWindowsTargeting=true where windows-targeted.

Which component(s) does this affect?

• Lite
• Darling
• Lite Tests
• Darling Tests
• SQL collection scripts
• Documentation
• Full Dashboard (deprecated)
• CLI Installer (deprecated)

How was this tested?

Above. Throwaway TimescaleDB 2.28.1 / PostgreSQL 18 container on Docker context desktop-linux, torn down. No production store or fleet server was read.

Checklist

• I have read the contributing guide
• My code builds with zero warnings (dotnet build -c Debug)
• I have tested my changes against at least one SQL Server version (no SQL Server surface is touched; every change is inside the pg_ vocabulary the shared switches route by prefix)
• I have not introduced any hardcoded credentials or server names

Partial for #3542 (lane 5, the wait profile). CHANGELOG deliberately untouched — the coordinator splices this wave's entries.

[erikdarlingdata]

Pushed cb643cc: the first CI run was 10,863/10,864 — the one red was lane 3's TheSaturationChain_AddsNoEdgeInV1_AndSaysWhyInTheFile, a pin this PR falsifies on purpose (the saturation file's commented lane-5 edge is now live, in both directions). The pin is moved with the change: it now asserts saturation's edges are exactly the two Lock-wait keys, the permissions fact and the v2 idle-in-transaction hook still have none, and the file carries four AddEdge calls with the v2 hook still a comment. Executed here (89/89 across the wait, sessions, collector-census, routing and posture files). No other change.