PostgreSQL I/O family: an operator now learns how slow each data-file read was, on measured Aurora-storage bars, and hears why when the pass cannot know — timing off or a pre-16 major (partial #3691)

[erikdarlingdata]

What does this PR do?

Headline mechanism. A PostgreSQL operator now learns how slow each data-file read was — milliseconds per read over the window, from pg_stat_io (pg_io_stats, PostgreSQL 16+) — graded on bars measured on the dogfood fleet's Aurora storage, and, when the pass cannot know, exactly why it cannot: track_io_timing is off (reads were counted, none were timed), or the major has no pg_stat_io at all. Lane 11 of #3691 fills every stub the v2 plumbing (#3715) declared for the pg_io family and adds its anomaly. PARTIAL for #3691 — checklist line: IO latency facts + anomaly. Do not close.

What the family concludes, and from which table

PG_IO_READ_LATENCY_MS — Fact.Value is Δread_time_ms ÷ Δreads over the window, from pg_io_stats (object_type = 'relation', every backend_type × context), the reset-aware per-identity LAG difference of PgTargetFactCollector.Write.cs / the Buffer partial's eviction CTE (GREATEST(raw, 0), ROW_NUMBER() > 1 AND stats_reset IS DISTINCT FROM LAG(stats_reset); the NULL→timestamp first-reset trap that DarlingPgDatabaseReader.PgDatabaseSql records is not re-derived). Summed over identities first, then divided in C# — so "0 ÷ 0" and "0 ÷ N" stay different shapes. A per-operation figure needs no time divisor, which is why it is the value (#3538 A7); ops_per_sec divides by ObservedDurationMs, never the nominal window.

The trackedness three-way, decided in order (the #3541 A8 lesson: never 0.000 ms).

1. Registry major < 16 (off the PG_SERVER_MAJOR_VERSION fact): the read fact is emitted unavailable = 1, reason_pg_stat_io_absent = 1, no store read. A registry with NO major is not "old" — the rows decide.
2. Fewer than two differenced collections, or a counter never reported (reads IS NULL throughout): no fact — structurally absent is the D6 disclosure's job.
3. Ops > 0 with time exactly 0 over the whole window: track_io_timing is off → unavailable = 1, reason_track_io_timing_off = 1, Value = 0 as the placeholder of a fact that makes no claim; the scorer refuses it, the advice renders the count of untimed reads and points at the CONFIG_PG_TRACK_IO_TIMING card (lane 2's — no duplicate). The knob's value, when the config snapshot was collected, rides beside the data-side decision as track_io_timing_config — corroboration, never the decision (the knob is reloadable and a window can straddle the flip; the advice says which case it sees).
4. Otherwise a measurement (latency_measured = 1), graded only when the window's operations clear the floor (insufficient_ops otherwise — stated, not graded).

PG_IO_WRITE_LATENCY_MS — same flags, same quotient, emitted only when the engine reported a write counter at all (on Aurora the write side of pg_stat_io is NULL — backends do not write data files there — so the fact is absent, not zero). Never graded in v2: the measured population does not report it and no stock bar has been read; a bar here would be an unmeasured number wearing a lineage comment. Context for the checkpoint story; no graph edge (an edge into a base-0 fact could never fire).

ANOMALY_PG_IO_LATENCY — the window's peak HOURLY ms per read (hours under the floor not rated) against the pg_io_read_latency bucket through the shared AnomalyGate, the z-score shape lane 9's session detector takes; registered deviation-scored by the plumbing; folds onto the read fact through AnomalyToFamilies. No amplifier arm of its own — the impact lives in the parent, which is never capped.

The lineage shape of every bar

Bar	Value	Lineage
IoReadLatencyWarningMs	10 ms	measured — ≈ fleet p90 of per-server p99 of hourly ms/read, 14 d × 50 Aurora PostgreSQL clusters, 2026-09-19 (§B1)
IoReadLatencyCriticalMs	30 ms	measured — ≈ fleet max, same read
IoMinimumOps	1,000 ops	measured — servers at 1–17 reads/h produced the calibration's p99 noise; window total in the collector, per hour in baseline + detector (the SQL literal 1000 is pinned equal to the constant by PgTargetIoTests)
IoStatIoMinimumMajor	16	engine-defined — pg_stat_io arrived in 16
PgIoLatencyFloorMs	2 ms	measured — above the median server's p50 (1.29 ms), below every server's p99
PgIoLatencyFallbackMs	20 ms	measured (derived) — the midpoint of the two measured bars; pinned strictly between them, so a young store's anomaly is never a finding the fact did not already make
IoCorroborationBoost	0.3	unmeasured — chosen; the calibrating read is named

Graded facts and fired anomalies carry threshold_lineage = 1 — the bars ARE measured; the honesty is in naming the population. Every scorer comment, the advice and this PR say it: these are Aurora-storage shapes; stock local-disk PostgreSQL is a different population (local NVMe reads under 1 ms; the advice tells a stock operator to treat the warning line as generous and read the number against the volume class). Self-gating below the warning bar (the session / CPU families' shape): 0 under 10 ms, 0.5 at it, 1.0 at 30 — a fired latency fact always means slow storage by the measured population's own standard.

The co-fire gating

IoAmplifiers on the read fact only — three corroborators at 0.3 each, every predicate reading a sibling's BaseSeverity (never Severity, lane 4's finding): the anomaly co-fired (unusual for THIS storage, not its routine); an IO wait crossed its own bar (the IO type rollup OR any named IO: standout — PgTargetScorer.Waits.cs grades one wait once, so exactly one of them is the fired one); PG_BUFFER_CACHE_PRESSURE fired (the cache is sending the reads). Executed: warning alone 0.5; + anomaly 0.65; all three 0.95; critical + all three 1.9 — one condition, one story, past the notify line without touching the cap.

Graph (BuildIoEdges, category io_latency): PG_WAIT_IO_DATAFILEREAD → PG_IO_READ_LATENCY_MS (these waits are long because each read is slow, not only because there are many) — the standout only, never the IO type rollup: lane 5's rule that a rollup roots nothing of its own (PgTargetWaitTests pins GetAllEdges(PG_WAIT_IO) empty; the first CI run caught a rollup edge here and it was removed, the rollup staying a corroborator on the amplifiers, which is a predicate and not a story edge); PG_IO_READ_LATENCY_MS → PG_BUFFER_CACHE_PRESSURE (on to shared_buffers through the memory chain); PG_IO_READ_LATENCY_MS → CONFIG_PG_EFFECTIVE_CACHE_SIZE / → CONFIG_PG_RANDOM_PAGE_COST as context — each fires on the knob's own engine-defined verdict, and neither knob is amplified by latency (D5 holds: the convention card stays a 0.4 leaf of the latency story; latency alone never lifts a knob past the incident line). Predicates read verdicts, never a bar of their own.

Baseline (pg_io_read_latency): hourly ms/read (per-identity MAX(counter) per hour, hour-over-hour difference clamped at 0, summed, floor applied), one CTE chain into the ONE RobustTierScaffold. Stated consequence: one hourly row per hour-of-week bucket per week is 4–5 samples in 30 days, under BaselineMath.RestoreThreshold (15), so SelectBucket collapses to the hour-of-day tier (confidence 0.85) on every pass — the live test pins Tier == HourOnly, and the arm's comment names the alternative (a 15-minute bucket with a 250-read floor would put ~17 samples in each hour-of-week bucket) as a coordinator's call, not this lane's.

Advice

Value-stated, one rendering per trackedness shape, none printing a latency it does not have: "PostgreSQL 15 has no pg_stat_io (16+)" with what that major offers instead (blk_read_time); "50,000 reads counted over the 4 observed hours, none timed (track_io_timing is off)" with the knob's counter-objective (one clock read per I/O — pg_test_timing first); "45.0 ms per operation over only 12 reads — under the floor, not graded"; and the measured block: "Data-file reads averaged 18.4 ms per operation over the 4 observed hours against a 1.3 ms routine for this hour of the week (Aurora storage tier)" — the routine clause off the co-fired anomaly's baseline median. The Aurora / stock split is in the lever: on Aurora there is no disk to swap (working set, shared_buffers, the query); on stock, compare with what the volume class advertises before touching PostgreSQL. fsync / synchronous_commit / full_page_writes are named as posture, never a lever; no DDL. next_tools are the plumbing's rows (get_pg_io_stats, get_pg_io_trend, get_pg_server_config); the prose names get_pg_top_queries (by shared_blks_read) and get_pg_buffer_usage, both verified against the tool inventory.

Boundary extensions (each one line; the coordinator should read these first)

• PerformanceMonitor.Analysis/PgTargetAdvice.Anomaly.cs (lane 9's file): ONE delegating case PgTargetFactKeys.AnomalyIoLatency: return ComposeIoLatencyAnomaly(factsByKey); plus the doc line it falsified ("outside the five" → six). The shared prefix routing sends every ANOMALY_PG_ key into ComposeAnomaly, whose default is null, so without this arm the anomaly card ships with no advice — the exit criterion ("advice in PostgreSQL nouns") is unreachable from the lane's files. The composer itself lives in PgTargetAdvice.Io.cs. The plumbing left no hook here; reported so the next family lanes get one.
• Darling/Darling.Tests/PgTargetAnomalyTests.cs (census pins the plumbing said each lane moves): pg_io_read_latency joins the six served metrics and the (metric, table) census, leaves the inert-stub loop; IoLatencyWindowSql joins the detector-SQL closure; the two Pg* constants join the AnomalyThresholds roster with the floor < fallback pin.
• Darling/Darling.Tests/PgTargetSharedSwitchRoutingTests.cs: the lane-11 delegation line moves Null → NotNull (both keys), and the anomaly is pinned to compose through the PostgreSQL block and never the SQL Server "Anomalous spike" composer.
• AnomalyThresholds.cs: the // lane 11 block at the END of the PG section, as the STEP brief assigns.

What it does NOT do

No write-latency bar (stated above). No change to PgTargetScorer.Anomaly.cs's amplifier table — ANOMALY_PG_IO_LATENCY has no corroborators of its own by design. No stock-PostgreSQL population (Aurora shapes, named as such). No change to the root partials, FactScorer, FactAdvice, InferenceEngine, the reconciler, PgTargetFactKeys, or PgTargetFactCollectorTests. No rung (StorageVersion at V132). The hour-of-day tier consequence is stated, not fixed. The stock-target 1.30 anomaly ceiling stays #3691's v3 item. A SQL Server target's pass is byte-identical by construction: every new arm is behind a pg_ source or a PG_/ANOMALY_PG_ key, and the routing pins that proved v1's byte-identity pass unchanged.

Test plan

New: Darling/Darling.Tests/PgTargetIoTests.cs (19 tests, 1 gated live). Scorer: the self-gating ramp at 9.99 / 10 / 20 / 25 / 30 / 45 ms with the measured stamp; under-floor stated-not-graded with the stamp; unavailable (either reason) as context with NO stamp and Value 0; the write fact never graded, no amplifiers. Amplifiers through FactScorer.ScoreAll with sibling fixtures shaped so the SHARED scorer grades them (a wait at its own concerning bar, the cache composite's hit-ratio arm at its bar, a knob at the shipped default): 0.5 / 0.65 (each of the three) / 0.95 / 1.9; a Lock wait does not count. The anomaly: deviation-scored, AnomalyToFamilies fold, 1.0 saturated with no amplifier results, 0.5 at the cutoff, the exceedance ramp between the bar and saturation. Constants: floor < warning < fallback < critical, major 16, the SQL literal equals IoMinimumOps, the population named where the lineage census reads. The three reads by text (per-identity partition, reset shape, relation filter, IS NOT NULL trackedness, no quotient in SQL; hourly CTEs shared between baseline and detector; scaffold suffix; fences, token, ObservedDurationMs, lineage stamp). Graph: quiet set has no active edge; fired set walks the standout edge in and both onward edges; the IO rollup, the write fact and the anomaly have no edges; exactly three edges out of the read fact. Advice: every trackedness rendering by exact string, never "0 ms" on the untimed shape, Aurora vs stock, the routine clause, the anomaly's headline / investigation / first-occurrence / static, never "Anomalous spike".

The gated exit criterion (ThirtyOneDaysAtOnePointFiveMs_ThenFourHoursAtTwentyFive_…): 31 days of one-minute pg_io_stats for an Aurora 17 cluster — two reading identities at 1.3–1.7 ms cycling by the hour, a checkpointer with NULL reads, writes NULL everywhere, a stats_reset mid-history — then 250 minutes at 25 ms; a PG 15 server with no I/O rows; a PG 18 server whose reads climb with read_time_ms pinned at 0. Asserts the collector's read fact at exactly 25.0 ms / 26,400 reads / 2 identities / no write fact; the absent and untimed shapes on the other two servers (24,000 untimed reads, Value 0); the bucket trustworthy at the hour-of-day tier; the detector at the display cap with threshold_lineage = 1; and through the REAL analyze_server anchored at the window's end: PG_IO_READ_LATENCY_MS rooting a pg_io finding ≥ 1.1375 (0.875 × 1.3, the anomaly co-fire), ANOMALY_PG_IO_LATENCY sharing its incident_id, both advices in PostgreSQL nouns, every next_tools entry get_pg_*, get_analysis_facts showing the one fact with its measured lineage.

Executed here, not only compiled: a net10.0 harness (AssemblyName=Darling.Tests, the 36-file PgTarget census set + this file) against a throwaway timescale/timescaledb:2.28.1-pg18 on Docker context desktop-linux, migrated to V132 by the fixture, torn down after — 441 / 442 green, the one red being the harness csproj tripping DocCommentHygieneTests.TheDerivedProjectListCoversEveryProjectInTheTree (the artifact #3665 and #3715 recorded). That includes every DARLING_TEST_PG e2e in the set (lane 9's, the plumbing's, lanes 6/7's) and this lane's. The three SQL texts were additionally run raw against a planted table with a mid-window reset, a NULL-reads identity and a temp relation identity (excluded), and the sums checked by hand (32,900 reads / 42,770 ms = 1.3 ms; the reset interval dropped; 1 reset counted). Builds, zero warnings of ours: PerformanceMonitor.Analysis, PerformanceMonitor.Darling.Analysis, PerformanceMonitor.Darling.Service, Darling.Tests, Lite.Tests (the one xUnit2000 in LogTailOverlapThresholdPinTests.cs:95 is pre-existing on dev), deprecated/Dashboard.Tests. First execution of the rest of the repo's censuses is CI's Windows job. No production store or fleet server was read; the bars come from the coordinator's published calibration file.

Which component(s) does this affect?

• Lite
• Darling
• Lite Tests
• Darling Tests
• SQL collection scripts
• Documentation
• Full Dashboard (deprecated)
• CLI Installer (deprecated)

How was this tested?

Above. Throwaway container only; no production read.

Checklist

• I have read the contributing guide
• My code builds with zero warnings (dotnet build -c Debug)
• I have tested my changes against at least one SQL Server version (the SQL Server pass is unreachable by every new arm; the routing pins that proved v1's byte-identity pass unchanged)
• I have not introduced any hardcoded credentials or server names

Partial for #3691 — IO latency facts + anomaly. CHANGELOG deliberately untouched — the coordinator splices this wave's entries.