Skip to content

The two PostgreSQL knobs learn to speak: shared_buffers and max_wal_size at their shipped defaults root advisory cards and reach the incident line only when requested-dominant checkpoints or a measurably short cache co-fire, with the measured share, WAL bytes per interval and per-arm evidence in the prose (#3542 lane 2) - #3671

Merged
erikdarlingdata merged 1 commit into
devfrom
feat/3542-pg-two-knobs
Sep 19, 2026

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

What does this PR do?

Headline mechanism. A PostgreSQL target's analysis pass now speaks about the two knobs the OtterTune field study found carried 75–95 % of measured tuning value. CONFIG_PG_SHARED_BUFFERS and CONFIG_PG_MAX_WAL_SIZE are cut from the latest pg_server_config snapshot, score the 0.4 advisory base when at (or below) their SHIPPED defaults — 128 MB, the line initdb writes; 1 GB, the postgresql.conf.sample value — and root standalone advisory cards on a quiet server. They reach the 0.5 incident line ONLY when the engine's own workload signal co-fires with BaseSeverity > 0: PG_CHECKPOINT_PRESSURE (requested-dominant checkpoints from pg_write_stats — PostgreSQL's own report that WAL hit max_wal_size before checkpoint_timeout) for the WAL knob, and PG_BUFFER_CACHE_PRESSURE (one composite over hit ratio, pg_stat_io evictions and bgwriter halts) for the cache knob. Two graph chains carry the stories — PG_CHECKPOINT_PRESSURE → CONFIG_PG_MAX_WAL_SIZE and PG_BUFFER_CACHE_PRESSURE → CONFIG_PG_SHARED_BUFFERS — and the advice states the server's numbers: the measured requested share, WAL bytes per second and per checkpoint interval, the count the timer alone would have produced, the miss share and block rate, cache turnover per hour, the bgwriter's halt share. Lane 2 of #3542 v1; partial, do not close.

The lie it stops telling

Until now a PostgreSQL target's pass had nothing to say about configuration: analyze_server answered empty beside a server running the initdb defaults on a 64 GB host, and audit_config said not_collected. After this PR the same server returns two CONFIG_PG_* advisory cards at 0.4 — "shared_buffers is 128 MB — the initdb default", "max_wal_size is 1 GB — the shipped default" — never an incident on their own; and a server whose every checkpoint is WAL-forced returns "Checkpoints are being forced by WAL volume — 240 of 240 checkpoints (100 %) were requested, not timed … It wrote 15 GB of WAL over the window — 1.1 MB/s, about 64 MB per checkpoint interval … the timer alone would have produced about 48" with max_wal_size as the leaf and the sizing arithmetic in the remediation. Both were executed through the real analyze_server against a planted TimescaleDB 2.28.1 / PostgreSQL 18 store on this machine (below).

What the family concludes, and from which table

  • PgTargetFactCollector.Config.cs — ONE read of the latest pg_server_config snapshot at or before the window's end (so a historical compare_analysis window states the setting that applied then), the reader's three-value source exclusion ('client', 'session', 'override') copied verbatim, is_default as PostgreSQL's own verdict, a closed name list of sixteen settings. Emits twelve CONFIG_PG_* facts: the two knobs; effective_cache_size, random_page_cost, track_io_timing (convention/meta, D5: advisory band only, never amplified — pinned); checkpoint_timeout and wal_compression as CONTEXT (base 0, stated in the write chain's prose — the design's "defaults noted alongside checkpoint-pressure findings only"); max_connections / superuser_reserved_connections for lane 3's ceiling; work_mem, maintenance_work_mem, autovacuum for lanes 6 and 4 to score. The bgwriter knobs ride the shared_buffers fact's metadata and the checkpoint rhythm rides max_wal_size's, because the buffer and write collectors (emission order: after Config) read them off the in-memory fact list. snapshot_age_s on every fact says the hourly stamp's age.
  • PgSettingValue (NEW, in the Config partial) — the unit normaliser: ToBytes (B/kB/MB/GB/TB and the block-multiple NkB), ToMs/ToSeconds (us/ms/s/min/h/d), ToNumber, ToBool, FormatBytes; -1 sentinels preserved in every unit; an embedded suffix honoured only when the unit column is empty (the column wins); an unknown unit is null, never the bare number. 60 unit cases in PgSettingValueTests — the harness caught my own arithmetic error in one expectation ("128MB" with unit 8kB IS 1 MB), which is the point of executing pins.
  • PgTargetFactCollector.Write.cs — pg_write_stats differenced with PgDatabaseSql's reset-aware shape (per-sample LAG, GREATEST(raw, 0), ROW_NUMBER() OVER series > 1 AND stamp IS DISTINCT FROM LAG(stamp)) over the checkpointer and WAL stamps independently — per-sample rather than the tool's first-minus-last so a mid-window pg_stat_reset_shared costs one interval, not the window, with the reset count as metadata. Value = requested share; metadata carries counts, checkpoints_per_hour / requested_per_hour / wal_bytes_per_sec over OBSERVED time (A7), wal_tracked (NULL on Aurora's pg_stat_wal gap is absence, not zero), expected_timed_checkpoints = observed seconds / checkpoint_timeout. No fact without two samples or with zero checkpoints.
  • PgTargetFactCollector.Buffer.cs — three CTEs, one row: pg_database_stats hits/reads (partitioned by database_name, NULL included), pg_io_stats evictions restricted to object_type = 'relation' AND context = 'normal' (ring-buffer contexts recycle by design), pg_write_stats bgwriter counters — each with the same differencing. Honesty flags: hit_ratio_suppressed (Aurora, off the registry fact's is_aurora; the V64 storage/orcache note), evictions_tracked (0 below PG 16 — the arm is structurally absent and the fact says so), bgwriter_tracked (0 below PG 14). Two dimensionless ratios with engine-defined denominators: cache_turnovers_per_hour (evictions × block size / shared_buffers per observed hour) and bgwriter_halt_share (maxwritten_clean / observed seconds ÷ bgwriter_delay). Silent on flat counters — the plumbing e2e's single-fact pin survives, executed.

The lineage shape of every bar

  • Engine-defined (PgTargetScorer.Config.cs): 128 MB (initdb's written shared_buffers), 1 GB (max_wal_size shipped default), 4 GB (effective_cache_size boot_val), 4.0 (random_page_cost boot_val), track_io_timing = off. These facts do NOT carry threshold_lineage = 0 — that stamp marks a chosen number, and these are PostgreSQL's.
  • Unmeasured, threshold_lineage = 0 stamped by the scorer (.Write.cs, .Buffer.cs): the requested-share ramp 0.5 (majority line) → 0.9; the three-checkpoint count floor; the miss-share ramp 0.10 → 0.50 behind a 50 blocks/s floor; turnover 1 → 10 per hour; bgwriter halt share 0.10 → 0.50; the co-fire boosts (0.25 — exactly 0.4 × 1.25 = 0.5, verified in double arithmetic; 0.3 cause; 0.2 trigger). Each says "calibrate against <table> before the next release" beside its constant. checkpoint_warning (the engine's 30 s spacing line) was considered and rejected as a bar: one-minute counters cannot resolve it. The expected-timed count is the engine-defined SENTENCE the advice states instead.
  • No SQL Server constant reused by value; no 3_600_000 in a scorer partial — every rate is computed in the collector over ObservedDurationMs. PgTargetThresholdLineageTests green on every partial.

The co-fire gating (D5), and the one design correction it forced

The pin is CONFIG_PG_SHARED_BUFFERS reaches ≥ 0.5 ONLY when PG_BUFFER_CACHE_PRESSURE.BaseSeverity > 0, through the shipped FactScorer.ScoreAll. Executing that pin exposed the trap: ApplyThresholdFormula grades a 1 % miss share as 0.05, so "BaseSeverity > 0" would have been "any server that ever read a block" — every default install an INCIDENT on day one. Both pressure facts therefore score 0 below their concerning line (GradeArm in Buffer; the share gate in Write) rather than the formula's fraction — "fired" means "at least concerning", the same self-gating shape THREADPOOL has on the SQL Server side. Pinned: a healthy cache (1 % misses, 50 000 blocks/s) and one nightly bulk load (7 % requested) both leave the knob at exactly 0.4; the composite at its line lifts it to exactly 0.5; a knob NOT at the default is 0 whatever the pressure. The graph edges read the destination's verdict (BaseSeverity > 0), never a size of their own.

Advice rules

Value-stated from the fact, never assumed (128 MB, 1 GB, 8 GB; the static shape names the default without fabricating the server's number). Every block names its counter-objective (OS page cache double-caching / checkpoint length / work_mem × connections; pg_wal footprint / crash-recovery replay; planner bias for the two cost knobs; clock-read overhead for track_io_timing). Sizing is arithmetic the reader can check ("1.1 MB/s × 5 min = 320 MB per timeout; max_wal_size is 1 GB"), labelled as a judgment. Aurora: "WAL volume is not reported on this flavour" rather than 0 B; the hit ratio "stated but not graded". PG 15: "pg_stat_io arrived in PostgreSQL 16 … rests on the hit ratio and the bgwriter only". No DDL (D8); effective_cache_size appears only in its own card (design §6 D) — both pinned over every composed and static block this lane writes.

What it does NOT do

No audit_config PG projection (plumbing's honest not_collected envelope stands). No CONFIG_PG_STAT_STATEMENTS_MISSING (needs pg_extension_availability, not this read). No shared_buffers "extreme vs effective_cache_size" arm (§6 D rules it out). No §4b composition checks (v2). No PG_WAL_VOLUME_SHIFT emission (v2 — its edge and amplifier are declared and inert, pinned). No work_mem / maintenance_work_mem / autovacuum SCORING (lanes 6 and 4; the facts exist as context). No wait-event edges (lane 5). No schema, no rung, no shared-file edit: every changed file is one of this lane's eleven partials plus two new test files.

Blast radius

PostgreSQL-target-only: no SQL Server switch, reader, or test changed. The plumbing e2e (APostgresTargetWithADayOfHistory… — Assert.Single(facts)) was EXECUTED with these families active and still holds: no config snapshot → no config facts; flat counters → no buffer fact; no pg_write_stats rows → no write fact. A store behind V102 / V88 / the pg_io_stats rung raises 42P01, classified quiet by the shared reporter.

Test plan

New, compile-verified here AND executed on this machine through a net10.0 harness over the built assemblies (327 tests including the plumbing family, PgTargetThresholdLineageTests, FactCollectorCommandTimeoutTests, FactCollectorFailureReportingTests, AnalysisPass*Tests, StoreSqlClockDisciplineTests, DocCommentHygieneTests, LiveCleanupConversionRatchetTests, FactSourceRegistryTests — all green; first CI execution is the Windows job):

  • PgSettingValueTests — 60 cases: every memory and time unit, block multiples, case, embedded suffix vs column, -1 in every unit, null for the unnormalisable, FormatBytes.
  • PgTargetKnobsTests — scorer bars (config at/above defaults; requested-share ramp with the majority-line gate and count floor; each composite arm with its floor and its structural-absence flag; MAX not SUM, arms_graded); the D5 pins through ScoreAll for both knobs; graph + inference (quiet → two lone 0.4 cards; churn → PG_CHECKPOINT_PRESSURE → CONFIG_PG_MAX_WAL_SIZE, knob consumed; short cache → the memory chain; edges read the verdict); advice (numbers, Aurora gap, PG 15/13 absence, static shapes, DDL and effective_cache_size sweeps); SQL text (reader's filter verbatim, <= $2 anchor, differencing shape, context = 'normal'); and the gated e2e: two servers, defaults snapshot with the convention checks deliberately tuned, 25 h of pg_database_stats, 241 pg_write_stats rows with a requested checkpoint and 64 MB of WAL per minute, analyze_server anchored as_of the planted window end → quiet: exactly two advisories at 0.4 with the stated headlines; churn: the story path, leaf, 1.0, the exact headline and every measured number in the prose, next_tools naming get_pg_write_stats; get_analysis_facts source=pg_write → threshold_lineage 0, both amplifiers (default matched, WAL-shift inert); source=pg_buffer → 0 shown of 13 total.

Builds (zero warnings of ours): PerformanceMonitor.Analysis, PerformanceMonitor.Darling.Analysis, Darling.Tests, Lite.Tests (the one pre-existing xUnit2000 in LogTailOverlapThresholdPinTests.cs:95 from #3646), deprecated/Dashboard.Tests, all -p:EnableWindowsTargeting=true for the test projects.

Which component(s) does this affect?

  • Lite
  • Darling
  • Lite Tests
  • Darling Tests
  • SQL collection scripts
  • Documentation
  • Full Dashboard (deprecated)
  • CLI Installer (deprecated)

How was this tested?

Above. Live store: throwaway timescale/timescaledb:2.28.1-pg18 on Docker context desktop-linux, migrated to the current ladder (V130), torn down after. No production store or fleet server was read. No SQL Server surface is touched.

Checklist

  • I have read the contributing guide
  • My code builds with zero warnings (dotnet build -c Debug)
  • I have tested my changes against at least one SQL Server version (not applicable — a PostgreSQL-target-only family; the SQL Server pass reads none of the changed files)
  • I have not introduced any hardcoded credentials or server names

Partial for #3542 (v1 step 2, lane 2). CHANGELOG deliberately untouched — the coordinator splices this wave's entries.

…ize at their shipped defaults root advisory cards, and reach the incident line only when the engine's own signal co-fires — requested-dominant checkpoints (PG_CHECKPOINT_PRESSURE) or a measurably short cache (PG_BUFFER_CACHE_PRESSURE) — with the measured share, WAL bytes per interval and per-arm evidence in the prose (#3542 lane 2)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant