Repository navigation
The two PostgreSQL knobs learn to speak: shared_buffers and max_wal_size at their shipped defaults root advisory cards and reach the incident line only when requested-dominant checkpoints or a measurably short cache co-fire, with the measured share, WAL bytes per interval and per-arm evidence in the prose (#3542 lane 2) - #3671
Merged
Merged
Conversation
…ize at their shipped defaults root advisory cards, and reach the incident line only when the engine's own signal co-fires — requested-dominant checkpoints (PG_CHECKPOINT_PRESSURE) or a measurably short cache (PG_BUFFER_CACHE_PRESSURE) — with the measured share, WAL bytes per interval and per-arm evidence in the prose (#3542 lane 2)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Headline mechanism. A PostgreSQL target's analysis pass now speaks about the two knobs the OtterTune field study found carried 75–95 % of measured tuning value.
CONFIG_PG_SHARED_BUFFERSandCONFIG_PG_MAX_WAL_SIZEare cut from the latestpg_server_configsnapshot, score the 0.4 advisory base when at (or below) their SHIPPED defaults — 128 MB, the lineinitdbwrites; 1 GB, thepostgresql.conf.samplevalue — and root standalone advisory cards on a quiet server. They reach the 0.5 incident line ONLY when the engine's own workload signal co-fires withBaseSeverity > 0:PG_CHECKPOINT_PRESSURE(requested-dominant checkpoints frompg_write_stats— PostgreSQL's own report that WAL hitmax_wal_sizebeforecheckpoint_timeout) for the WAL knob, andPG_BUFFER_CACHE_PRESSURE(one composite over hit ratio,pg_stat_ioevictions and bgwriter halts) for the cache knob. Two graph chains carry the stories —PG_CHECKPOINT_PRESSURE → CONFIG_PG_MAX_WAL_SIZEandPG_BUFFER_CACHE_PRESSURE → CONFIG_PG_SHARED_BUFFERS— and the advice states the server's numbers: the measured requested share, WAL bytes per second and per checkpoint interval, the count the timer alone would have produced, the miss share and block rate, cache turnover per hour, the bgwriter's halt share. Lane 2 of #3542 v1; partial, do not close.The lie it stops telling
Until now a PostgreSQL target's pass had nothing to say about configuration:
analyze_serveransweredemptybeside a server running theinitdbdefaults on a 64 GB host, andaudit_configsaidnot_collected. After this PR the same server returns twoCONFIG_PG_*advisory cards at 0.4 — "shared_buffers is 128 MB — the initdb default", "max_wal_size is 1 GB — the shipped default" — never an incident on their own; and a server whose every checkpoint is WAL-forced returns "Checkpoints are being forced by WAL volume — 240 of 240 checkpoints (100 %) were requested, not timed … It wrote 15 GB of WAL over the window — 1.1 MB/s, about 64 MB per checkpoint interval … the timer alone would have produced about 48" withmax_wal_sizeas the leaf and the sizing arithmetic in the remediation. Both were executed through the realanalyze_serveragainst a planted TimescaleDB 2.28.1 / PostgreSQL 18 store on this machine (below).What the family concludes, and from which table
PgTargetFactCollector.Config.cs— ONE read of the latestpg_server_configsnapshot at or before the window's end (so a historicalcompare_analysiswindow states the setting that applied then), the reader's three-valuesourceexclusion ('client', 'session', 'override') copied verbatim,is_defaultas PostgreSQL's own verdict, a closed name list of sixteen settings. Emits twelveCONFIG_PG_*facts: the two knobs;effective_cache_size,random_page_cost,track_io_timing(convention/meta, D5: advisory band only, never amplified — pinned);checkpoint_timeoutandwal_compressionas CONTEXT (base 0, stated in the write chain's prose — the design's "defaults noted alongside checkpoint-pressure findings only");max_connections/superuser_reserved_connectionsfor lane 3's ceiling;work_mem,maintenance_work_mem,autovacuumfor lanes 6 and 4 to score. The bgwriter knobs ride theshared_buffersfact's metadata and the checkpoint rhythm ridesmax_wal_size's, because the buffer and write collectors (emission order: after Config) read them off the in-memory fact list.snapshot_age_son every fact says the hourly stamp's age.PgSettingValue(NEW, in the Config partial) — the unit normaliser:ToBytes(B/kB/MB/GB/TBand the block-multipleNkB),ToMs/ToSeconds(us/ms/s/min/h/d),ToNumber,ToBool,FormatBytes;-1sentinels preserved in every unit; an embedded suffix honoured only when the unit column is empty (the column wins); an unknown unit isnull, never the bare number. 60 unit cases inPgSettingValueTests— the harness caught my own arithmetic error in one expectation ("128MB"with unit8kBIS 1 MB), which is the point of executing pins.PgTargetFactCollector.Write.cs—pg_write_statsdifferenced withPgDatabaseSql's reset-aware shape (per-sampleLAG,GREATEST(raw, 0),ROW_NUMBER() OVER series > 1 AND stamp IS DISTINCT FROM LAG(stamp)) over the checkpointer and WAL stamps independently — per-sample rather than the tool's first-minus-last so a mid-windowpg_stat_reset_sharedcosts one interval, not the window, with the reset count as metadata. Value = requested share; metadata carries counts,checkpoints_per_hour/requested_per_hour/wal_bytes_per_secover OBSERVED time (A7),wal_tracked(NULL on Aurora'spg_stat_walgap is absence, not zero),expected_timed_checkpoints= observed seconds /checkpoint_timeout. No fact without two samples or with zero checkpoints.PgTargetFactCollector.Buffer.cs— three CTEs, one row:pg_database_statshits/reads (partitioned bydatabase_name, NULL included),pg_io_statsevictions restricted toobject_type = 'relation' AND context = 'normal'(ring-buffer contexts recycle by design),pg_write_statsbgwriter counters — each with the same differencing. Honesty flags:hit_ratio_suppressed(Aurora, off the registry fact'sis_aurora; the V64 storage/orcache note),evictions_tracked(0 below PG 16 — the arm is structurally absent and the fact says so),bgwriter_tracked(0 below PG 14). Two dimensionless ratios with engine-defined denominators:cache_turnovers_per_hour(evictions × block size /shared_buffersper observed hour) andbgwriter_halt_share(maxwritten_clean/ observed seconds ÷bgwriter_delay). Silent on flat counters — the plumbing e2e's single-fact pin survives, executed.The lineage shape of every bar
PgTargetScorer.Config.cs): 128 MB (initdb's writtenshared_buffers), 1 GB (max_wal_sizeshipped default), 4 GB (effective_cache_sizeboot_val), 4.0 (random_page_costboot_val),track_io_timing = off. These facts do NOT carrythreshold_lineage = 0— that stamp marks a chosen number, and these are PostgreSQL's.threshold_lineage = 0stamped by the scorer (.Write.cs,.Buffer.cs): the requested-share ramp 0.5 (majority line) → 0.9; the three-checkpoint count floor; the miss-share ramp 0.10 → 0.50 behind a 50 blocks/s floor; turnover 1 → 10 per hour; bgwriter halt share 0.10 → 0.50; the co-fire boosts (0.25 — exactly 0.4 × 1.25 = 0.5, verified in double arithmetic; 0.3 cause; 0.2 trigger). Each says "calibrate against<table>before the next release" beside its constant.checkpoint_warning(the engine's 30 s spacing line) was considered and rejected as a bar: one-minute counters cannot resolve it. The expected-timed count is the engine-defined SENTENCE the advice states instead.3_600_000in a scorer partial — every rate is computed in the collector overObservedDurationMs.PgTargetThresholdLineageTestsgreen on every partial.The co-fire gating (D5), and the one design correction it forced
The pin is
CONFIG_PG_SHARED_BUFFERSreaches ≥ 0.5 ONLY whenPG_BUFFER_CACHE_PRESSURE.BaseSeverity > 0, through the shippedFactScorer.ScoreAll. Executing that pin exposed the trap:ApplyThresholdFormulagrades a 1 % miss share as 0.05, so "BaseSeverity > 0" would have been "any server that ever read a block" — every default install an INCIDENT on day one. Both pressure facts therefore score 0 below their concerning line (GradeArmin Buffer; the share gate in Write) rather than the formula's fraction — "fired" means "at least concerning", the same self-gating shape THREADPOOL has on the SQL Server side. Pinned: a healthy cache (1 % misses, 50 000 blocks/s) and one nightly bulk load (7 % requested) both leave the knob at exactly 0.4; the composite at its line lifts it to exactly 0.5; a knob NOT at the default is 0 whatever the pressure. The graph edges read the destination's verdict (BaseSeverity > 0), never a size of their own.Advice rules
Value-stated from the fact, never assumed (
128 MB,1 GB,8 GB; the static shape names the default without fabricating the server's number). Every block names its counter-objective (OS page cache double-caching / checkpoint length /work_mem × connections;pg_walfootprint / crash-recovery replay; planner bias for the two cost knobs; clock-read overhead fortrack_io_timing). Sizing is arithmetic the reader can check ("1.1 MB/s × 5 min = 320 MB per timeout; max_wal_size is 1 GB"), labelled as a judgment. Aurora: "WAL volume is not reported on this flavour" rather than0 B; the hit ratio "stated but not graded". PG 15: "pg_stat_io arrived in PostgreSQL 16 … rests on the hit ratio and the bgwriter only". No DDL (D8);effective_cache_sizeappears only in its own card (design §6 D) — both pinned over every composed and static block this lane writes.What it does NOT do
No
audit_configPG projection (plumbing's honestnot_collectedenvelope stands). NoCONFIG_PG_STAT_STATEMENTS_MISSING(needspg_extension_availability, not this read). Noshared_buffers"extreme vseffective_cache_size" arm (§6 D rules it out). No §4b composition checks (v2). NoPG_WAL_VOLUME_SHIFTemission (v2 — its edge and amplifier are declared and inert, pinned). Nowork_mem/maintenance_work_mem/autovacuumSCORING (lanes 6 and 4; the facts exist as context). No wait-event edges (lane 5). No schema, no rung, no shared-file edit: every changed file is one of this lane's eleven partials plus two new test files.Blast radius
PostgreSQL-target-only: no SQL Server switch, reader, or test changed. The plumbing e2e (
APostgresTargetWithADayOfHistory…—Assert.Single(facts)) was EXECUTED with these families active and still holds: no config snapshot → no config facts; flat counters → no buffer fact; nopg_write_statsrows → no write fact. A store behind V102 / V88 / thepg_io_statsrung raises 42P01, classified quiet by the shared reporter.Test plan
New, compile-verified here AND executed on this machine through a net10.0 harness over the built assemblies (327 tests including the plumbing family,
PgTargetThresholdLineageTests,FactCollectorCommandTimeoutTests,FactCollectorFailureReportingTests,AnalysisPass*Tests,StoreSqlClockDisciplineTests,DocCommentHygieneTests,LiveCleanupConversionRatchetTests,FactSourceRegistryTests— all green; first CI execution is the Windows job):PgSettingValueTests— 60 cases: every memory and time unit, block multiples, case, embedded suffix vs column,-1in every unit,nullfor the unnormalisable,FormatBytes.PgTargetKnobsTests— scorer bars (config at/above defaults; requested-share ramp with the majority-line gate and count floor; each composite arm with its floor and its structural-absence flag; MAX not SUM,arms_graded); the D5 pins throughScoreAllfor both knobs; graph + inference (quiet → two lone 0.4 cards; churn →PG_CHECKPOINT_PRESSURE → CONFIG_PG_MAX_WAL_SIZE, knob consumed; short cache → the memory chain; edges read the verdict); advice (numbers, Aurora gap, PG 15/13 absence, static shapes, DDL andeffective_cache_sizesweeps); SQL text (reader's filter verbatim,<= $2anchor, differencing shape,context = 'normal'); and the gated e2e: two servers, defaults snapshot with the convention checks deliberately tuned, 25 h ofpg_database_stats, 241pg_write_statsrows with a requested checkpoint and 64 MB of WAL per minute,analyze_serveranchoredas_ofthe planted window end → quiet: exactly two advisories at 0.4 with the stated headlines; churn: the story path, leaf,1.0, the exact headline and every measured number in the prose,next_toolsnamingget_pg_write_stats;get_analysis_facts source=pg_write→threshold_lineage 0, both amplifiers (default matched, WAL-shift inert);source=pg_buffer→ 0 shown of 13 total.Builds (zero warnings of ours):
PerformanceMonitor.Analysis,PerformanceMonitor.Darling.Analysis,Darling.Tests,Lite.Tests(the one pre-existing xUnit2000 inLogTailOverlapThresholdPinTests.cs:95from #3646),deprecated/Dashboard.Tests, all-p:EnableWindowsTargeting=truefor the test projects.Which component(s) does this affect?
How was this tested?
Above. Live store: throwaway
timescale/timescaledb:2.28.1-pg18on Docker contextdesktop-linux, migrated to the current ladder (V130), torn down after. No production store or fleet server was read. No SQL Server surface is touched.Checklist
dotnet build -c Debug)Partial for #3542 (v1 step 2, lane 2). CHANGELOG deliberately untouched — the coordinator splices this wave's entries.