Repository navigation
The vacuum family of a PostgreSQL analysis pass: a table persistently past its own autovacuum trigger line is named with its ratio, slope and whether autovacuum ran and lost, the wraparound counter and the xmin holder grade on the very constants the Tier-0 alerts page on, and the three chain into one story (#3542 lane 4) - #3674
Merged
Merged
Conversation
… past its own autovacuum trigger line is named with the ratio to that line, its slope and whether autovacuum ran and lost; the wraparound counter and the xmin holder are graded on the very constants the Tier-0 alerts page on, moved to one shared definition; and the three chain into one story (#3542 lane 4) PG_AUTOVACUUM_BACKLOG reads pg_autovacuum_stats and emits one fact for the worst table whose latest three hourly samples sit past the engine's own per-table line (dead/threshold or the insert arm), with the slope of the gauge over the run, the cumulative autovacuum_count differenced (never summed), and the per-table autovacuum_enabled reloption. PG_WRAPAROUND_TREND reads pg_wraparound_stats, grades XID and MultiXact separately against their own settings through the same four arms PostgresAlertEvaluator walks, and adds the slope → time-to-wall estimate with its not-computable branch. PG_XMIN_HOLD reads pg_xmin_horizon and grades the alert's identity arm (majority of holder-bearing collections, ≥ 5 observations, ≥ 50M back), naming the holder kind so the advice states the fix per source. D9: the eight outage-predictor constants move to PostgresOutagePredictorThresholds in PerformanceMonitor.Analysis (the assembly has no project references and Alerting already reaches it through Notifications, so no csproj or lock file moves); PostgresAlertEvaluator keeps the names as aliases, and a source pin holds that neither consumer retypes a literal. A grid pin walks both graders and asserts the verdicts agree. Lineage: the wraparound and xmin bars are engine-defined (the shared symbols); the backlog's concerning bar is the engine's own trigger line, its 10× critical multiple and the three-sample persistence gate are unmeasured and the fact carries threshold_lineage = 0; every amplifier boost is marked unmeasured. Executed here through a net10.0 harness against a throwaway TimescaleDB/PG18 store (376 green incl. the gated e2e); the execution caught two defects reading would not have: amplifier predicates must read BaseSeverity (Severity is assigned fact-by-fact), and a one-directional chain lets an amplified leaf root first and split the incident — the chain is a mesh, as the SQL Server CPU chain is.
This was referenced Sep 19, 2026
erikdarlingdata
added a commit
that referenced
this pull request
Sep 22, 2026
erikdarlingdata
added a commit
that referenced
this pull request
Sep 22, 2026
…three PostgreSQL cards stop hiding table names in metadata keys (#3691) (#3864) * A summary fact can rank its top few objects by name, so a card stops saying "two more, 3.1x for 5 h" without saying which (#3691) Fact carried one string per scope (DatabaseName, ObjectName) and a doubles-only Metadata, so a top-N card could name exactly one object. Three closed lanes each worked around it differently: #3761 rode ranks 2 and 3 as numbers without names, #3722 encoded the name into the metadata KEY (growth_bytes_<schema.table>), #3674 capped its read at one row and deferred the list to a tool. Fact.Ranked is a typed, capped list of RankedObject (ObjectName, DatabaseName, Value, Figures) whose [0] IS the fact's own subject, so a card has one truth about what it is about. Cap three (FactRanked.MaxObjects) - the card is a summary, not the list. Fact is never persisted (only AnalysisFinding is; get_analysis_facts re-runs the pass), so the seam is ephemeral by construction: no migration rung, no store column. Metadata stays doubles-only. FactAdvice.NameTheRest is the one grammar for the clause every migrated family gains ("; and two more: public.orders (3.1x for 5 hours), ..."), empty below two ranked objects - which is the byte-identity arm for every fact that ranks nothing, including every SQL Server fact. Each family passes its own figures formatter because each ranks in its own units. * The autovacuum-disabled card names the other disabled tables instead of describing their shape (#3691, #3761) M1 of the Fact.Ranked migration. The disabled read has returned three rows since #3761, but only the worst one had a name: ranks 2 and 3 rode as disabled_rank_2_ratio / disabled_rank_2_hours, doubles in a doubles-only dictionary, so the best the card could do was "2 more disabled tables have met the same gate (3.1x for 5 h, 1.4x for 2 h)" - the reader was told the shape of two problems and had to call a tool to learn which tables they were. Every returned row now rides on Fact.Ranked with its name, its ratio and its hours, the worst included as Ranked[0] (the seam's invariant: [0] IS the fact's ObjectName/Value). The clause lands on the sentence that states the SUBJECT's ratio and hours, through FactAdvice.NameTheRest, precisely because [0] is the subject - "and two more" can then only be read as "besides the table just described", never against the population count that follows. That population sentence keeps its own number and its own pointer to get_pg_autovacuum_health: the gate's population can exceed the three rows the read returns, so the named clause is a subset of it on purpose. The two rank-key builders are deleted rather than left beside Ranked. Two places to read the same rank is how a card ends up disagreeing with itself. A row whose name triple is entirely null is not ranked at all - an unnamed entry would break the list's own promise. The row limit is now FactRanked.MaxObjects by name, which is the three this read chose first and the seam then adopted. Executed on macOS against the real composer before the pins were written. * The bloat trends' top objects stop hiding their names inside metadata keys (#3691, #3722) M2 of the Fact.Ranked migration. Since #3722 the two bloat trends have carried their top three objects as KEY-ENCODED metadata: growth_bytes_<schema.table>, growth_pct_<...>, dead_tuples_<...> - the object's name inside the key, because Fact.Metadata is doubles-only - and PgTargetScorer.BloatNamedObjects recovered those names with a StartsWith test over every key on the fact. A naming convention doing a type's job, and the graph predicate that reads it had to trust that the convention held. The rows now ride on Fact.Ranked, one typed entry per returned row with its name, its database and its three figures under the fact's own UN-PREFIXED metadata names (growth_bytes, growth_pct, dead_tuples), the worst as Ranked[0] per the seam's invariant. The three prefixes are deleted, not kept beside the list: while both existed an object could be named in one and not the other, and the graph predicate reading one would disagree with the advice reading the other. BloatNamedObjects keeps its signature and its semantics exactly - it is a graph/amplifier predicate helper (PgTargetRelationshipGraph.Bloat, two scorer amplifiers, the index card's table co-fire) - and only its SOURCE moved, including the parent-table projection that turns schema.table.index into schema.table. It still unions Fact.ObjectName in rather than trusting Ranked[0], because hand-built facts reach it and a fact that names an object without ranking must still match. The cards' prose is byte-identical: "Next by growth: public.events (300 MB, 40%, 20,000 dead tuples now); sales.ledger (10 MB)." renders the same characters through the shared NameTheRestList grammar, executed on macOS against the real composer before the pins moved. A fact with one object over the floor emits no such sentence at all, exactly as before. The database-growth family's growth_bytes_<name> encoding (PgTargetAdvice.Growth.cs) is a different family and is out of this lane; its reader stays where it was, named as a candidate for the same move. * [WIP M3] The backlog card reads three rows and ranks two more tables beside the worst (#3691, #3674) — advice prose and pins unfinished * M3 done: the backlog card's live pins name the tables it ranks (#3691, #3674) * Both engines' analysis payloads name the objects their cards rank, instead of leaving an agent to parse them out of the prose (#3691) * The ranked seam's own pins: the invariant every ranking collector owes, the payload's two-or-more rule, and both SKUs' parity (#3691)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Headline mechanism. A PostgreSQL target's analysis pass now has its vacuum family — the [D3] "workload-triggered maintenance" failure class the design names as the one the OtterTune authors lacked metrics for. Three facts under
pg_vacuum, each read from a table the collectors already keep, each graded against a line PostgreSQL itself draws, chained into one story with value-stated advice:PG_AUTOVACUUM_BACKLOG— frompg_autovacuum_stats(hourly, per table, the collector having already computed each table's OWN trigger line with reloptions honoured). The fact is the worst table whose latest three consecutive hourly samples sit past that line — the dead-tuple arm (n_dead_tup > autovacuum_vacuum_threshold + scale_factor × n_live_tup) or the append-only insert arm — with the ratio to its own threshold, the slope of the gauge across the backlog run (a level, differenced by the run's own span — never the nominal window, never an assumed cadence), the autovacuum runs during the run (the cumulativeautovacuum_countdifferenced, never summed; a negative difference is a statistics reset and reads as "unknown", not zero),last_autovacuum, the per-tableautovacuum_enabledreloption, and how many other tables met the gate. One fact per pass, the wayANOMALY_OBJECT_GROWTHcarries its one table inObjectName— the engine keys facts byKey, and the rest of the list isget_pg_autovacuum_health.PG_WRAPAROUND_TREND— frompg_wraparound_stats: the latest reading per database with the window's peak and first reading per counter. XID and MultiXact are graded separately against their own settings (PostgresAlertInfo's per-counter argument, never collapsed) through the same four arms the alert evaluator walks —vacuum_failsafe_age's 74.5% of the 2^31 space, half the space, 2× and 1.0×autovacuum_freeze_max_agewith the PG wraparound WARNING fires on every healthy database — 0.9 × autovacuum_freeze_max_age is routine, not a risk #2689 "keeping up" gate (latest below the window peak). The relatively-worse (database, counter) carries the fact. Analysis adds what only analysis can: the window slope → hours to the wall, with an explicit not-computable branch when the age fell or held.PG_XMIN_HOLD— frompg_xmin_horizon: the latest winning holder and how persistently that (source, holder) won — the alert's identity arm on the alert's bars (≥ 50M back, majority of holder-bearing collections, ≥ 5 observations). The fact carriesholder_source(encoded) andsource:holderasObjectName, so the advice names the fix per kind: end the idle-in-transaction session, drop the abandoned slot, fix the logical consumer, disable the standby's feedback,COMMIT|ROLLBACK PREPARED. Severity ramps from 0.5 at the bar to 1.0 where the held age reaches the server's ownautovacuum_freeze_max_age— engine-defined, because a horizon held that far back stops the forced anti-wraparound vacuum advancingrelfrozenxid(that IS §3.3's chain, as a bar); flat at 0.5 when the wraparound read supplied no setting.D9, made structural. The eight outage-predictor constants (
WraparoundWarningFractionOfFreezeMaxAge,…CriticalMultiple…,WraparoundCeiling, the two ceiling fractions,XminAgeWarningThreshold,XminPersistenceFraction,XminMinimumObservations, plusSlotRetainedWalWarningBytesfor completeness) move toPostgresOutagePredictorThresholdswith their full derivation comments;PostgresAlertEvaluatorkeeps every name as an alias (public const X = PostgresOutagePredictorThresholds.X, the shape its poison-wait constants already take fromPoisonWaitEvaluator), so the read adapter, host, MCP wraparound tool and tests keep compiling. Three pins hold it: value equality of every alias; a source pin that neitherPostgresAlertEvaluator.csnorPgTargetScorer.Vacuum.cscarries0.745/50_000_000/2_147_483_648in code and that each evaluator declaration IS an alias; and a grid pin that walks fourteen (age, setting, keeping-up) points throughPostgresAlertEvaluator.EvaluateWraparoundandPgTargetScorer.GradeWraparoundCounterand asserts null ↔ 0, Warning ↔ [0.5, 1), Critical ↔ 1.0.Where the class lives, and why not
Common: the plan saidPerformanceMonitor.Common.PerformanceMonitor.Analysisreferences nothing — no project, no package — by design, and aCommonreference would dragModelContextProtocol+CredentialManagementinto its closure and rewrite six lockedpackages.lock.jsonfiles for one static class. Alerting already reaches Analysis transitively (Alerting → Notifications → Analysis;AnalysisNotificationServiceis the existing consumer), so the class sits in Analysis at zero project-edge cost. The doc comment on the class states this.Lineage of every bar (
PgTargetThresholdLineageTestsexecuted green)BacklogCriticalRatio)threshold_lineage = 0BacklogPersistenceSamples, bound as$4)autovacuum_naptimeoffers)PostgresOutagePredictorThresholdssymbolsautovacuum_freeze_max_ageBacklogManyTables = 5No absolute-ms gate anywhere; slopes divide by the series' own timestamps.
Co-fire gating and the chain
Amplifiers (all read the sibling's
BaseSeverity, neverSeverity— see below): backlog ← rising slope AND autovacuum ran (running and losing), per-tableautovacuum_enabledoff,PG_XMIN_HOLD,CONFIG_PG_AUTOVACUUM_OFF, ≥ 5 tables in backlog; wraparound ← hold, backlog; hold ← wraparound, backlog;CONFIG_PG_AUTOVACUUM_OFF/CONFIG_PG_MAINT_WORK_MEM← backlog (their D5 half — the base is the config lane'sScoreConfigFact, their source beingpg_config).Graph: the three facts are a mesh (both directions, six edges) with the two config keys as leaves under the backlog. The engine roots at the highest amplified severity and each fact lifts the others, so a one-directional chain rooted at the wraparound let the hold root first and split the incident into three one-fact stories — executing the first draft's pins showed exactly that. The mesh is the shape the SQL Server CPU chain (
SOS_SCHEDULER_YIELD ↔ CXPACKET ↔ THREADPOOL) already takes; every edge description names its hop's mechanism so it reads correctly from either end. Pinned: three facts → one story, wraparound-led in one arrangement, hold-led in another, backlog-led in the live e2e.Advice (OtterTune doctrine)
Every number read from the fact: the table's threshold "sits at 10.5% of the table", the slope per hour, the run count, the setting the counter is graded against, the holder's own age and win fraction. Every recommendation names its counter-objective. The per-table statements carry the table's own figures —
ALTER TABLE public.hot SET (autovacuum_vacuum_cost_delay = 0)(0 is the engine's own "unthrottled") andSET (autovacuum_vacuum_scale_factor = 0.053)(half the table's current effective fraction, computed, not a folklore constant) — in prose, notRemediationTsql. Wraparound states routine vs emergency by arm (at the setting and not recovering → confirm the forced vacuum is winning; past 2× orvacuum_failsafe_age→VACUUM (FREEZE)now withvacuum_cost_delay = 0), and every vacuum block says never disable autovacuum — a regex pin holds that no block recommendsset/turn/switch autovacuum offorautovacuum_enabled = falsewhile allowing the prohibition to be stated. NoCREATE INDEX(D8).What it does NOT do
collection_log(The xmin horizon alert false-fires on 1-of-1 observations and misses rotating holders #3537, Zero is a measurement: health parsers say whether their source was ever seen, a regression with no baseline stays null, and the first point of a differenced trend is no longer a fabricated 0 (#3541 A12) #3642), and an analysis read may name only collector tables andservers(PgTargetFactCollectorTests.AllSql_EveryFromJoinTarget…). Fractioning over holder-bearing collections instead is precisely the dishonest denominator Zero is a measurement: health parsers say whether their source was ever seen, a regression with no baseline stays null, and the first point of a differenced trend is no longer a fabricated 0 (#3541 A12) #3642 removed, and is not done. The numerator (observations_above_threshold) rides in the metadata and the advice says why it is not scored. Out-of-lane item below.CONFIG_PG_AUTOVACUUM_DISABLED0.9-band fact (design §3.1) — no key exists; the reloption rides on the backlog fact as metadata + a 0.5 amplifier + advice.CONFIG_PG_AUTOVACUUM_OFF— that key's source ispg_config, so its base isScoreConfigFactinPgTargetScorer.Config.cs, a file the plumbing marked multi-lane and lane 2 is filling now. Its advice, amplifier and graph edge are here.PG_XMIN_HOLD → PG_IDLE_IN_TRANSACTION / PG_SLOT_XMIN), noPG_BLOAT_TRENDedge, no schema, no knob.Blast radius
PostgresAlertEvaluator: constants aliased, behaviour byte-identical (PostgresAlertEvaluatorTests35/35 green here; the grid pin cross-checks). Every other change is inside the four lane-4 partials, two new test files and the new constants class. The plumbing e2e (Assert.Single(facts)on a store with no vacuum rows) still holds because every read is evidence-gated: no rows → no fact.Test plan
New (compile-verified AND executed here through a net10.0 harness —
AssemblyName=Darling.Tests— against a throwawaytimescale/timescaledb:2.28.1-pg18ondesktop-linux, migrated to V130; 376 green, 0 skipped, incl. the gated e2e; first CI execution is the Windows job):PgTargetVacuumTests(53): D9 alias equality; source pin (no retyped literal, alias shape); 14-point grader agreement grid; warning-ramp ordering; backlog persistence gate + ratio ramp + lineage flag; amplifier predicates (slope×runs, reloption, hold co-fire both ways); xmin identity arm (5 rows incl. the The xmin horizon alert false-fires on 1-of-1 observations and misses rotating holders #3537 3-of-4 false-fire shape) + freeze-age ramp + degenerate setting; holder-source round trip; three-fact one-story wraparound-led; hold-led one-story; two-fact reach; healthy sawtooth + transient holder →server_health; advice value pins (backlog with slope/runs/ALTER statements, insert arm + not-computable branches, wraparound routine vs emergency + time-to-wall + MultiXact naming, xmin per-source remedy ×6, transient-as-context, static blocks for all five keys, config co-fire blocks).PgTargetVacuumLiveTests(gated,[Collection("live-postgres")],LiveStoreCleanup): plants 3 tables × 5 hourly samples (rising-while-running, insert-arm, a run-of-two that must NOT qualify), 2 databases × 49 wraparound readings (climbing at the setting; healthy sawtooth), 41 xmin captures (31 session wins, 10 slot wins) + the 25 h span/coverage series; asserts every metadata number (ratio 5.0, slope 1083.3/h, 2 runs, 2 tables in backlog; arm 1, slope 5M/h, 385.5 h to wall; 31/41, peak 70M, freeze age handed across), the scores, then drives the REALanalyze_server:findings, ONE storyPG_AUTOVACUUM_BACKLOG → PG_WRAPAROUND_TREND → PG_XMIN_HOLD, no second root for the other two, advice text,next_tools=get_pg_*only;get_analysis_facts source=pg_vacuumshows the three.Pre-checked rosters (all executed green):
PgTargetThresholdLineageTests,PgTargetFactCollectorTests(census, AllSql closure, dialect, FROM-target set, e2e),PgTargetSharedSwitchRoutingTests,PgTargetMcpSurfaceTests,FactCollectorCommandTimeoutTests,FactCollectorFailureReportingTests,AnalysisPassCommandTimeoutTests,AnalysisPassTokenThreadingTests,StoreSqlClockDisciplineTests,FactSourceRegistryTests,LiveCleanupConversionRatchetTests,RepoFileAdoptionTests(no LF reader used → no roster entry),CommentFilterAdoptionTests,DocCommentHygieneTests,AsOfWindowAnchorTests,PostgresAlertEvaluatorTests,DarlingPgXminReaderTests.Two defects the execution caught that compile-only would have shipped: (1) amplifier predicates reading
Severitymatched by emission order (Layer 2 assigns it fact-by-fact) — nowBaseSeverity; (2) the one-directional chain splitting the incident — now a mesh.Builds, zero warnings of ours:
PerformanceMonitor.Analysis,PerformanceMonitor.Alerting,PerformanceMonitor.Darling.Analysis,PerformanceMonitor.Darling.Service,Darling.Tests,Lite.Tests(the one xUnit2000 inLogTailOverlapThresholdPinTests.cs:95is pre-existing on dev),deprecated/Dashboard.Tests, all with-p:EnableWindowsTargeting=truewhere windows-targeted.Which component(s) does this affect?
How was this tested?
Above. Throwaway TimescaleDB 2.28.1 / PostgreSQL 18 container on Docker context
desktop-linux, torn down. No production store or fleet server was read.Checklist
dotnet build -c Debug)Partial for #3542 (lane 4, the vacuum family). CHANGELOG deliberately untouched — the coordinator splices this wave's entries.