Skip to content

v2 plumbing for the PostgreSQL-target engine: the I/O, replication and bloat families exist as stubs, and the wait-profile anomaly folds onto its wait and can page (partial #3691) - #3715

Merged
erikdarlingdata merged 1 commit into
devfrom
feat/3691-v2-plumbing
Sep 19, 2026
Merged

erikdarlingdata merged 1 commit into
devfrom
feat/3691-v2-plumbing

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

What does this PR do?

Headline mechanism. The v2 plumbing keystone for the PostgreSQL-target analysis engine (#3691) — what #3665 did for v1, done again for the second wave: the v2 vocabulary is declared once (PgTargetSources / PgTargetFactKeys / MetricNames), every new family exists as a stub in its own partial file with a /* filled by lane N */ marker, every shared switch grew exactly one delegating arm per family, and the censuses that pin the shared surface are re-pinned deliberately. Lanes 11 (I/O latency), 12 (replication), 13 (bloat) and 15 (WAL volume) fan out from this merge and never touch a shared file again. PARTIAL for #3691 — do not close.

No content: no threshold, no advice prose, no graph edge, no SQL beyond the existing reads, no rung (StorageVersion stays at V132), no new AnomalyThresholds constant (those arrive with their lineage in the content lanes).

The one behavioural item — the #3689 §5 residue, ticked verbatim from #3691

ANOMALY_PG_WAIT_PROFILE stays solo and can never escape the cap — AnomalyIncidentReconciler.ResolveFamilies resolves the dominant contributor only for the literal ANOMALY_WAIT_PROFILE; PgTargetFactKeys.AnomalyToFamilies has no wait entry; FactScorer.IsExtremeAnomaly has no PG wait-profile arm. Shared-file item.

Both halves close here, because both are shared-file edits and the plumbing lane is the only lane allowed in those files:

  1. Fold. PgTargetFactKeys.WaitProfileFamilies(metadata) resolves the profile's fold parent per story from its dominant (largest-ms) contrib_Type:event entry — the SQL Server profile's treatment, with one difference the PostgreSQL wait family forces: it returns both keys the wait family may have emitted for that wait, standout first (PG_WAIT_LOCK_RELATION) then type rollup (PG_WAIT_LOCK), because PgTargetScorer.Waits.cs grades one wait once (a rollup scores 0 whenever its standout fired), so exactly one of the two can be the fired parent and the reconciler folds into the first candidate that has one. A contributor without an event yields the rollup alone; ties break on the ordinal name; no contributor metadata → empty → the anomaly stays solo (v1's behaviour for every profile story). AnomalyIncidentReconciler.ResolveFamilies gets one arm, ahead of both maps, delegating to it — the resolution lives beside WaitKey, the only thing that knows how a contributor name becomes a PG_WAIT_* key.
  2. Escape. PgTargetScorer.IsExtremeWaitProfileAnomaly(fact, extremeMultiple) is the PostgreSQL arm of FactScorer.IsExtremeAnomaly, routed off the SAME metadata ScoreRatioAnomaly grades from so the fact can never be extreme on one statistic while scored on another: never for is_new; robust bucket → modified_z ≥ 3 × HeavyTailModifiedZThreshold (15σ, the SQL Server profile's own bar); ratio trigger → ratio ≥ 3 × PgRatioAnomalyThreshold (9× — the PostgreSQL profile's own firing multiple, where the SQL Server arm reads 3 × its 4.0 floor). FactScorer passes its private ExtremeAnomalyMultiple so "extreme" is one number for both engines and the two differ only in the anchor each fired at. The deadlock-rate ratio family stays capped (no amplifier arm; it reaches CRITICAL through its never-capped parent). In v1 a Lock-storm profile at 40σ with every corroborator lit sat at 1.49 — one hundredth under the page line, "by design tonight". Tonight is over.

Every declaration, stub and switch arm — with the lane that fills it

Sources (PgTargetSources, every name ending in Source for the registry sweep): IoSource = "pg_io" (11), ReplicationSource = "pg_replication" (12), BloatSource = "pg_bloat" (13). All 11 → 14; FactScorer.KnownSources 26 → 29; both SKUs' get_analysis_facts source descriptions spell the new list (the Lite literal is the one Lite edit — the same census-pin boundary extension #3665 took).

Keys (PgTargetFactKeys, doc-commented with design § and table): IoReadLatencyMs = "PG_IO_READ_LATENCY_MS", IoWriteLatencyMs = "PG_IO_WRITE_LATENCY_MS", AnomalyIoLatency = "ANOMALY_PG_IO_LATENCY" (11, pg_io_stats); ReplicationLag = "PG_REPLICATION_LAG", SlotRetention = "PG_SLOT_RETENTION", SlotXmin = "PG_SLOT_XMIN", AnomalyReplicationLag = "ANOMALY_PG_REPLICATION_LAG" (12, pg_replication_stats / pg_replication_slot_stats); BloatTrend = "PG_BLOAT_TREND", IndexBloatTrend = "PG_INDEX_BLOAT_TREND" (13, pg_table_bloat_stats hourly / pg_index_bloat daily); AnomalyWalVolume = "ANOMALY_PG_WAL_VOLUME" (15, the detector behind the v1-declared WalVolumeShift); MaintenanceShapeShift = "PG_MAINTENANCE_SHAPE_SHIFT" (wave 2 — declared, no stub, no lane). AnomalyToFamilies gains AnomalyIoLatency → IoReadLatencyMs, AnomalyReplicationLag → ReplicationLag, AnomalyWalVolume → CheckpointPressure (§3.11: WAL volume is the leading edge of checkpoint pressure, so the operator's incident is the checkpoint one). ConfigAdvisoryRoots unchanged.

Baseline metric names (MetricNames, constants only): PgIoReadLatency = "pg_io_read_latency" (11), PgReplayLagBytes = "pg_replay_lag_bytes" (12), PgWalBytesPerSec = "pg_wal_bytes_per_sec" (15), PgAutovacuumWorkers = "pg_autovacuum_workers" (wave 2, no arm).

Anomaly membership by detector shape (PgTargetScorer.IsDeviationScoredAnomalyKey, the one edit outside a root file): the three v2 anomalies are z-score-on-one-series detectors and are registered deviation-scored now, so the shared ramp and the extremity escape grade them the day their detector lands. Membership is vocabulary, not a bar.

Stub partials, each with the exact marker the content briefs quote:

File Marker Stub body
Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Io.cs /* filled by lane 11 CollectIoFactsAsync → Task.CompletedTask
…/PgTargetFactCollector.Replication.cs /* filled by lane 12 CollectReplicationFactsAsync → Task.CompletedTask
…/PgTargetFactCollector.Bloat.cs /* filled by lane 13 CollectBloatFactsAsync → Task.CompletedTask
PerformanceMonitor.Analysis/PgTargetScorer.Io.cs /* filled by lane 11 */ ScoreIoFact → 0.0, IoAmplifiers → []
…/PgTargetScorer.Replication.cs /* filled by lane 12 */ ScoreReplicationFact → 0.0, ReplicationAmplifiers → []
…/PgTargetScorer.Bloat.cs /* filled by lane 13 */ ScoreBloatFact → 0.0, BloatAmplifiers → []
…/PgTargetAdvice.Io.cs / .Replication.cs / .Bloat.cs /* filled by lane 11 */ / 12 / 13 ComposeIo / ComposeReplication / ComposeBloat → null
…/PgTargetRelationshipGraph.Io.cs / .Replication.cs / .Bloat.cs /* filled by lane 11 */ / 12 / 13 BuildIoEdges / BuildReplicationEdges / BuildBloatEdges — empty
Darling/…/PgTargetAnomalyDetector.Io.cs / .Replication.cs / .Wal.cs /* filled by lane 11 / 12 / 15 DetectIoAnomalies / DetectReplicationAnomalies / DetectWalVolumeAnomalies → Task.CompletedTask
Darling/…/PgTargetBaselineProvider.Io.cs / .Replication.cs / .Wal.cs /* filled by lane 11 / 12 / 15 IoReadLatencyBaselineQuery / ReplayLagBaselineQuery / WalBytesPerSecBaselineQuery → null

Shared-switch arms (one per family, nothing else moved):

  • PgTargetScorer.ScoreBase: IoSource → ScoreIoFact, ReplicationSource → ScoreReplicationFact, BloatSource → ScoreBloatFact; Amplifiers: the two I/O keys → IoAmplifiers, the three replication keys → ReplicationAmplifiers, the two bloat keys → BloatAmplifiers.
  • PgTargetAdvice.Compose: the same key groups → ComposeIo / ComposeReplication / ComposeBloat (the anomalies take the existing ANOMALY_PG_ arm into ComposeAnomaly).
  • PgTargetRelationshipGraph ctor: BuildIoEdges(); BuildReplicationEdges(); BuildBloatEdges(); after the five v1 chains.
  • PgTargetFactCollector.CollectFactsAsync: CollectIoFactsAsync, CollectReplicationFactsAsync, CollectBloatFactsAsync after CollectCpuFactsAsync — after every v1 family, so a v2 fact composed at collect time can read a v1 context fact already in the list.
  • PgTargetAnomalyDetector (now partial, nothing moved): DetectAnomaliesAsync awaits the three stubs after the five v1 detectors, behind the same baseline-data gate.
  • PgTargetBaselineProvider (now partial, nothing moved): GetPgTargetBaselineQuery gains three arms to the stub partials; a stub's null is the same answer as "no arm", so the shared reader reports no baseline rather than a bucket built from nothing.
  • FactScorer.IsExtremeAnomaly: the one PG arm (above). AnomalyIncidentReconciler.ResolveFamilies: the one PG arm (above).
  • PgTargetToolRecommendations.ByFactKey: a row for every new key (PgTargetMcpSurfaceTests requires one the day a key is declared; which get_pg_* read a family's table is a fact about the product, not a threshold) — get_pg_io_stats / get_pg_io_trend / get_pg_server_config (track_io_timing) for I/O; get_pg_replication_stats / get_pg_replication_slots / get_pg_xmin_horizon / get_pg_write_stats for replication; get_pg_table_bloat / get_pg_index_bloat / get_pg_autovacuum_health / get_pg_index_usage for bloat; get_pg_write_stats for WAL volume; get_pg_autovacuum_health for the wave-2 key.

Lite twin check: FactScorer.IsExtremeAnomaly and AnomalyIncidentReconciler are the shared PerformanceMonitor.Analysis classes — Lite has no private copy, so no twin. Lite's one edit is the get_analysis_facts source-description literal (census pin).

Censuses re-pinned (each edited deliberately, none deleted)

  • PgTargetFactCollectorTests: collect surface 12 → 15 names (V1CollectSurface → CollectSurface), stub-file count 10 → 13, the marker regex filled by lane \d → \d+\b (two-digit lanes — \d alone would have matched 11 on its first digit and told no one), the three v2 markers pinned by exact string. FROM/JOIN whitelist: the nine v2 tables (pg_io_stats, pg_replication_stats, pg_replication_slot_stats, pg_table_bloat_stats, pg_index_bloat, pg_deadlocks, pg_blocking_edges, pg_lock_stats, pg_wait_sampling) are already CollectorCatalog target tables, so the whitelist admits them with no edit — each is now pinned BY NAME with the lane that reads it, so a table leaving the catalog fails here rather than in the lane.
  • PgTargetSharedSwitchRoutingTests: sources 11 → 14; AnomalyToFamilies three new entries + the wait profile is NOT a static entry; new TheWaitProfile_ResolvesItsFamilyFromTheDominantContributor_StandoutThenRollup; the reconciler test folds the profile onto the standout, onto the rollup, and leaves a bare profile solo; the FactScorer IsPgRatioAnomalyKey delegation count 1 → 2 plus the exact escape line; the advice delegation census gains the three v2 stubs as Null (null IS the delegation — each lane moves its own line to NotNull).
  • PgTargetAnomalyTests: anomaly keys 5 → 8 (each in exactly one of deviation / ratio — the three v2 ones deviation); TheRatioFamilies_NeverEscapeTheCap… → TheWaitProfile_EscapesTheCapWhenExtreme_TheDeadlockRate_NeverDoes_AndHasNoAmplifierArm (the 1.49 pin for a 40σ corroborated profile was the residue; it now asserts > 1.49, pins the three readings at their exact bars — 15σ / 14.9σ, 9× / 8.9×, is_new never, deadlock rate never — and that a 14σ corroborated profile is STILL capped at 1.49, so the escape is the only way past); new TheV2BaselinesAndDetectors_AreReachableInertStubs_EachNamingItsLane.
  • FactSourceRegistryTests: 26 → 29 with the sentence extended.
  • PgTargetThresholdLineageTests: passes with zero bars added (the escape's comparisons carry only trivial literals). AuroraOnlySqlIsGatedTests: no new file mentions aurora_stat_ (grepped). DocCommentHygiene: green over every new file (harness run below).

What it does NOT do

No PostgreSQL SQL, no threshold, no advice text, no graph edge, no baseline CTE, no AnomalyThresholds constant, no rung, no MaintenanceShapeShift stub (wave 2). No numbackends, no calibration. The stock-target 1.30 anomaly ceiling stays a v3 item. A SQL Server target's pass is byte-identical by construction: the two new shared arms are gated on PgTargetScorer.IsPgRatioAnomalyKey / the exact ANOMALY_PG_WAIT_PROFILE key, neither of which a SQL Server fact can satisfy, and the routing pins that proved v1's byte-identity pass unchanged.

Test plan

Builds, zero warnings of ours: PerformanceMonitor.Analysis, PerformanceMonitor.Darling.Service (pulls Darling.Analysis), Darling.Tests, Lite.Tests, deprecated/Dashboard.Tests (-p:EnableWindowsTargeting=true on the test projects). The one Lite.Tests warning (LogTailOverlapThresholdPinTests.cs:95 xUnit2000) is pre-existing on dev.

EXECUTED here through a net10.0 harness (AssemblyName=Darling.Tests, the 35-file PgTarget census set) against a throwaway timescale/timescaledb:2.28.1-pg18 store on Docker context desktop-linux, migrated to V132 by the fixture, torn down after: 422 / 423 green, the one red being the harness csproj's own presence tripping DocCommentHygieneTests.TheDerivedProjectListCoversEveryProjectInTheTree (the same artifact #3665 recorded). That includes all six DARLING_TEST_PG-gated e2es — the exit criterion: PgTargetFactCollectorTests.APostgresTargetWithADayOfHistory… (analyze_server on a PostgreSQL target → empty, coverage 1.0, one PG_SERVER_MAJOR_VERSION fact — the same payload as before this PR; the stubs emit nothing) and lane 9's ThirtyDaysOfHistoryAndAFourHourSpike… (corroborated anomalies > 1.5 folded onto their parents — the detector stubs are inert). Lite's AnomalyIncidentReconcilerTests (38) executed green through a second harness against the rebuilt PerformanceMonitor.Analysis. First execution of the rest is CI's Windows job.

Which component(s) does this affect?

  • Lite (one string literal: the get_analysis_facts source description, a census pin)
  • Darling
  • Lite Tests
  • Darling Tests
  • SQL collection scripts
  • Documentation
  • Full Dashboard (deprecated)
  • CLI Installer (deprecated)

How was this tested?

Above. No production store or fleet server was read.

Checklist

  • I have read the contributing guide
  • My code builds with zero warnings (dotnet build -c Debug)
  • I have tested my changes against at least one SQL Server version (the SQL Server pass is unreachable by every new arm; the routing pins that proved v1's byte-identity pass unchanged)
  • I have not introduced any hardcoded credentials or server names

Partial for #3691 (v2 plumbing; lanes 11–16 follow). CHANGELOG deliberately untouched — the coordinator splices this wave's entries.

…ch arms, censuses re-pinned (partial #3691)

Declares the v2 sources (pg_io, pg_replication, pg_bloat), keys and baseline metric names once; stubs the
I/O, replication and bloat families in their own partials with "/* filled by lane N */" markers (collector,
scorer, advice, graph) and the three v2 detectors / baseline arms (Io, Replication, Wal) as reachable inert
partials; wires one delegating arm per shared switch; pre-fills the tool-recommendation rows.

One behavioural item, the v1 residue from #3689 s5: ANOMALY_PG_WAIT_PROFILE now folds onto the wait card
its dominant contributor names (standout then rollup) and can leave the 1.49 tuning-class cap on its own
evidence through the same three readings its SQL Server twin has, anchored on its own firing multiple.

Censuses re-pinned deliberately: collect surface 12 -> 15, KnownSources 26 -> 29 (both SKUs' descriptions),
anomaly keys 5 -> 8, sources 11 -> 14, the FactScorer delegation count, the v2 whitelist tables by name.
@erikdarlingdata
erikdarlingdata enabled auto-merge (squash) September 19, 2026 10:41
@erikdarlingdata
erikdarlingdata merged commit 495a5fc into dev Sep 19, 2026
7 of 8 checks passed
@erikdarlingdata
erikdarlingdata deleted the feat/3691-v2-plumbing branch September 19, 2026 10:41
This was referenced Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant