Skip to content

The PostgreSQL engine absorbs what the v2 lanes reported back: the I/O and WAL baseline tables are floored, the I/O baseline restores hour-of-week at a quarter-hour grain, the WAL co-fire reads the anomaly and not a base-0 fact, a Lock wait reaches its idle holder, and the blocking family exists as stubs for lane 17 (partial #3691, between waves) - #3737

Merged
erikdarlingdata merged 1 commit into
devfrom
feat/3691-between-waves
Sep 19, 2026

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

What does this PR do?

Headline mechanism. The between-waves shared-file batch of #3691 — what #3688 was for v1. Four content lanes of the v2 wave (11 I/O, 12 replication, 14 idle-in-transaction, 15 WAL volume) each reported items they could not make inside their file-disjoint boundary: two baseline tables nobody floored, a baseline grain that could never restore hour-of-week, a co-fire that read a fact whose base severity is 0 by design, an edge lane 14 could not declare without moving lane 5's pin, next-reads the idle fact's advice names but its row did not, three stale docs and a tool description two engines share. This PR decides and makes all of them as one shared-file change, adds the census that makes the class of miss lane 14 found (a declared key no root dispatcher routes) impossible, and lays the wave-3 stubs for the blocking family (lane 17) so that lane never touches a shared file. PARTIAL for #3691 — do not close. No checklist line on #3691 is this PR's; the body says what it does for the coordinator to record.

The lies it stops telling

  • Shortening PostgreSQL retention silently starved two anomaly detectors. DarlingRetentionHorizons.BaselineServingRawCollectors floors a baseline-serving table's purge horizon at the 30-day baseline window (PostgreSQL baselines survive a shortened retention, and pg_cpu carries the CPU dispersion floor, so a 7-day PG retention no longer silently switches anomaly detection off (#3691) #3711 for the v1 four; lane 12 added pg_replication_stats with its arm). Lane 11's pg_io_stats and lane 15's pg_write_stats serve pg_io_read_latency and pg_wal_bytes_per_sec directly and were unfloored: an operator setting PostgreSQL retention to 7 days would have emptied ANOMALY_PG_IO_LATENCY and ANOMALY_PG_WAL_VOLUME with nothing said. Both are floored now, and BaselineSupplyTests no longer keeps a hand list — it derives the expected set by reflecting every MetricNames.Pg* constant, asking the provider for its SQL, and collecting every FROM pg_* it names. The next arm without a floor fails there by construction.
  • Every I/O latency anomaly was judged against the wrong tier. Lane 11 built pg_io_read_latency at the HOUR grain; the scaffold keys rows by hour-of-day × day-of-week, so one row an hour is one sample per hour-of-week bucket per week — four or five in 30 days, under BaselineMath.RestoreThreshold (15) — and SelectBucket collapsed to hour-of-day at 0.85 confidence on every pass; lane 11's live e2e pinned the collapse and reported it. The baseline arm and the detector's window read now bucket at 15 minutes (date_bin('15 minutes', collection_time, TIMESTAMP '2000-01-01') — a fixed quarter-hour origin so every server samples at :00/:15/:30/:45) under a 250-read per-sample floor (PgTargetScorer.IoBaselineBucketMinimumReads, a quarter-hour's share of the collector's hourly IoMinimumOps, which the window fact keeps). The ms-per-read quotient survives re-bucketing (a ratio of two sums is the same ratio over any partition of the rows); 16–20 samples per bucket restore the Full tier. Lineage: the restore threshold served is engine-defined; the 250 is unmeasured and says so — §B1 distributed reads per HOUR (per-server p50 spanning 1–790 k) and never a per-quarter-hour count — and it is a sample-ADMISSION floor, not a grading bar, so the anomaly keeps threshold_lineage = 1 for its measured fire bars and publishes the floor beside them as bucket_reads_floor. The live e2e's pin flips from HourOnly / 0.85 to Full / ≥ 0.8, deliberately; the advice's "peak HOURLY … 1,000-read floor" sentence follows the grain.
  • Three WAL co-fires and one edge were inert by construction. Lane 12 declared PG_WAL_VOLUME_SHIFT → PG_SLOT_RETENTION and two amplifiers (on the lag fact and the slot fact) reading the shift's BaseSeverity > 0; lane 15 shipped the shift as a CONTEXT fact (base 0 by design, the numbers for the advice) with the judgment in ANOMALY_PG_WAL_VOLUME. InferenceEngine walks edges only from facts in the fired set, and BaseSeverity > 0 on a base-0 fact is never true — so the edge could not open and neither amplifier could match; the two advice sentences that read Severity > 0 on the shift never rendered either. The edge is gone; both amplifiers and both sentences read the anomaly (ANOMALY_PG_WAL_VOLUME co-fired — …), the shape lane 15 used for the checkpoint trigger amplifier. No edge from the anomaly either — it folds onto PG_CHECKPOINT_PRESSURE and PgTargetWriteTests pins it has no edges. PgTargetKnobsTests' membership pin on the shift edge and PgTargetReplicationTests' two pins moved with it, each saying why.
  • A Lock wait could not reach the parked transaction holding the locks. Design §3.10 names the idle transaction as the blocking leaf; lane 14 wrote idle → Lock but not Lock → idle, because a Lock wait's edges were lane 5's and PgTargetWaitTests pinned them Single. Declared now from the partial that owns the Lock waits' edges (PgTargetRelationshipGraph.Saturation.cs — lane 5 wrote the saturation mesh there; one source node's edges live in one place), predicate on the idle fact's BASE severity, NOT gated on the saturation share (the pool need not be full for a parked holder to be what the waiters are behind). A Lock wait with no idle fact keeps exactly its previous single active edge — pinned byte-identical. The Single pins became two-destination pins; the file's AddEdge count 9 → 11.
  • The idle fact's next reads did not follow its own advice. The advice sends the operator to get_pg_xmin_horizon and get_pg_blocking; the row offered only get_pg_session_states. Both added, after the session read.

The items, as decided

# Item (reporting lane) Decision Change Pin
1 pg_io_stats / pg_write_stats unfloored (11, 15) Floor both; derive the census DailySummaryHorizon.cs set + doc; BaselineSupplyTests derives the set from the provider's SQL over every MetricNames.Pg* const (a stub arm contributes nothing) Count 2 + 7; the seven tables by name; each is a collector table; the purge seam floors each at 30
2 I/O baseline grain collapses to hour-of-day (11) 15-minute grain, 250-read floor, both reads PgTargetBaselineProvider.Io.cs, PgTargetAnomalyDetector.Io.cs (SQL + metadata rated_samples / peak_sample_reads / peak_sample_ticks / bucket_reads_floor), PgTargetScorer.Io.cs new const with lineage, PgTargetAdvice.Io.cs prose Const = 1000/4; same date_bin expression twice in each read; same floor literal beside the constant's name in both; no reads >= 1000 left; collector SQL carries no floor; live e2e: Full tier, ≥ 15 samples, confidence 0.8–1.0, peak_sample_reads ≥ 250, bucket_reads_floor = 250
3 Inert WalVolumeShift → SlotRetention edge (15) Remove; the two replication WAL amplifiers + two advice sentences read the anomaly PgTargetRelationshipGraph.Replication.cs, PgTargetScorer.Replication.cs, PgTargetAdvice.Replication.cs Shift + big numbers: both WAL amplifiers declared, unmatched, none names the shift; 7σ anomaly: both matched at 0.2, slot severity lifted, both advice blocks say ANOMALY_PG_WAL_VOLUME co-fired; GetAllEdges(WalVolumeShift) empty; source: no WalVolumeShift token in either file
4 Lock → idle leaf (14) Declare both Lock keys → idle, base-gated, in the Lock waits' partial PgTargetRelationshipGraph.Saturation.cs (+2 edges, traversal para rewritten) Wait outranks (0.994 vs 0.556): path [Lock:relation, PG_IDLE_IN_TRANSACTION]; holder outranks (1.2 vs 0.688): [idle, Lock:relation]; saturation-only: single active edge, path [Lock:relation, saturation]; under-floor idle with inflated Severity opens nothing; every Lock edge lives in this file
5 Idle next reads (14) + get_pg_xmin_horizon, + get_pg_blocking PgTargetToolRecommendations.cs Order pinned; both named in PgTargetAdvice.Sessions.cs; the McpSurface census verifies registration
6 Wave-3 blocking stubs (P2's shape) Source pg_blocking; keys PG_BLOCKING_CHAIN, PG_LOCK_WAIT_EVENTS, PG_LONG_RUNNING_QUERY, ANOMALY_PG_BLOCKING; MetricNames.PgBlockedSessions; AnomalyToFamilies fold; deviation-scored membership; six stub files /* filled by lane 17 */; root arms in scorer / advice / graph / collector / detector / baseline; ComposeAnomaly hook (ComposeBlockingAnomaly, a partial declared in the root so lane 17 never edits the anomaly partial); tool rows PgTargetFactKeys.cs, MetricNames.cs, FactScorer.KnownSources 29 → 30, both SKUs' get_analysis_facts source list, 6 new files, 7 root files, PgTargetToolRecommendations.cs Inert through every entry point (base 0, no amplifier, null advice, null anomaly advice, no edge, null baseline, rows present); marker in six files; no aurora_stat_; whitelist pinned by name for pg_log_events / pg_session_states (pg_blocking_edges, pg_lock_stats already pinned)
7 Routing census (14's lesson) New test in PgTargetSharedSwitchRoutingTests EveryDeclaredPgKey_IsRoutedByEveryRootDispatcher_OrExcludedHereByName Source-read: every measured PG_ key named in Amplifiers and Compose or excluded by name with reason (10 / 2 exclusions, each also asserted NOT routed so a stale exclusion fails); every ANOMALY_PG_ key has a case in ComposeAnomaly; every source has a ScoreBase arm; the four prefix arms by exact text; every key has a tool row
8 Stale docs Rewritten PgTargetBaselineProvider.cs D10 para (seven tables, floored); PgTargetFactKeys.WalVolumeShift doc (context fact; the anomaly carries the co-fire; no edge); PgTargetAnomalyTests fence test renamed …FencesEachOfEightFilledDetectors… and extended to the three v2 partials, the blocking stub named as the exemption Source pins on both docs
9 compare_analysis description + band rule (since #3713) Both SKUs name the PostgreSQL four; BandRulesPayload.baseline says "the reading's delta" DarlingMcpTools.cs, Lite/Mcp/McpAnalysisTools.cs, ComparisonBanding.cs Descriptions byte-identical across SKUs and name the four; each of the four maps to a pg_ metric; the rule text

Left undone, deliberately: nothing from the brief. Item 3 went one step further than asked — the LAG fact's WAL amplifier and both advice sentences had the same base-0 defect as the slot's and are fixed with it (falsified-code rule, same PR).

Censuses that fired, and how each was satisfied

  • PgTargetFactCollectorTests: collect surface 15 → 16, stub files 13 → 14, Blocking.cs marker pinned; whitelist gains two named pins (tables already catalog targets, no admission). Edited deliberately (the plumbing precedent).
  • PgTargetSharedSwitchRoutingTests: sources 14 → 15; AnomalyToFamilies gains the blocking fold; advice delegation census gains four Null lines for the stubs (null IS the delegation); the new routing census.
  • PgTargetAnomalyTests: anomaly keys 8 → 9 (deviation-scored); the inert-stub loop gains PgBlockedSessions and the two Darling stub files; the detector order check gains the blocking call; fence test renamed and extended.
  • FactSourceRegistryTests: 29 → 30 with the sentence extended. PgTargetMcpSurfaceTests: derived — rows added for the four keys, no edit.
  • PgTargetIoTests: constants pin, baseline-arm pin, detector pin, static-advice pin and the live e2e's tier pin — each moved with the grain, each saying so. PgTargetWaitTests: the two Single pins → two destinations. PgTargetSessionsTests: AddEdge 9 → 11. PgTargetReplicationTests / PgTargetKnobsTests: the shift-edge pins → Empty. BaselineSupplyTests: hand list → derivation.
  • PgTargetThresholdLineageTests (new const carries unmeasured; no new comparison line), StoreSqlClockDisciplineTests, DocCommentHygieneTests, RepoFileAdoptionTests (single-line anchors; ReadRepoFile, not Lf), CommentFilterAdoptionTests, AuroraOnlySqlIsGatedTests (no new aurora_stat_ mention), LiveCleanupConversionRatchetTests: green, no edit.

Blast radius

SQL Server pass: byte-identical. The only shared-class edits are FactScorer.KnownSources (+1 pg_ member the registry sweep requires), ComparisonBanding.BandRulesPayload (one word of prose), and both SKUs' two description strings. No shared switch changed; the routing pins that proved v1's byte-identity pass unchanged. PostgreSQL pass: a Lock-wait root now walks to a fired idle holder (one story where there were two); a slot-retention or lag finding beside a fired WAL anomaly is lifted 0.2 and its advice says so (previously never); the I/O anomaly is judged against the hour-of-week bucket (previously hour-of-day) and its metadata keys are peak_sample_* (previously peak_hour_* — read by no tool or view; grepped); a 7-day PostgreSQL retention now keeps 30 days of pg_io_stats / pg_write_stats. The blocking stubs emit nothing. No schema, no rung, no StorageVersion bump (V135 is dev's).

What it does NOT do

No blocking content (lane 17). No Aurora suppression of checkpoint pressure (the residue lane). No MaintenanceShapeShift stub (v3). No second calibration read — the 250 floor is stated unmeasured. No two-sided TPS banding (v3 on #3691). No Lite twin beyond the two literals (D1).

Test plan

New Darling/Darling.Tests/PgTargetBetweenWavesV2Tests.cs (7 tests, the table above) and the routing census in PgTargetSharedSwitchRoutingTests. Executed here, not only compiled — a net10.0 harness (AssemblyName=Darling.Tests, the 45-file PgTarget census set incl. BaselineSupplyTests, PgTargetReplicationTests, PgTargetWriteTests, PgTargetBloatTests, PgTargetDeadlockDrillDownTests) against a throwaway timescale/timescaledb:2.28.1-pg18 on Docker context desktop-linux, migrated by the fixture, torn down after: 635 / 636 green, the one red being the harness csproj's own presence tripping DocCommentHygieneTests.TheDerivedProjectListCoversEveryProjectInTheTree (the artifact #3665 / #3688 / #3715 each recorded). That run includes every DARLING_TEST_PG-gated e2e in the set — lane 11's ThirtyOneDaysAtOnePointFiveMs_… executed alone as well (1 / 1 green) with the flipped Full-tier expectation, so the exit criterion (analyze_server on the planted Aurora 17 target: latency finding ≥ 1.1375 with its anomaly in the same incident) holds at the new grain.

Builds, zero warnings of ours: PerformanceMonitor.Analysis, PerformanceMonitor.Darling.Service (pulls Darling.Analysis + Storage), Darling.Tests, Lite.Tests, deprecated/Dashboard.Tests (-p:EnableWindowsTargeting=true on the test projects). The one Lite.Tests warning (LogTailOverlapThresholdPinTests.cs:95 xUnit2000) is pre-existing on dev. CI's Windows job is the first execution of the Lite pins on the two Lite literals.

Which component(s) does this affect?

  • Lite (two string literals: the get_analysis_facts source list and the compare_analysis description — census pins)
  • Darling
  • Lite Tests
  • Darling Tests
  • SQL collection scripts
  • Documentation
  • Full Dashboard (deprecated)
  • CLI Installer (deprecated)

How was this tested?

Above. No production store or fleet server was read; the rig was a local container, torn down.

Checklist

  • I have read the contributing guide
  • My code builds with zero warnings (dotnet build -c Debug)
  • I have tested my changes against at least one SQL Server version (not applicable — the SQL Server pass is unreachable by every changed arm; the shared-class edits are a registry member and prose, and the routing pins that proved v1's byte-identity pass unchanged)
  • I have not introduced any hardcoded credentials or server names

Partial for #3691 (between waves; wave-3 stubs for the blocking family, lane 17). CHANGELOG deliberately untouched — the coordinator splices this wave's entries.

…tables are floored at the baseline window, the I/O baseline restores hour-of-week at a quarter-hour grain with a 250-read floor, the WAL co-fire reads the anomaly instead of a base-0 context fact, a Lock wait walks to its idle-in-transaction holder, the idle fact gets its horizon and blocking reads, the wave-3 blocking family exists as stubs, and a census makes an unrouted key impossible
@erikdarlingdata
erikdarlingdata merged commit 98d52fe into dev Sep 19, 2026
6 of 7 checks passed
@erikdarlingdata
erikdarlingdata deleted the feat/3691-between-waves branch September 19, 2026 13:53
This was referenced Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant