Repository navigation
issue-3653 A8 option B lane L3c: PG-target WAL/CPU-burn/sampled-waits tiles - #4178
Merged
Merged
Conversation
…waits fallback and census after the tile move
… read (lesson 1 — WindowCommand only binds $1..$3, so this family silently never fired live)
…pin kernel worst-tile window_samples
…or WAL/CPU-burn/sampled-waits — WAL scenario 1 still failing, needs plant-data debugging
…aviour tests DetectAnomaliesAsync gates the entire detector run on HasBaselineDataAsync (a canary read of pg_database_stats within 30 days of the window end). None of the six tests in this file called the already-written PlantCanaryDatabaseStatsRowAsync helper, so every detector — including the one under test — was skipped and DetectAnomaliesAsync returned zero facts. WalVolume_TwoHourShiftInFourHourWindow now fires as designed; printed evidence (buckets trustworthy, tiles correctly split at hours 12-13, WAL rate correctly differenced) is left in as diagnostic Console.WriteLine calls for the PR body.
Both fixtures planted a shift far larger than 6\u03c3, so the dev-comparison assertion (whole-window mean z must stay BELOW the cutoff) failed: the shift was so large it cleared the cutoff even diluted across the whole window, defeating the point of the scenario. Sized both shifts from the bucket actually read (printed): CPU burn to +30% (~6\u03c3 on its ripple-derived sigma, same shape as the WAL family); sampled waits to 30 relation units (~6\u03c3 on its measured robust sigma). Both scenario-1 tests pass live now.
…n per coordinator ruling The mutation check (AnomalyGate.EvaluateTiles -> null, forcing the never-blind fallback to the whole-window path) already proves 'the whole-window path would have stayed quiet' for every shift test, so the in-test wholeWindowMeanZ < k assertion in WAL 4h / CPU burn / sampled waits was redundant and kept forcing shift-magnitude tuning that had nothing to do with the tile behaviour under test. Removed all three; the mutation check re-run confirms all four shift tests (WAL 4h, WAL 24h, CPU burn, sampled waits) fail when EvaluateTiles is forced null, and the two non-firing tests (lone-spike, sparse-fallback) still pass under the same mutation, as expected.
Debug prints left in from developing PgTargetTileBehaviourWalKernelSampledTests.cs. Assertion messages are untouched.
erikdarlingdata
force-pushed
the
feat/3653-b-l3c
branch
from
September 24, 2026 18:43
3cd9a6c to
c332dad
Compare
erikdarlingdata
changed the title
erikdarlingdata
marked this pull request as ready for review
September 24, 2026 18:43
erikdarlingdata
enabled auto-merge (squash)
September 24, 2026 18:43
…es_total Under tiles, window_samples reports the worst hour tile's count (12, one hour at the 5-min sampler cadence), not the whole window's; the whole window's count moved to window_samples_total (48). The test's "exact source wrote the window" comment is about exact_collections (sampled vs exact), unrelated to this. Coordinator's diagnosis (lesson 3, the same class as #4172/#4178's earlier re-pins).
Coordinator's diagnosis: the stock scenario is uniformly heavy (peak == mean already asserted above), so every hour tile ties on Sigma. EvaluateTiles breaks a tie to the LATER local hour, so the worst tile is the newest one. windowStart is anchored on wall-clock now minus 4 hours, not hour-aligned, so that newest tile is a partial hour whose sample count depends on the minute the test runs -- 12 was never a stable number, and the product behaviour (report the most recent hour of a whole-window problem) is correct. Replaced the exact pin with Assert.InRange(MinTileSamples, 12), and added tiles_scored == tiles_fired (every scored hour is uniformly heavy too, so every one fires). Neither tile_local_hour nor tile_start_ticks is pinned in this test, so nothing to relax there.
erikdarlingdata
added a commit
that referenced
this pull request
Sep 24, 2026
…mments Two real CI failures after the dev rebase, neither a flake: 1. AssertEveryWindowSqlRan reported CpuBurnWindowSql and WalVolumeWindowSql as never-ran. Same situation as the three consts already excluded (SessionWindowSql/CpuWindowSql/WaitRateWindowSql, from lane L3a): L3c (#4178) declared these two only so the census keeps seeing the same SQL text PgTargetFactCollector uses to build the PG_CPU_BURN_CORES/ PG_WAL_VOLUME_SHIFT facts -- the detector itself only ever calls their tile twins, CpuBurnTileWindowSql/WalVolumeTileWindowSql. Added both to the existing supersededByTile exclusion set. 2. DocCommentHygieneTests.NoMemberCarriesTwoStackedSummaryBlocks: two <summary> blocks stacked on SeedSqlServerFamilyTablesAsync (247/252) and SeedPgTargetFamilyTablesAsync (349/353). Checked each pair first, per that test's own warning -- both describe the SAME member, the first an earlier/shorter draft ("one row... background history row") superseded by a later, more complete one written when a lane raised the seeding from 1 row to 12 (explaining the CollapseThreshold floor). Deleted the stale first block in each case; nothing lost, the second already carries everything the first said and more. Local: real dotnet build clean; DocCommentHygieneTests class run clean (0 failed). The guard tests themselves skip locally (no DARLING_TEST_PG) -- Windows CI's live PG service is the arbiter for the fixture-gap fix.
erikdarlingdata
added a commit
that referenced
this pull request
Sep 24, 2026
… error (silent-family guard) (#4179) * Add a live guard: no anomaly detector logs an error over a seeded window (#3653 A8 option B) Both PgAnomalyDetector (SQL Server store) and PgTargetAnomalyDetector (PG target) swallow their own per-family exceptions into a logged Error line, so a SQL error or a missing parameter bind silently kills a family with no fact and no failed test. This adds Darling/Darling.Tests/AnomalyDetectorErrorGuardLiveTests.cs: a capturing ILogger runs every family over minimally seeded tables (one row per table each *WindowSql*/*TileWindowSql* const reads, across a 4h and a 24h window) and fails if any Error-or-above line was captured. Part of #3653 * issue-3653 A8 option B: make the silent-detector guard non-vacuous (lane G1-2) Seed 12 same-hour/day-of-week rows per family table (12 weeks back) instead of 1, so every detector's baseline.SampleCount clears BaselineMath.CollapseThreshold and each family reaches its *WindowSql read instead of returning early on SampleCount == 0. Capture the SQL Npgsql actually executed via CommandCapture.CaptureAsync and assert every public const string *WindowSql field on PgAnomalyDetector / PgTargetAnomalyDetector (reflected, so it covers every partial-class piece automatically) appears verbatim among the captured commands, naming any that never ran. * issue-3653 A8 option B: fix guard build break + wait-profile fixture gap (lane G1-3) - RunBothWindowsAsync now returns Task<bool> (CommandCapture.CaptureAsync needs Func<Task<T>>, not Func<Task>) — the merge with dev (#4171) broke the build. - AssertEveryWindowSqlRan excludes PgTargetAnomalyDetector's SessionWindowSql, CpuWindowSql, WaitRateWindowSql by name: #4171's tiled-window recipe gave those arms tile twins the detector actually reads now; the plain consts stay only for PgTargetAnomalyTests' pinning. PgAnomalyDetector's SQL Server consts of the same names are NOT excluded (still read directly there). - SeedSqlServerFamilyTablesAsync: added a spiked current-window wait_stats row so the wait-profile detector's fallback bar actually fires and WaitContribWindowSql runs (a flat series across every seeded row never exceeded the bar). * issue-3653 B (G1-4): seed PgTarget fixtures for the detector error guard's 4 window consts WIP: DatabaseCounterWindowSql, WaitContribWindowSql and SampledWaitContribWindowSql now run; IoLatencyWindowSql still needs a fix (baseline gate: PgIoReadLatency SampleCount==0). * issue-3653 B (G1-4): attempt IoLatencyWindowSql baseline predecessor row (still gapped) DatabaseCounterWindowSql, WaitContribWindowSql and SampledWaitContribWindowSql now run live. IoLatencyWindowSql still 'never ran' -- the baseline arm's per-15-minute date_bin bucket collapses same-bucket rows via MAX before LAG runs, so a same-bucket predecessor/successor pair does not give LAG anything to diff against. Needs a predecessor in the PRECEDING 15-minute bucket, not the same one. Left for a follow-up; documented in the PR body. * issue-3653 A8 option B (G1-5): seed IoLatencyWindowSql's LAG predecessor in the preceding 15-minute bucket The current-window pg_io_stats pair sat 30 seconds apart, inside the SAME date_bin('15 minutes') bucket, so the sampled CTE's MAX() aggregation collapsed them to one row per bucket before LAG ever ran -- raw_reads stayed NULL every time and IoLatencyWindowSql never executed, exactly the silent-family failure this guard exists to catch. Move the pair 15 minutes apart (one bucket earlier, then the bucket itself) with an increasing reads/read_time_ms counter so LAG resolves a real predecessor. Part of #3653 * 3653 B guard: exclude L3c's two superseded consts; fix stacked doc comments Two real CI failures after the dev rebase, neither a flake: 1. AssertEveryWindowSqlRan reported CpuBurnWindowSql and WalVolumeWindowSql as never-ran. Same situation as the three consts already excluded (SessionWindowSql/CpuWindowSql/WaitRateWindowSql, from lane L3a): L3c (#4178) declared these two only so the census keeps seeing the same SQL text PgTargetFactCollector uses to build the PG_CPU_BURN_CORES/ PG_WAL_VOLUME_SHIFT facts -- the detector itself only ever calls their tile twins, CpuBurnTileWindowSql/WalVolumeTileWindowSql. Added both to the existing supersededByTile exclusion set. 2. DocCommentHygieneTests.NoMemberCarriesTwoStackedSummaryBlocks: two <summary> blocks stacked on SeedSqlServerFamilyTablesAsync (247/252) and SeedPgTargetFamilyTablesAsync (349/353). Checked each pair first, per that test's own warning -- both describe the SAME member, the first an earlier/shorter draft ("one row... background history row") superseded by a later, more complete one written when a lane raised the seeding from 1 row to 12 (explaining the CollapseThreshold floor). Deleted the stale first block in each case; nothing lost, the second already carries everything the first said and more. Local: real dotnet build clean; DocCommentHygieneTests class run clean (0 failed). The guard tests themselves skip locally (no DARLING_TEST_PG) -- Windows CI's live PG service is the arbiter for the fixture-gap fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Behaviour tests
Lane T4178-2 (tests-only, issue-3653 A8 option B), continuing PR #4178's branch
feat/3653-b-l3c.Root cause of the one red test (WAL scenario 1)
WalVolume_TwoHourShiftInFourHourWindow_FiresPerTile…was red with NO fact returned at all. Printedevidence (diagnostic
Console.WriteLinecalls, left in the test file) showed correctly seeded rows,trustworthy buckets and a correctly-split tile SQL, but
DetectAnomaliesAsyncreturned 0 facts of anykind.
PgTargetAnomalyDetector.DetectAnomaliesAsyncgates its entire detector run onHasBaselineDataAsync, a canary read ofpg_database_stats. The file already had aPlantCanaryDatabaseStatsRowAsynchelper written for exactly this, but no test ever called it — atest-fixture bug, not a product bug. Fixed by calling it right after
RegisterServerAsyncin all sixtests.
Coordinator ruling: dropped the in-test "dev did this" z-score assertions
WAL 4h, CPU burn and sampled waits each carried an in-test assertion (
wholeWindowMeanZ < k) meant toprove the whole-window path stays quiet where the tile path fires. Sizing shifts to satisfy it forced
tuning unrelated to the tile behaviour under test. Per the coordinator: the mutation check already
proves the same fact (forcing
AnomalyGate.EvaluateTilesto return null routes every shift testthrough the exact whole-window fallback the assertion was trying to reason about in isolation), so the
in-test assertions were deleted rather than resized. The WAL 24h scenario never had this assertion.
Results (all live against
timescale/timescaledb:2.28.1-pg18)dotnet … -class Darling.Tests.PgTargetTileBehaviourWalKernelSampledTests: Total: 6, Failed: 0,Skipped: 0.
WalVolume_TwoHourShiftInFourHourWindow_FiresPerTile_WhereTheWholeWindowGateWouldStayQuietWalVolume_TwoHourShiftInTwentyFourHourWindow_Fires_WithTheRaisedCutoffWalVolume_LoneSpikeInsideAnHour_DoesNotFireCpuBurn_TwoHourShiftInFourHourWindow_FiresPerTile_WhereTheWholeWindowGateWouldStayQuietSampledWaits_TwoHourShiftInFourHourWindow_FiresPerTile_WhereTheWholeWindowGateWouldStayQuietMutation check — per-test results, as required
Forced
AnomalyGate.EvaluateTilestoreturn nullunconditionally (its first line), rebuilt, re-ran thewhole class live:
WalVolume_TwoHourShiftInFourHourWindow_…(shift)WalVolume_TwoHourShiftInTwentyFourHourWindow_…(shift)WalVolume_LoneSpikeInsideAnHour_DoesNotFire(no-fire)CpuBurn_TwoHourShiftInFourHourWindow_…(shift)SampledWaits_TwoHourShiftInFourHourWindow_…(shift)Total: 6, Failed: 4, Skipped: 0under the mutation — exactly the four shift tests, and only those.Reverted the mutation, rebuilt (0 warnings, confirming the unreachable-code warning the mutation
introduced is gone), re-ran the class: back to 6/6 green.
git statusshows no product-code diffremaining — the mutation was never committed or pushed, only the revert state is in the branch.
Diagnostic prints left in the test file
Per the brief, the printed evidence for WAL scenario 1 (
[DIAG]lines) and the CPU/sampled-wait bucketdumps (
[DIAG cpu],[DIAG sampled]) are left in asConsole.WriteLinecalls in the pushed test file.Not verified
The full Darling suite (214-baseline parity check) and the
*PgTargetWrite*/*PgTargetKernel*/*PgTargetSampledWait*classes were NOT run in this lane — the coordinator's scope trim ended the laneat the mutation check. A follow-up should run those and then the full suite with
DARLING_TEST_PGunset, per the original brief's step 4.
CHANGELOG entry
(
PgTargetTileBehaviourWalKernelSampledTests.cs): none of the six live tests seeded thepg_database_statsbaseline-data canary rowPgTargetAnomalyDetector.DetectAnomaliesAsyncgates on(every detector sat out the pass, not just the one under test). Also dropped three redundant in-test
z-score assertions in favor of the mutation check that already covers the same claim. No product code
changed; mutation-checked
AnomalyGate.EvaluateTilesagainst all four shift scenarios (WAL 4h, WAL24h, CPU burn, sampled waits) — all four fail under the mutation, the two non-firing scenarios do not.