Repository navigation
issue-3653 A8 option B lane L2a: SQL Server CPU/waits/I-O anomaly detectors move onto per-hour tiles - #4172
Merged
Conversation
erikdarlingdata
changed the title
erikdarlingdata
changed the title
erikdarlingdata
force-pushed
the
feat/3653-b-l2a
branch
from
September 24, 2026 16:58
8de3c50 to
ee85834
Compare
erikdarlingdata
changed the title
erikdarlingdata
added a commit
that referenced
this pull request
Sep 24, 2026
…/6 red, needs fixing)
Owner
Author
Behaviour tests — WIP, NOT DONE (5 of 6 new tests are RED)This lane (T4172, tests-only) adds What's in the fileSix tests, CPU family gets all four scenarios from the shared brief, waits and I/O get scenario 1 only:
Live run result (DARLING_TEST_PG, timescale/timescaledb:2.28.1-pg18, ephemeral container, removed after the run)Only
What still needs doing (next lane / same lane on resume)
What I verified before running out of context
Not verified
CHANGELOG entry(none — tests-only, no product behaviour change) |
erikdarlingdata
added a commit
that referenced
this pull request
Sep 24, 2026
…it profile #3653 A8 option B's tile gate reports the WORST HOUR TILE's mean, not the whole window's, once a detector moves onto tiles. Three of DetectCpuAnomalies_ SustainedWindow, DetectIoAnomalies_..._ASustainedWindowDoes and DetectWaitAnomalies_SustainedHeavyWindow still pinned the old whole-window mean (71.25, 41.25, 1606.25). Each fixture's 16 samples at a 15-minute step span 4 hour tiles of 4 samples each; the hot sample (index 7) falls in the second tile alongside 3 base-value neighbors, so that tile's own mean is what fires (75, 45, 1925). Re-pinned the same way #4172's SQL Server twin was (ee85834): derive the tile's sample count and mean from the fixture's own seeding loop, not a hardcoded number, so a change to the fixture shape keeps the pin honest. The wait profile test's downstream mean_modified_z assertion used the same stale literal and is updated to use the derived tile mean.
Closed
39 tasks done
Owner
Author
T4172-4 lane report (tests-only, branch
|
…tors onto per-hour tiles Part of #3653. CPU: DetectCpuAnomalies reads CpuTileWindowSql (one row per target-local hour), scores every tile through AnomalyGate.EvaluateTiles, and falls back to today's whole-window EvaluateZScore when no tile scores (never-blind rule). Waits: per coordinator ruling, only the robust-z start-bucket arm moves onto EvaluateTiles; the classical-ratio and no-baseline arms stay whole-window, fed from WindowTiles.WholeWindow(tiles), with no second SQL read. total_wait_ms is summed from the tiled rows since it isn't a WindowTile field. Added CpuTileWindowSql, WaitRateTileWindowSql, and shared BindTiledWindow / ReadTilesAsync helpers for reuse by lane L2b (batch, sessions, query duration, memory) in the same file. I/O family and test-pin updates are NOT yet in this commit; see the PR body.
…per-hour tiles Part of #3653. ONE tiled read (IoTileWindowSql) feeds both the read-latency and write-latency gates, each through its own WindowTile list and its own EvaluateTiles call (each family's fallback re-fetches its own bucket map lookup, since the tile key is shared but the gate call and floor are per-direction). Never-blind fallback to today's whole-window EvaluateZScore per direction when no tile scores.
…, delete unused old consts - CpuWindowSql, WaitRateWindowSql, IoWindowSql were dead once L2a's tile move landed (no remaining production reader); deleted, with cref updates on the two class-doc comments that named them. - DarlingAnomalyBaselineTests: census list, the I/O peak/mean pair pin, and the wait-rate pair pin now point at the Tile consts. The wait-rate pin's stale reader-ordinal regexes (which assumed the old single-collapsed-row read) are replaced with a direct pin on the new whole.Peak/whole.Mean/ totalReader.IsDBNull(3) shape; Lite's own single-row ordinals are pinned unchanged. The EvaluateZScore baseline-argument capture widened to catch readBaseline/writeBaseline (the tiled I/O arms' own locals), and the never-blind fallback's window: argument now accepts either window (PG) or the inline TimeRangeEnd - TimeRangeStart (Lite, unmoved). - LatestCpuReadShapeTests' two crefs already point at the still-live CpuTileWindowSql text (unchanged, no fix needed there). Verified: DarlingAnomalyBaselineTests, LatestCpuReadShapeTests, LocalClockBucketKeyTests, MeasurementContractCensusTests, CollectorMeasurementSeamTests, DocCommentHygieneTests all green, no PG.
…ts mean over the hour holding the hot row, located by tile_start_ticks), not the whole-window mean
…/6 red, needs fixing)
CpuFloorPct is 50.0%. The prior fixture (mu=20, sigmaAmp=2) put mu+6sigma around 29 -- never clears the floor, so nothing fired in any of the 3 CPU shift/24h/fallback scenarios. Raised to mu=30, sigmaAmp=5 (sample stddev ~3.69, mu+6sigma ~52.1). Also corrected two assertions that expected the classical DefaultDeviationThreshold (2.0): this CPU baseline carries a positive robust sigma, so the tiled gate runs the modified-z frame (ModifiedZThreshold, 3.5) on both the peak and the tile-corrected mean, not the classical one. CPU: 3 of 4 scenarios now pass (shift, lone spike, fallback). The 24h as_of scenario and both wait/IO scenario-1 tests are still red; next commit.
…t a RED product bug in waits 24h scenario: the window spans two calendar days (Tue 12:00 - Wed 12:00). SeedCpuBaselineAsync anchors every hour it's given to ONE calendar date, so seeding all 24 hours off windowStart's date alone left every Wednesday-side tile bucket empty. Now seeds each hour off the day it actually falls on in the window. Passes. Waits scenario 1: RED, documented rather than bent. DetectWaitAnomalies never writes fire_threshold to the fact's metadata on any arm, unlike every sibling tiled family (CPU, I/O) which stamp it from decision.ThresholdUsed. The fact still fires correctly (right worst tile, tiles_scored/tiles_fired) -- asserted -- but the missing key is a product gap, described in the PR body's Behaviour tests section per the lane rule (a red test after true preconditions is a product bug, not something to bend the assertion around). I/O scenario 1 is still red; not yet diagnosed (context budget). Left for follow-up, described in the PR body.
…nator ruling) The SQL Server-store waits detector never wrote a fire_threshold key on ANY arm, while every sibling tiled family (CPU, I/O) does. Added: - robust-z arm: decision.ThresholdUsed (the tiled/whole-window cutoff actually applied) - ratio arm: DefaultRatioThreshold - no-baseline arm: 0 Flipped Waits_TwoHourSustainedShift_FiresOnWorstTile_... from asserting the key's absence to asserting its value (AnomalyThresholds.HeavyTailModifiedZThreshold on this scenario, since the robust-z tiled arm fires). Live run (DARLING_TEST_PG, timescale/timescaledb:2.28.1-pg18): SqlServerStoreTileBehaviourTests: Total 6, Failed 1 (Io_TwoHourSustainedShift_..., still being diagnosed per step 2), Skipped 0.
…tile map's lookup key Diagnosed: GetBaselineAsync(serverId, IoLatency, T.AddHours(-24), ...) missed the seeded Wednesday 10/11 Full bucket (T-24h is a Tuesday) and fell through to a pooled HourOnly bucket, an under-sized/untrustworthy sigma that mis-sized the shift. Reading at T (the tile map's own lookup key) now reports the seeded Full bucket and Assert.True(IsTrustworthy) passes, but the detector still emits no ANOMALY_READ_LATENCY fact. Not yet green -- see PR body handoff for the next diagnostic step (dump IoTileWindowSql's tile rows + EvaluateTiles verdict, per lane brief step 1's escape hatch).
Same fix as feat/3653-b-l3a's 584ace6, applied here since #4171 has not merged yet: CommandCapture.IsBaselineRead matched any SQL containing BaselineLocalClock.LocalCollectionTimeSql. B's tiled window reads (this PR: CPU, waits, I/O read and write) key their tiles through WindowTiles.LocalHourSql, which wraps that same spelling in date_trunc('hour', ...), so every tiled window read was miscounted as a baseline compute regardless of cache state. Excludes any SQL containing WindowTiles.LocalHourSql from the classifier. No baseline arm on dev uses date_trunc('hour', <LocalCollectionTimeSql>); only WindowTiles.LocalHourSql does.
erikdarlingdata
force-pushed
the
feat/3653-b-l2a
branch
from
September 24, 2026 18:38
b233b23 to
322036d
Compare
erikdarlingdata
changed the title
erikdarlingdata
marked this pull request as ready for review
September 24, 2026 18:39
erikdarlingdata
enabled auto-merge (squash)
September 24, 2026 18:39
erikdarlingdata
added a commit
that referenced
this pull request
Sep 24, 2026
…es_total Under tiles, window_samples reports the worst hour tile's count (12, one hour at the 5-min sampler cadence), not the whole window's; the whole window's count moved to window_samples_total (48). The test's "exact source wrote the window" comment is about exact_collections (sampled vs exact), unrelated to this. Coordinator's diagnosis (lesson 3, the same class as #4172/#4178's earlier re-pins).
erikdarlingdata
added a commit
that referenced
this pull request
Sep 24, 2026
…it profile #3653 A8 option B's tile gate reports the WORST HOUR TILE's mean, not the whole window's, once a detector moves onto tiles. Three of DetectCpuAnomalies_ SustainedWindow, DetectIoAnomalies_..._ASustainedWindowDoes and DetectWaitAnomalies_SustainedHeavyWindow still pinned the old whole-window mean (71.25, 41.25, 1606.25). Each fixture's 16 samples at a 15-minute step span 4 hour tiles of 4 samples each; the hot sample (index 7) falls in the second tile alongside 3 base-value neighbors, so that tile's own mean is what fires (75, 45, 1925). Re-pinned the same way #4172's SQL Server twin was (ee85834): derive the tile's sample count and mean from the fixture's own seeding loop, not a hardcoded number, so a change to the fixture shape keeps the pin honest. The wait profile test's downstream mean_modified_z assertion used the same stale literal and is updated to use the derived tile mean.
erikdarlingdata
added a commit
that referenced
this pull request
Sep 24, 2026
… tiles (#4178) * 3653 B L3c step 1: PG-target WAL volume moves onto per-hour tiles * 3653 B L3c step 2: PG-target CPU burn moves onto per-hour tiles * 3653 B L3c step 3: PG-target sampled waits' robust arm moves onto per-hour tiles * issue-3653 A8 option B (lane L3c-2): re-pin the WAL/CPU-burn/sampled-waits fallback and census after the tile move * issue-3653 A8 option B (lane L3c-2): bind $4..$6 on the CPU-burn tile read (lesson 1 — WindowCommand only binds $1..$3, so this family silently never fired live) * 3653 B L3c-3: fix CpuBurnTileWindowSql ungrouped-column subquery, re-pin kernel worst-tile window_samples * 3653 B L3c-3: re-pin WAL worst-tile window_samples * issue-3653 A8 option B T4178 (WIP, RED): draft tile-behaviour tests for WAL/CPU-burn/sampled-waits — WAL scenario 1 still failing, needs plant-data debugging * 3653 B T4178-2: seed the pg_database_stats canary row in the tile-behaviour tests DetectAnomaliesAsync gates the entire detector run on HasBaselineDataAsync (a canary read of pg_database_stats within 30 days of the window end). None of the six tests in this file called the already-written PlantCanaryDatabaseStatsRowAsync helper, so every detector — including the one under test — was skipped and DetectAnomaliesAsync returned zero facts. WalVolume_TwoHourShiftInFourHourWindow now fires as designed; printed evidence (buckets trustworthy, tiles correctly split at hours 12-13, WAL rate correctly differenced) is left in as diagnostic Console.WriteLine calls for the PR body. * 3653 B T4178-2: fix CPU-burn/sampled-waits scenario-1 shift magnitudes Both fixtures planted a shift far larger than 6\u03c3, so the dev-comparison assertion (whole-window mean z must stay BELOW the cutoff) failed: the shift was so large it cleared the cutoff even diluted across the whole window, defeating the point of the scenario. Sized both shifts from the bucket actually read (printed): CPU burn to +30% (~6\u03c3 on its ripple-derived sigma, same shape as the WAL family); sampled waits to 30 relation units (~6\u03c3 on its measured robust sigma). Both scenario-1 tests pass live now. * 3653 B T4178-2: drop the shift tests' in-test dev-comparison assertion per coordinator ruling The mutation check (AnomalyGate.EvaluateTiles -> null, forcing the never-blind fallback to the whole-window path) already proves 'the whole-window path would have stayed quiet' for every shift test, so the in-test wholeWindowMeanZ < k assertion in WAL 4h / CPU burn / sampled waits was redundant and kept forcing shift-magnitude tuning that had nothing to do with the tile behaviour under test. Removed all three; the mutation check re-run confirms all four shift tests (WAL 4h, WAL 24h, CPU burn, sampled waits) fail when EvaluateTiles is forced null, and the two non-firing tests (lone-spike, sparse-fallback) still pass under the same mutation, as expected. * 3653 B L3c: drop the 7 leftover diagnostic Console.WriteLine lines Debug prints left in from developing PgTargetTileBehaviourWalKernelSampledTests.cs. Assertion messages are untouched. * 3653 B L3c: re-pin window_samples to the worst tile, add window_samples_total Under tiles, window_samples reports the worst hour tile's count (12, one hour at the 5-min sampler cadence), not the whole window's; the whole window's count moved to window_samples_total (48). The test's "exact source wrote the window" comment is about exact_collections (sampled vs exact), unrelated to this. Coordinator's diagnosis (lesson 3, the same class as #4172/#4178's earlier re-pins). * 3653 B L3c: assert window_samples is alignment-proof, not an exact 12 Coordinator's diagnosis: the stock scenario is uniformly heavy (peak == mean already asserted above), so every hour tile ties on Sigma. EvaluateTiles breaks a tie to the LATER local hour, so the worst tile is the newest one. windowStart is anchored on wall-clock now minus 4 hours, not hour-aligned, so that newest tile is a partial hour whose sample count depends on the minute the test runs -- 12 was never a stable number, and the product behaviour (report the most recent hour of a whole-window problem) is correct. Replaced the exact pin with Assert.InRange(MinTileSamples, 12), and added tiles_scored == tiles_fired (every scored hour is uniformly heavy too, so every one fires). Neither tile_local_hour nor tile_start_ticks is pinned in this test, so nothing to relax there.
erikdarlingdata
added a commit
that referenced
this pull request
Sep 24, 2026
… Lite CPU, waits and I/O anomaly detectors move onto per-hour tiles (#4169) * 3653 B L4a: Lite CPU anomaly detector moves onto per-hour tiles DetectCpuAnomalies groups the DuckDB window read by target-local hour (WindowTiles.LocalHourSql), reads one WindowTile per row via WindowTiles.ReadTile, and scores the tiles through AnomalyGate.EvaluateTiles against BaselineBucketMap. arg_max replaces the ORDER BY ... LIMIT 1 peak-time subquery. When no tile scores it falls back to today's whole-window EvaluateZScore against the start bucket (never-blind rule). WindowTiles.AddTileMetadata stamps the new tile_* / window_* keys when a tile verdict was used. Waits and I/O families are not yet converted; see PR body for the handoff. * 3653 B L4a-2 step 1: Lite wait-profile detector's robust-z arm moves onto tiles DetectWaitAnomalies keeps the arm choice on the START bucket exactly as today. Only the trusted robust-z arm (EffectiveRobustSigma > 0) now scores AnomalyGate.EvaluateTiles against one row per target-local hour (the per_collection CTE is kept, grouped a second time by the outer select on WindowTiles.LocalHourSql); when no tile scores, it falls back to today's whole-window EvaluateZScore. The classical-ratio and no-baseline arms stay whole-window and unchanged, fed from WindowTiles.WholeWindow(tiles)'s Peak/Mean — no second SQL read, per the coordinator's wait-family ruling. modified_z / mean_modified_z / ratio / current_ms_per_sec and the baseline keys now come from the bucket and values actually scored on each arm. * 3653 B L4a-2 step 2: Lite I/O anomaly detectors move onto per-hour tiles DetectIoAnomalies reads one row per target-local hour tile, carrying read-latency and write-latency PEAK/MEAN/sample-count per tile in the same query (WindowTiles.LocalHourSql, $4..$6 bound from map.WindowClock). Each of the two families (read, write) builds its own WindowTile list and scores it independently through AnomalyGate.EvaluateTiles with today's thresholds; when EvaluateTiles returns null for a family, that family falls back to today's whole-window EvaluateZScore against the start bucket (never-blind rule). Metadata comes from the bucket and tile actually scored. * 3653 B L4a: re-pin the tile-gate's worst-hour mean on CPU, I/O and wait profile #3653 A8 option B's tile gate reports the WORST HOUR TILE's mean, not the whole window's, once a detector moves onto tiles. Three of DetectCpuAnomalies_ SustainedWindow, DetectIoAnomalies_..._ASustainedWindowDoes and DetectWaitAnomalies_SustainedHeavyWindow still pinned the old whole-window mean (71.25, 41.25, 1606.25). Each fixture's 16 samples at a 15-minute step span 4 hour tiles of 4 samples each; the hot sample (index 7) falls in the second tile alongside 3 base-value neighbors, so that tile's own mean is what fires (75, 45, 1925). Re-pinned the same way #4172's SQL Server twin was (ee85834): derive the tile's sample count and mean from the fixture's own seeding loop, not a hardcoded number, so a change to the fixture shape keeps the pin honest. The wait profile test's downstream mean_modified_z assertion used the same stale literal and is updated to use the derived tile mean. * issue-3653 A8 option B: Lite tile-gate behaviour tests (scenario 1: CPU, waits, I/O) * issue-3653 A8 option B: drop the in-test dev-comparison, keep the fire+metadata assertions (coordinator ruling) * issue-3653 A8 option B: stamp fire_threshold on Lite wait anomaly facts Mirrors Darling's PgAnomalyDetector.cs fix: DetectWaitAnomalies writes fire_threshold from decision.ThresholdUsed on the tiled robust arm and its EvaluateZScore fallback, DefaultRatioThreshold on the ratio arm, and 0 on the no-baseline arm. Same local-variable name and shape as Darling so the *LiteVerbatim parity pins keep comparing the text. * issue-3653 A8 option B: print I/O tile evidence to CI (lane T4169-3) The I/O scenario fired in CI round 8875188 and did not fire in round d5370df for a reason not yet known. Add ITestOutputHelper prints of the bucket map's per-hour tiles (Tier/SampleCount/DistinctDays/ IsTrustworthy/Median/EffectiveRobustSigma), the seeded whole-window mu/sigma/shift, and the fired fact count, so a red CI run's log shows what was actually scored without a local repro (the Lite.Tests host hangs on this Mac). * 3653 B T4169-3 follow-up: drop the red I/O tile behaviour test Deleted ReadLatency_TwoHourShiftInFourHourWindow_FiresPerTile_... (2 rounds red per T4169-3), its IoBaseReads/IoMuMs/IoSigmaMs/IoShiftMs consts, SeedIoBaseline21Days, SeedIoWindow (used only by that test), and the ITestOutputHelper evidence-print field/constructor param it alone used. Lite I/O's tiled fire path is covered by AnomalyDetectorTests. DetectIoAnomalies_...ASustainedWindowDoes instead (#3653 B). Updated the class doc comment to name only CPU and wait-profile, and to point at that test for I/O. Coordinator's scope-trim ruling. * 3653 B (#4169): make WaitRateWindow_..._LiteVerbatim's reader pin symmetric The 3 Lite-only single-row reader asserts (avgRate = rateReader.IsDBNull(1), totalWaitMs = rateReader.IsDBNull(2), collectionCount = rateReader.IsDBNull(3)) pinned a shape Lite left behind when #4169 tiled its wait reader. Replaced with a shared loop asserting the 3 lines that ARE byte-identical between the two detectors now (whole = WindowTiles.WholeWindow(tiles); peakRate = whole.Peak; avgRate = whole.Mean), run against both pg and liteCode. Updated two stale "Lite has not moved onto tiles" comments (the SQL column-shift note and the window: kwarg note) to say both sides tile now. Left the window-total accumulation assertion Darling-only, un-symmetric: the two detectors sum it differently (PG in a second SQL round trip against its own totalReader; Lite in the same pass as the tile read, into windowTotalWaitMs) -- a real structural difference, not a naming one. Coordinator's call on whether to make that symmetric too. * 3653 B (#4169): add the EvaluateTiles call-count and per-family parity check New AnomalyGate_EvaluateTilesCalls_AreFourPerSku_AndMatchBetweenPgAndLite: asserts both PgAnomalyDetector.cs and Lite's AnomalyDetector.cs call AnomalyGate.EvaluateTiles exactly 4 times (cpu, wait, io-read, io-write). CPU and wait's calls are checked byte-identical. I/O's two calls tolerate the same two differences this file already tolerates elsewhere by design: the baseline-map argument name (PG's readMap/writeMap vs Lite's shared map -- same cache key, same value) and the modified-z threshold argument (PG's hoisted local vs Lite's inline ModifiedZThresholdFor call -- same pure function, same inputs), mirroring the existing \w*[Bb]aseline and window/inline-expression tolerances already in this test. * 3653 B (#4169): note why totalWaitMs stays Darling-only in the parity test Coordinator confirmed: both sides sum the same ordinal-3 column over the same WaitRateTileWindowSql rows and get the same value. Lite sums it inside its one tile-read loop; Darling re-reads the tile SQL a second time (totalCmd) to sum it. Equivalent, just one redundant query on Darling's side, so the assertion correctly stays pinned to Darling's own text rather than forced into the shared loop. * 3653 B L4a: conform to dev's L2b/L3c landings mid-flight Two rounds of CI drift while this PR sat armed, both from merges on dev, neither a flake: 1. Lite's wait-rate ratio arm and no-baseline arm recomputed reportPeak/ reportAvg from whole.Peak/whole.Mean instead of reusing peakRate/ avgRate (already the same values, set at the top of the method) -- a naming-only redundancy that broke the LiteVerbatim byte-identical pin against PgAnomalyDetector's text once nothing else masked it. Swapped the ratio/meanRatio and no-baseline-ratio computations onto peakRate/avgRate directly; reportPeak/reportAvg still get assigned for the metadata reads below, unchanged. 2. #4177 (L2b) landed batch/sessions/query/memory tiles on the pg side before #4176 (L4b) mirrors them into Lite, so pg's EvaluateTiles census legitimately runs ahead of Lite's for now. Relaxed the total- count check from == 4 to >= 4 with a comment explaining why; the four per-metric shape assertions below it (this lane's actual scope) are unaffected and still pin byte-for-byte parity on cpu/wait/io-read/ io-write. Both failing assertions verified locally against the built test DLL before push (pure source-text regex checks, no live PG needed).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Behaviour tests (T4172-2, tests-only)
PR #4172's
SqlServerStoreTileBehaviourTests.cs(6 tests) started this lane with 5 of 6 RED becausethe fixtures never cleared the detectors' magnitude floors (
CpuFloorPct= 50%). Status now:Cpu_TwoHourSustainedShift_FiresOnWorstTile_WhereWholeWindowMeanWouldNotClearCutoffCpu_TwoHourShift_In24HourWindow_FiresWithRaisedCutoffCpu_LoneSpike_DoesNotFireCpu_UnderMinTileSamples_FallsBackToWholeWindow_NoTileKeysWaits_TwoHourSustainedShift_FiresOnWorstTile_...Io_TwoHourSustainedShift_FiresOnWorstTile_...Live run (
DARLING_TEST_PG,timescale/timescaledb:2.28.1-pg18, ephemeral container, removed after):What was wrong and what changed
CPU magnitude floor. The original fixture used
CpuMu = 20.0, CpuSigmaAmp = 2.0(sample stddev≈ 1.48), so
mu + 6·sigma ≈ 28.9— nowhere nearAnomalyThresholds.CpuFloorPct(50.0%). No factcould ever fire. Raised to
CpuMu = 30.0, CpuSigmaAmp = 5.0(stddev ≈ 3.69), givingmu + 6·sigma ≈ 52.1— clears the floor with margin.Wrong threshold family in two assertions. Scenarios 1 and 2 asserted
fire_thresholdagainstAnomalyThresholds.DefaultDeviationThreshold(2.0, the classical cutoff). But this CPU baselinecarries a positive robust sigma (MAD from the deterministic sine noise), so
AnomalyGate'strustworthy path runs the modified-z frame (
ModifiedZThreshold= 3.5) on both the peak and thetile-corrected mean, not the classical one. Fixed both assertions (scenario 1's direct check and
scenario 2's
NAwarePeakCutoffbase) to useModifiedZThreshold. Also fixed scenario 1's"what the whole-window gate would have seen" comparison to compute z against
baseline.Median/baseline.EffectiveRobustSigma(the frame the detector actually uses here), notbaseline.Mean/baseline.EffectiveStdDev.24h scenario's seeding gap.
Cpu_TwoHourShift_In24HourWindow_FiresWithRaisedCutoff's window[T-22h, T+2h)spans two calendar days (Tuesday 12:00 through Wednesday 12:00). The originalseeding called
SeedCpuBaselineAsynconce withwindowStartas the anchor and all 24 hours-of-day —but that helper anchors every hour it's given to
anchor's ONE calendar date, so every Wednesday-sidetile's (hour, day-of-week) bucket landed with
SampleCount == 0and the tiled gate skipped it.Fixed by seeding each of the 24 hours off the calendar day it actually falls on inside the window, so
every (hour, day-of-week) bucket the 24 tiles will look up has 21 weeks of matching history behind
it.
RED: a real product gap found in the waits detector (not fixed here — tests-only lane)
PgAnomalyDetector.DetectWaitAnomaliesnever writesfire_thresholdto anANOMALY_WAIT_PROFILEfact's metadata, on any arm. Every sibling tiled family (CPU
DetectCpuAnomalies, I/O read/write inDetectIoAnomalies) stamps["fire_threshold"] = decision.ThresholdUsedright beside their othergate-decision keys; the wait detector's metadata dictionary literal has no such entry at all. The fact
still fires correctly — right worst tile (
tile_local_hour12 or 13), righttiles_scored/tiles_firedcounts, all asserted in the test — this is purely a missing metadata key, described honestly rather
than removed/bent to make the test pass.
Waits_TwoHourSustainedShift_FiresOnWorstTile_...now assertsfact.Metadata.ContainsKey("fire_threshold")is false, with a comment naming the gap, so a futurefix (adding the stamp) will need to flip that one assertion — a clear signal, not a silent regression.
Still red, not diagnosed:
Io_TwoHourSustainedShift_...Fires nothing (
Assert.Singlesees an empty collection). The seeding follows the same shape as thesibling live
EndToEnd_IoDetector_...test and readsbaseline.Mean/baseline.EffectiveRobustSigmafrom a live
GetBaselineAsynccall before computing the shift (so it should track whatever the realbaseline reports, unlike the pre-lane CPU bug), but something about the 21-Wednesday
file_io_statsseeding or the
IoTileWindowSqltile lookup is not landing the way the CPU/waits fixtures did. Leftundiagnosed — flagging for the next lane rather than guessing further under this lane's context
budget. Suggested next step: dump
provider.GetBucketMapAsync/ the rawIoTileWindowSqlrows forserverId = -4172_06right after seeding, the way the CPU fixes above eventually needed to.Mutation check — NOT RUN
Ran out of context budget before reaching this step (
AnomalyGate.EvaluateTilesforced to return null;CPU scenario 1 should fail, lone-spike should still pass; then revert/rebuild/re-verify). Not done in
this lane.
Full suite — NOT RUN
Also not reached in this lane's budget. Only the
SqlServerStoreTileBehaviourTestsclass was run live.What I verified
-p:EnableWindowsTargeting=true.scrubcheckon the touched file: 0 hits, exit 0.timescale/timescaledb:2.28.1-pg18on an ephemeral container (removed after), withDARLING_TEST_PGset:Total: 6, Errors: 0, Failed: 1, Skipped: 0(only the I/O scenario red).Not verified / deferred
AnomalyGate.EvaluateTiles-> null).DARLING_TEST_PGunset (the 214-Windows-only-failures baseline diff).CHANGELOG entry
(none — tests-only, no product behaviour change)
T4172-3 update (fire_threshold fixed; I/O still red; mutation check + full suite NOT run — context wall)
Step 1 — DONE, committed (4851077), pushed to feat/3653-b-l2a
PgAnomalyDetector.DetectWaitAnomaliesnow stampsfire_thresholdon every arm, matching the PG-targettwin's convention:
decision.ThresholdUsedDefaultRatioThreshold0Waits_TwoHourSustainedShift_FiresOnWorstTile_...now asserts the key's VALUE(
AnomalyThresholds.HeavyTailModifiedZThreshold, since this scenario's robust-z tiled arm fires) instead ofasserting it was absent. Grepped
MeasurementContractCensusTests/CollectorMeasurementSeamTestsfor apinned waits metadata key SET — neither file references
ANOMALY_WAIT_PROFILEorfire_threshold, sonothing else needed updating.
Live run (
DARLING_TEST_PG,timescale/timescaledb:2.28.1-pg18, ephemeral container, removed after):The one failure is
Io_TwoHourSustainedShift_...(step 2, below) — everything else, including the flippedwaits assertion, is green.
Step 2 — CLOSED (coordinator ruling, T4172-5: DELETE the fixture)
Four lanes could not make
Io_TwoHourSustainedShift_...fire. The coordinator ruled this out of scope(PR comment 5819831541): the tiled I/O fire path is already covered live by
DarlingAnomalyBaselineTests.EndToEnd_IoDetector_..., and the silent-detector guard (#4179) covers I/OSQL errors separately.
Io_TwoHourSustainedShift_FiresOnWorstTile_...and its only-used helper(
CleanupIoAsync) were deleted fromSqlServerStoreTileBehaviourTests.cs; a one-line pointer to thecovering E2E test and the drop rationale was added to the class doc comment. Commit b233b23.
Step 2b — Mutation check (T4172-5)
Made
AnomalyGate.EvaluateTilesreturnnullunconditionally (first line of the method body), rebuilt,and ran
*SqlServerStoreTileBehaviour*live:Cpu_TwoHourSustainedShift_FiresOnWorstTile_...FAILED as required (along withCpu_TwoHourShift_In24HourWindow_...andWaits_TwoHourSustainedShift_...— all three depend on thetiled arm actually running). Reverted the mutation, rebuilt with
--no-incremental(confirmedgit diffonAnomalyGate.cswas clean before rebuilding), re-ran the same class:All 5 remaining behaviour tests pass live once the gate is restored — the mutation check holds.
Step 3 — Live classes (T4172-5)
All run against
timescale/timescaledb:2.28.1-pg18on an ephemeral container (removed after):SqlServerStoreTileBehaviourTestsDarlingAnomalyBaselineTestsAnomalyGateTilesTestsAnomalyGateNullWindowMonteCarloTests(
AnomalyGateTestsdoes not exist as a class name;*AnomalyGate*resolves to the two classes above,both run.)
Step 4 — Full suite (T4172-5)
DARLING_TEST_PGunset, fullDarling.Tests.dllrun:214 failed matches the documented Mac-vs-Windows-CI base exactly (all Windows-only/WPF/registry-shaped
failures unrelated to this lane). No new failures introduced.
Final state
SqlServerStoreTileBehaviourTests.csnow has 5 tests, all green live: 4 CPU scenarios plus 1 waitsscenario. The I/O tiled-shift scenario is retired in favor of existing E2E coverage. The tiled-vs-whole-
window gate (
AnomalyGate.EvaluateTiles) is proven load-bearing by the mutation check. The full suite'sfailure count is unchanged from the documented baseline.
CHANGELOG entry
ANOMALY_WAIT_PROFILE) now stamps afire_thresholdvalue in its fact metadata on every gate arm, matching every sibling anomaly family.