Repository navigation
Make the PgTarget anomaly and blocking tests independent of the wall clock (#4274) - #4323
Merged
Merged
Conversation
… wall clock (#4274) Both tests derived their analysis window from a raw DateTime.UtcNow minute, so the target-local-hour tile holding the seeded spike/chain varied in size run to run. AnomalyGate.EvaluateTiles skips any tile under MinTileSamples (3), so a tile landing at 1-2 samples got excluded and the wrong tile won "worst-scoring", misreporting the peak or dragging the mean below the pinned assertion range. Anchoring the window end to the hour (:59 of the prior hour, always) keeps every scored tile a safe size regardless of wall time. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
erikdarlingdata
marked this pull request as ready for review
September 25, 2026 17:07
erikdarlingdata
enabled auto-merge (squash)
September 25, 2026 17:07
erikdarlingdata
added a commit
that referenced
this pull request
Sep 25, 2026
Picks up #4323's wall-clock fix for the PgTarget anomaly test that failed this PR's CI three times. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FVjn4PBJN71NQXdFo6ZxNQ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4274.
Why
Both tests derived their analysis window from a raw
DateTime.UtcNowminute(
TruncateToMinutes(DateTime.UtcNow).AddMinutes(-1)).AnomalyGate.EvaluateTiles(
PerformanceMonitor.Analysis/AnomalyGate.cs) skips any target-local-hour tile underAnomalyThresholds.MinTileSamples(3) before it is even scored (design: #3653 A8 option B). Depending onthe wall clock at test-run time, the tile holding the seeded spike (anomaly test) or the window's sole
zero-blocked minute (blocking test) could land at 1-2 samples, get excluded, and let a different tile "win"
worst-scoring — misreporting the peak, or dragging the mean below a pinned assertion range.
Diagnosis, confirmed with numbers
Added a test-only clock seam (
DARLING_TEST_NOW_UTC, read by a newSimulatedUtcNow()helper in eachfile; unset in every normal run, CI included) so the exact CI failures could be reproduced on demand
against a real dev Postgres, instead of waiting for the wall clock to land on a bad minute.
Anomaly test,
TheAuroraWaitProfile_..._AgainstDevPostgres, old formula, anchor minute :02 (spike'sown tile sized 2, under the floor of 3):
Exact match to the two CI failures in #4274. At anchor minute :31 (tile sized 31, safely over the floor)
the same old formula passed.
Blocking test,
ThirtyOneDaysOfLightBlocking_..._AndTheAnomalyInOneIncident, old formula, anchorminute :49 (
windowStart's own tile — the one holding the window's sole zero-blocked minute — sized60 - 49 = 11):
Exact match to the CI failure in #4274, digit for digit. At anchor minute :21 (tile sized 39) the same old
formula passed.
The under-report, not acted on: at the anomaly test's bad anchor, before reverting to the fix, I
probed the fact's other metadata.
current_ms_per_secread 1500 (wrong), butwindow_peakread 3200(right) and
window_samples(of the winning, wrong tile) read 60 —WindowTiles.WholeWindowsums theunfiltered tile list, so the true peak survives in a less prominent field even when the headline number
doesn't. This is real product behavior under #3653 A8 option B's design (a too-small edge tile never gets
scored, full stop), not a test bug, so
AnomalyGateandMinTileSamplesare untouched here. Posted thenumbers as a question for the coordinator on #4274 rather than changing product code.
What changes
Both files get the same three additions, plus one anchor-line swap and one refreshed comment:
TruncateToHour(DateTime value)— truncates to the top of the hour (mirrors the existingTruncateToMinutesalready in both files).SimulatedUtcNow()— returnsDateTime.UtcNow, or an ISO-8601 parse ofDARLING_TEST_NOW_UTCwhenthat env var is set. The test-only clock seam the brief asked for.
AnchorEndUtc()—TruncateToHour(SimulatedUtcNow()).AddMinutes(-1). Always :59 of the PRIOR hour,regardless of what minute
SimulatedUtcNow()returns, so the tile holding the seed is never undersized.TruncateToMinutes(DateTime.UtcNow).AddMinutes(-1)anchor lines now callAnchorEndUtc().replaced with the current mechanism and the now-deterministic value (see the table below).
Assert.InRange(anomaly.Metadata["avg_blocked_sessions"], 2.8, 3.0)itself is untouched — not widened,not tightened — because the range predates this fix and 3.0 sits inside it.
No product code changes. No CHANGELOG entry (test-only).
Two other tests in these same files (
PgTargetBlockingTests.cslines 913 and 1125, in a different methodthan the one fixed here) share the same raw-minute
TruncateToMinutes(DateTime.UtcNow).AddMinutes(-1))anchor pattern but were not named in #4274 and were not observed failing — left alone, in scope for a
follow-up if they ever flake.
Test plan
Rig: Postgres 55972 in
C:\GitHub\worktrees\rig-fx2(fresh extract ofpg-runtime.zip),timezone/log_timezonepinned to UTC,timescaledb.max_background_workers/max_worker_processesat CI's 74/85.Proof matrix, both tests, final (fixed) build,
AnchorEndUtc()driven byDARLING_TEST_NOW_UTC:DARLING_TEST_NOW_UTC)windowEndafterAnchorEndUtc()(An eighth anchor,
2026-09-26T02:15:00Z, was tried first for the midnight case and produced an unrelated"invalid"analyze_serverstatus — that anchor was in the future relative to the real clock, and theservice correctly rejects an
as_ofin the future. Not a tile bug; swapped for the past-dated midnightanchor above.)
Probed the fixed formula's
avg_blocked_sessionsdirectly (temporary forced-mismatch assertion, reverted)at two of the anchors above: both read exactly
3.0.windowStart's own tile (the one that used to dragthe mean down) is now always sized 1 — under the floor of 3 — so it never scores, and every tile that does
score is a clean 3.0.
dotnet build Darling/Darling.Tests/Darling.Tests.csproj -c Debug— Build succeeded, 0 Warning(s), 0 Error(s).git merge origin/dev— clean, no conflicts.Darling.Tests.exe, freshdarlingtest, real clock (noDARLING_TEST_NOW_UTC), foreground, once:Total: 14070, Errors: 0, Failed: 0, Skipped: 49, Not Run: 1, Time: 731.212s. The 49 skips and 1 not-run are pre-existing (gated live tests needing other env vars / runtimes this rig didn't set up) and unrelated to the two files this PR touches.pg_ctl stop) after the run.Coordinator check
current_ms_per_sec/window_peakunder-report as a question, not aruling — please confirm whether it needs a follow-up issue or is accepted design.
PgTargetBlockingTests.cs:913and:1125(out of scope here) areworth a follow-up if they start flaking too.
🤖 Generated with Claude Code
https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3