Skip to content

Make the PgTarget anomaly and blocking tests independent of the wall clock (#4274) - #4323

Merged
erikdarlingdata merged 2 commits into
devfrom
fix/4274-pgtarget-tile-flake
Sep 25, 2026
Merged

erikdarlingdata merged 2 commits into
devfrom
fix/4274-pgtarget-tile-flake

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Closes #4274.

Why

Both tests derived their analysis window from a raw DateTime.UtcNow minute
(TruncateToMinutes(DateTime.UtcNow).AddMinutes(-1)). AnomalyGate.EvaluateTiles
(PerformanceMonitor.Analysis/AnomalyGate.cs) skips any target-local-hour tile under
AnomalyThresholds.MinTileSamples (3) before it is even scored (design: #3653 A8 option B). Depending on
the wall clock at test-run time, the tile holding the seeded spike (anomaly test) or the window's sole
zero-blocked minute (blocking test) could land at 1-2 samples, get excluded, and let a different tile "win"
worst-scoring — misreporting the peak, or dragging the mean below a pinned assertion range.

Diagnosis, confirmed with numbers

Added a test-only clock seam (DARLING_TEST_NOW_UTC, read by a new SimulatedUtcNow() helper in each
file; unset in every normal run, CI included) so the exact CI failures could be reproduced on demand
against a real dev Postgres, instead of waiting for the wall clock to land on a bad minute.

Anomaly test, TheAuroraWaitProfile_..._AgainstDevPostgres, old formula, anchor minute :02 (spike's
own tile sized 2, under the floor of 3):

Assert.Equal() Failure: Values are not within 3 decimal places
Expected: 3200 (rounded from 3200)
Actual:   1500 (rounded from 1500)

Exact match to the two CI failures in #4274. At anchor minute :31 (tile sized 31, safely over the floor)
the same old formula passed.

Blocking test, ThirtyOneDaysOfLightBlocking_..._AndTheAnomalyInOneIncident, old formula, anchor
minute :49 (windowStart's own tile — the one holding the window's sole zero-blocked minute — sized
60 - 49 = 11):

Assert.InRange() Failure: Value not in range
Range:  (2.7999999999999998 - 3)
Actual: 2.7272727272727271

Exact match to the CI failure in #4274, digit for digit. At anchor minute :21 (tile sized 39) the same old
formula passed.

The under-report, not acted on: at the anomaly test's bad anchor, before reverting to the fix, I
probed the fact's other metadata. current_ms_per_sec read 1500 (wrong), but window_peak read 3200
(right) and window_samples (of the winning, wrong tile) read 60 — WindowTiles.WholeWindow sums the
unfiltered tile list, so the true peak survives in a less prominent field even when the headline number
doesn't. This is real product behavior under #3653 A8 option B's design (a too-small edge tile never gets
scored, full stop), not a test bug, so AnomalyGate and MinTileSamples are untouched here. Posted the
numbers as a question for the coordinator on #4274 rather than changing product code.

What changes

Both files get the same three additions, plus one anchor-line swap and one refreshed comment:

  • TruncateToHour(DateTime value) — truncates to the top of the hour (mirrors the existing
    TruncateToMinutes already in both files).
  • SimulatedUtcNow() — returns DateTime.UtcNow, or an ISO-8601 parse of DARLING_TEST_NOW_UTC when
    that env var is set. The test-only clock seam the brief asked for.
  • AnchorEndUtc() — TruncateToHour(SimulatedUtcNow()).AddMinutes(-1). Always :59 of the PRIOR hour,
    regardless of what minute SimulatedUtcNow() returns, so the tile holding the seed is never undersized.
  • The two TruncateToMinutes(DateTime.UtcNow).AddMinutes(-1) anchor lines now call AnchorEndUtc().
  • The blocking test's stale comment ("was pinned 2.9-3.0; the worst tile's hour reads 2.8333...") is
    replaced with the current mechanism and the now-deterministic value (see the table below).
    Assert.InRange(anomaly.Metadata["avg_blocked_sessions"], 2.8, 3.0) itself is untouched — not widened,
    not tightened — because the range predates this fix and 3.0 sits inside it.

No product code changes. No CHANGELOG entry (test-only).

Two other tests in these same files (PgTargetBlockingTests.cs lines 913 and 1125, in a different method
than the one fixed here) share the same raw-minute TruncateToMinutes(DateTime.UtcNow).AddMinutes(-1))
anchor pattern but were not named in #4274 and were not observed failing — left alone, in scope for a
follow-up if they ever flake.

Test plan

Rig: Postgres 55972 in C:\GitHub\worktrees\rig-fx2 (fresh extract of pg-runtime.zip), timezone/
log_timezone pinned to UTC, timescaledb.max_background_workers/max_worker_processes at CI's 74/85.

Proof matrix, both tests, final (fixed) build, AnchorEndUtc() driven by DARLING_TEST_NOW_UTC:

Anchor (DARLING_TEST_NOW_UTC) windowEnd after AnchorEndUtc() Anomaly test Blocking test
2026-09-25T10:00:30Z (minute :00) 09:59:00 pass pass
2026-09-25T10:01:15Z (minute :01) 09:59:00 pass pass
2026-09-25T10:15:00Z (minute :15) 09:59:00 pass pass
2026-09-25T10:30:00Z (minute :30) 09:59:00 pass pass
2026-09-25T10:45:00Z (minute :45) 09:59:00 pass pass
2026-09-25T10:59:59Z (minute :59) 09:59:00 pass pass
2026-09-25T02:15:00Z (crosses UTC midnight: window runs 2026-09-24 21:59 to 2026-09-25 01:59) 2026-09-25T01:59:00 pass pass

(An eighth anchor, 2026-09-26T02:15:00Z, was tried first for the midnight case and produced an unrelated
"invalid" analyze_server status — that anchor was in the future relative to the real clock, and the
service correctly rejects an as_of in the future. Not a tile bug; swapped for the past-dated midnight
anchor above.)

Probed the fixed formula's avg_blocked_sessions directly (temporary forced-mismatch assertion, reverted)
at two of the anchors above: both read exactly 3.0. windowStart's own tile (the one that used to drag
the mean down) is now always sized 1 — under the floor of 3 — so it never scores, and every tile that does
score is a clean 3.0.

  • dotnet build Darling/Darling.Tests/Darling.Tests.csproj -c Debug — Build succeeded, 0 Warning(s), 0 Error(s).
  • git merge origin/dev — clean, no conflicts.
  • Full Darling.Tests.exe, fresh darlingtest, real clock (no DARLING_TEST_NOW_UTC), foreground, once: Total: 14070, Errors: 0, Failed: 0, Skipped: 49, Not Run: 1, Time: 731.212s. The 49 skips and 1 not-run are pre-existing (gated live tests needing other env vars / runtimes this rig didn't set up) and unrelated to the two files this PR touches.
  • Rig stopped (pg_ctl stop) after the run.

Coordinator check

🤖 Generated with Claude Code

https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

erikdarlingdata and others added 2 commits September 25, 2026 12:49
… wall clock (#4274)

Both tests derived their analysis window from a raw DateTime.UtcNow minute,
so the target-local-hour tile holding the seeded spike/chain varied in size
run to run. AnomalyGate.EvaluateTiles skips any tile under MinTileSamples
(3), so a tile landing at 1-2 samples got excluded and the wrong tile won
"worst-scoring", misreporting the peak or dragging the mean below the
pinned assertion range. Anchoring the window end to the hour (:59 of the
prior hour, always) keeps every scored tile a safe size regardless of wall
time.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 25, 2026 17:07
@erikdarlingdata
erikdarlingdata enabled auto-merge (squash) September 25, 2026 17:07
@erikdarlingdata
erikdarlingdata merged commit db75613 into dev Sep 25, 2026
15 of 16 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/4274-pgtarget-tile-flake branch September 25, 2026 17:25
erikdarlingdata added a commit that referenced this pull request Sep 25, 2026
Picks up #4323's wall-clock fix for the PgTarget anomaly test that failed this PR's CI three times.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FVjn4PBJN71NQXdFo6ZxNQ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant