Skip to content

Flaky test: the failure-facts reader always reads once before checking the writers (#4312) - #4335

Merged
erikdarlingdata merged 2 commits into
devfrom
fix/4312-alert-read-failure-reader-runs-once
Sep 25, 2026
Merged

erikdarlingdata merged 2 commits into
devfrom
fix/4312-alert-read-failure-reader-runs-once

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Closes #4312.

Why

TheNewestFailuresFactsAreNeverABlendOfTwo polls the writer tasks with a while (!writers.All(w => w.IsCompleted)) loop and counts each iteration in observations, then asserts observations > 0 after the writers finish. A while loop checks the exit condition before the body ever runs. On a fast run, or a slow test thread, all five writer tasks can finish (WritesPerWriter = 200_000 each) before the test thread's first check of writers.All(...), so the loop body never executes, observations stays 0, and the "the reader observed nothing" assert fails. That is the flake filed as #4312.

The same shape exists in UnderConcurrency_TheInstanceTotalNeverTrailsItsParts a few hundred lines above: a while loop over its own two writer tasks, counting observations, with the same Assert.True(observations > 0, ...) after the loop. It has the identical race and was fixed in this PR too, per the brief's instruction to fix every loop of this shape in one pass. A full grep of both test projects (git grep -n "while (!.*IsCompleted" -- Darling/Darling.Tests Lite.Tests, plus a broader while (! sweep) found only these two occurrences; no other writer-polling loop of this kind exists in either project.

What changes

In Darling/Darling.Tests/AlertReadFailureSurfaceTests.cs:

  • Both loops changed from while (condition) { body } to do { body } while (condition);, so the reader body runs at least once regardless of how fast the writers finish.
  • The comment above each Assert.True(observations > 0, ...) was reworded to say the observation count is now guaranteed by the loop shape rather than hoped for from scheduling luck. The first test's comment also notes the pre-seeded bucket makes that guaranteed first observation a complete trio, matching what the assert right below it already checks.
  • No behavior of the counter under test changed, and the body of each loop is untouched.
  • UnderConcurrency_TheInstanceTotalNeverTrailsItsParts also now seeds both parts before its writers start, the same way TheNewestFailuresFactsAreNeverABlendOfTwo seeds its bucket: the do guarantees a first observation, but without this seed that first observation (and any others before the scheduler resumes the writers) can still see both parts at zero, so the observedBothParts > 0 liveness assert could still fail under starvation. The settled counts (2 * WritesPerWriter + 1) and the comment above the liveness asserts were updated for the extra seeded failure.

Test plan

  • dotnet build Darling/Darling.Tests/Darling.Tests.csproj — 0 Warning(s), 0 Error(s).
  • Darling.Tests.exe -class "*AlertReadFailureSurfaceTests*" run 5 times in a row: 27/27 passed each time, 0 failed.
  • Darling.Tests.exe -class "*DocCommentHygiene*": 77/77 passed.
  • Full suite: not run here, CI runs it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

erikdarlingdata and others added 2 commits September 25, 2026 13:48
…g the writers (#4312)

TheNewestFailuresFactsAreNeverABlendOfTwo and
UnderConcurrency_TheInstanceTotalNeverTrailsItsParts both polled writer
tasks with a `while` loop that checks completion before the body runs.
When all writer tasks finish before the test thread's first check, the
body never runs, observations stays 0, and the "reader observed
nothing" guard fails. Converted both to `do ... while` so the reader
body always runs at least once.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
…otalNeverTrailsItsParts (#4312)

The observedBothParts > 0 liveness assert could still fail under starvation: the
first read (from the do) runs before any writer has written, and if the scheduler
then starves the test thread until both writers finish, the loop exits with no
observation that saw both parts nonzero. Seed both parts before the writers start,
the same way TheNewestFailuresFactsAreNeverABlendOfTwo seeds its bucket, so every
observation sees both parts nonzero regardless of scheduling. Update the settled
counts for the extra seeded failure and reword the comment above the liveness
asserts.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 25, 2026 17:59
@erikdarlingdata
erikdarlingdata enabled auto-merge (squash) September 25, 2026 17:59
@erikdarlingdata
erikdarlingdata merged commit c48b696 into dev Sep 25, 2026
16 of 18 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/4312-alert-read-failure-reader-runs-once branch September 25, 2026 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant