Skip to content

Flaky: learn tournament E2E fails on a missing mutator-attempts-1 barrier file (same commit passes on re-run) #467

Description

@justrach

Observed on PR #465, 2026-08-06. Two zig CI runs against the identical head SHA (0fea4f3): one failed, one passed.

tournament E2E failed; fixture preserved at /tmp/codegraff-learn-tournament.ob35k_n2
    assert (mutation_failure_barrier / "mutator-attempts-1").read_text() == "2"
FileNotFoundError: [Errno 2] No such file or directory:
  '/tmp/codegraff-learn-tournament.<tmp>/barrier-mutation-failure/mutator-attempts-1'

Failing run: https://github.com/justrach/codegraff/actions/runs/31104029504/job/92624467497
Passing run, same SHA: https://github.com/justrach/codegraff/actions/runs/31104035680

Same commit, opposite outcomes, so this is genuinely non-deterministic rather than a content regression. The PR that surfaced it (#418) changed only a new test file, test_hooks.zig and comments, none of which the tournament path touches.

What the assertion wants: the barrier fixture expects the mutator to have been attempted twice, recorded as a counter file. The file is absent entirely, not holding a wrong value — so the failure is upstream of the count: either the barrier directory was never populated, or the assertion ran before the writer finished.

Given #444 established that this repo's other long-standing "flake" was a misread of build-runner output rather than real non-determinism, it is worth resisting the same assumption here. This one has hard evidence of true non-determinism (identical SHA, both outcomes), so the mechanism is real and worth finding rather than papering over with a retry.

Likely candidates, in the order worth checking:

  1. A write/read race — the harness asserts on a counter file the mutator subprocess writes, without waiting for it. A missing file rather than a stale value fits this best.
  2. Fixture teardown or /tmp reuse — the fixture path is preserved on failure, so a preserved-but-empty directory from the failing run would distinguish this quickly.
  3. A spawn that failed silently, leaving the barrier unpopulated, with the assertion reporting the symptom rather than the cause.

Whoever takes this: the preserved fixture path in the failure output is the first thing to look at, and the two runs above give a working pair to diff.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions