Skip to content

RequirePrebuiltParkTest flake is a real defect: the failed-verdict re-drive un-parks from a stale snapshot and drops a newer park #2260

Closed

Description

@rbuergi

What is failing

MeshWeaver.Hosting.Monolith.Test.RequirePrebuiltParkTest.AValidTypeWithNoAdoptedAssembly_ParksWithTheNamedRefusal_AndNeverCompiles fails intermittently on CI (first seen: run 32882908160, shard 1):

AssertionException : Expected value to be true because a require-prebuilt refusal must park
the type so no later trigger can storm, but found False.
   at RequirePrebuiltParkTest.cs(89)   // parkRegistry.IsParked(NodeTypePath).Should().BeTrue(...)

The test is right and the production code is wrong. A type that was parked ends up un-parked, so "no later trigger can storm" is not actually guaranteed.

Root cause — an unvalidated Unpark() from a stale snapshot

NodeTypeCompilationHelpers' failed-verdict re-drive subscribes to the NodeType's own stream and, on an emission it judges stale, does:

parkRegistry?.Unpark(hubPath);                      // ← decided from `node`, a SNAPSHOT
using var systemScope = redriveAccess?.ImpersonateAsSystem();
workspace.GetMeshNodeStream().Update(curr => { … });  // re-checks CURRENT state inside the lambda

The comment above the DistinctUntilChanged says it outright — "correctness is the re-check inside the Update lambda, not this". That is true for the flip to Pending, and false for the Unpark(): the un-park runs before the Update, is never re-validated, and therefore can remove a park that landed after the snapshot it was judged from.

Nothing re-parks afterwards, because the adopt-only gate only parks on a Pending flip and the type is now settled at Error.

The observed sequence (identical in CI and locally)

.402 CompileWatcher: … PARKING with a named refusal      ← gate refuses
.402 NodeType '…' PARKED after compile failure           ← park #1  (TryAdd succeeded)
     re-drive: FailedBuildInputs is "(never stamped)" ≠ live inputs → Unpark() + flip Pending
.420 CompileWatcher: … PARKING with a named refusal      ← gate refuses again
.421 NodeType '…' PARKED after compile failure           ← park #2  (TryAdd succeeded again)
.421 NodeType … is STUCK at Error … the automatic re-drive correctly declines
.427 TEST FAILED — IsParked == false

The double log line is the proof, not a cosmetic detail: ParkAndNotify logs only after a successful _parked.TryAdd, and nothing else in the registry removes an entry. Two "PARKED after compile failure" lines therefore mean the entry was removed between them — i.e. Unpark() ran. A third, late Unpark() from an in-flight re-drive subscriber then removes park #2 and leaves IsParked == false.

SettleAsError (the gate's settle) does not stamp FailedBuildInputs, which is why the re-drive fires at all on the very first park: the stamp is written only by the re-drive's own Update, so the first emission always looks "never stamped ≠ live".

Reproduction — deterministic with an amplifier

The window is tiny, so it is rare under a plain runner. Running the test with the detailed console logger widens it (more I/O between the emission and the un-park) and makes it fail every time:

DOTNET_PROCESSOR_COUNT=4 dotnet test test/MeshWeaver.Hosting.Monolith.Test --no-build \
  --filter "FullyQualifiedName~RequirePrebuiltParkTest" --logger "console;verbosity=detailed"
arm result
plain runner, isolated 20 / 20 pass
verbose logger, isolated 0 / 15 pass
full assembly, plain runner 758 / 761 pass (this test passed)

Every failing run logs exactly the double-park above.

Not caused by #2238

Found while validating PR #2238 (IoPool / MessageService disposal), which CI red-flagged with this test. Measured A/B on the same machine, same amplifier, DOTNET_PROCESSOR_COUNT=4:

source result
PR #2238 0 / 15 pass
origin/main, both touched files reverted 0 / 15 pass

Identical failure and identical double-park signature on unpatched main, so #2238's diff is not involved — as the mechanism already implies: it touches IoPool and the gated-deferral path in MessageService, while this defect is entirely inside NodeTypeCompilationHelpers + NodeTypeCompileParkRegistry, and this test never runs a compile at all (attempt count 0).

The fix is NOT a wait or a retry in the test

The assertion is correct — a parked type really is being un-parked. Options:

  • Validate the un-park against current state instead of the snapshot. The registry knows when it parked (ParkedCompileFailure.ParkedAt) and under which sources; a conditional UnparkIf(path, parkedBefore: snapshotTime) — or an Unpark that takes the park token it intends to retire — makes the operation compare-and-swap rather than unconditional remove. This matches the discipline the rest of the file already uses (the Update lambda re-checks).
  • Or move the Unpark() inside the same Update lambda that re-checks, so the un-park and the Pending flip commit against one observed state.
  • Additionally, SettleAsError arguably should stamp FailedBuildInputs with the inputs the refusal was formed under, so the very first park does not read as "never stamped" and trigger a re-drive that was never needed.

Where to look

  • src/MeshWeaver.Graph/Configuration/NodeTypeCompilationHelpers.cs — the failed-verdict re-drive Subscribe body (parkRegistry?.Unpark(hubPath); immediately before the Update), and SettleAsError in the adopt-only gate.
  • src/MeshWeaver.Graph/Configuration/NodeTypeCompileParkRegistry.cs — ParkAndNotify (TryAdd, logs only on success — the signal that proves the removal), Unpark.
  • test/MeshWeaver.Hosting.Monolith.Test/RequirePrebuiltParkTest.cs:89.

Activity

  1. added a commit that references this issue on Aug 25, 2026
    ed45882
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions