Repository navigation
StaleActivationSeedRollbackTest: out-of-band IStorageAdapter write not visible to immediate durable read under concurrent load #980
Description
Surfaced while measuring intra-project test parallelism (#979). The race exists on main today — single-threaded execution just never schedules the two operations closely enough to expose it.
Symptom
MeshWeaver.Hosting.Monolith.Test.StaleActivationSeedRollbackTest, both tests:
MeshWeaver.Reactive.Assertions.AssertionException :
Expected value to be 5000 because the out-of-band advance must be durable
before we reactivate, but found 1.
at StaleActivationSeedRollbackTest.cs:100
ReactivatedOwner_SeedsFromDurableStorage_NotFromTheRoutingCachePostRecycleWrite_LandsAboveTheDurableVersion_AndDoesNotRollTheStoreBack
Reproduction
Release build, native xUnit v3 host, test/MeshWeaver.Hosting.Monolith.Test:
| Config | Result |
|---|---|
full suite, maxParallelThreads: 1 |
499 tests, 0 failed, 458.3s |
full suite, maxParallelThreads: 4 + parallelizeTestCollections: true |
499 tests, 2 failed, 146.0s |
| that class alone, single-threaded | 2 tests, 0 failed, 3.0s |
To reproduce, drop this at test/MeshWeaver.Hosting.Monolith.Test/xunit.runner.json (the per-project opt-in added in #979) and run the project's native host:
{
"parallelizeAssembly": false,
"parallelizeTestCollections": true,
"maxParallelThreads": 4,
"methodTimeout": 30000
}What it is not
Not a path collision. The node path is GUID-suffixed per test (TestData/stale-seed-<guid>), so concurrent classes are not sharing a node.
The actual question
The test deliberately writes straight through IStorageAdapter, bypassing IMeshChangeFeed, so the path-resolution cache keeps serving the pre-recycle snapshot:
const long durableVersion = 5000L;
await Storage.Write(created with { Name = "durable-advance", Version = durableVersion }, JsonOptions)
.Should().Within(30.Seconds()).Emit();
(await ReadDurable(path).Should().Within(30.Seconds()).Emit())!
.Version.Should().Be(durableVersion, "the out-of-band advance must be durable before we reactivate");The write is awaited to completion, and the immediately-following durable read still observes the pre-recycle version 1.
So: is an acked out-of-band IStorageAdapter.Write guaranteed visible to a durable read that follows it? Two candidate answers, and they need distinguishing before anything is changed:
- Test assumption — the assertion only holds on an otherwise-idle machine, and the correct fix is to wait on the actual condition rather than assume immediate visibility.
- Product race — a real write/read visibility gap in the storage adapter that concurrency exposes, in which case it can bite production writers that bypass the change feed.
Please do not "fix" this by widening a timeout, adding a retry, or pinning the class to a serial collection — per AGENTS.md those are band-aids and (2) would stay live underneath.
Impact
Blocks enabling parallelism on Hosting.Monolith.Test, which measured 3.14x (458.3s -> 146.0s) — the single largest CI win available in the suite, and the project that sets the shard floor (shard-assign.sh weights it at 345s).
Activity
rbuergi commentedon Aug 9, 2026
Does not reproduce on current
main— closing.Re-measured after merging main (44 commits ahead of the base this was found on), same Release build, same
maxParallelThreads: 4+parallelizeTestCollections: true:StaleActivationSeedRollbackTestmainStaleActivationSeedRollbackTestfailures, 148.5sAlmost certainly fixed by #975 (
fix/945-resubscribe-reassert-rollback) and/or the work that addedMonotonicWriteGuardWindowTest— both landed on exactly the version-floor / write-ordering path this failure sat on.The only failure left in that run is unrelated and environmental —
MeshNodeLanguageServiceTest.CheckSpeculative_AddsNuGetReferenceDirectivefailing onAn error occurred while retrieving package metadata for 'Humanizer.2.14.1' from source 'nuget.org', on a machine that was dropping connections to nuget.org, management.azure.com and api.github.com all session.Consequently #979 now enables parallelism on
Hosting.Monolith.Test(458.3s -> 148.5s, 3.08x) rather than holding it back. Reopen if it resurfaces in CI.rbuergi commentedon Aug 9, 2026
Reopening — I closed this prematurely. The defect is not fixed; it moved to a sibling class.
CI run 31300348689 on #979, with
maxParallelThreads: 4onHosting.Monolith.Test:Same durable-seed / routing-cache family as the originally reported
StaleActivationSeedRollbackTest. So what I observed locally — the original two failures no longer reproducing after merging main — was current main narrowing the window, not removing the write-visibility assumption. My "does not reproduce" conclusion was drawn from a single local run on a machine that also could not run half the relevant assertions (see below).The open question stands unchanged: is an acked out-of-band
IStorageAdapterwrite guaranteed visible to a durable read that immediately follows it? Under concurrent load, it is observably not.Second, separate blocker found by the same run
This one is not a race and should not be tracked here as one — it is structural. The test asserts the mesh hub is GC-collected after disposal, which is a claim about process-wide reachability. It cannot be meaningful while other test classes hold live meshes in the same process. Parallelism does not make it flaky; it makes it wrong.
Worth noting it can never fail on macOS: its ClrMD root analysis throws
PlatformNotSupportedExceptionand the test self-reports "the verdict is inconclusive here, not green (#674)". Any local green on Apple silicon is not evidence for this test.Status
Hosting.Monolith.Testparallelism is reverted in #979 (Content.Testkeeps it). Monolith measured 458.3s -> 148.5s (3.08x) and is the project that sets the shard floor, so unblocking it is worth real effort — but it needs:MeshHubDisposalLeakTestwhere process-wide GC reachability is a valid assertion (its own process/collection, or an assertion that does not depend on the rest of the process being quiet).Explicitly NOT acceptable as a fix for either: a serial
[Collection]to hide the leak test, a retry, or a widened timeout.rbuergi commentedon Aug 9, 2026
Closing as already fixed by the store-level write CAS (#971 / PR #988) — verified by measurement, not inferred.
What the issue actually described. The report was "an out-of-band
IStorageAdapterwrite is not visible to an immediate read" (Expected value to be 5000 … but found 1). That framing was slightly off: an acked out-of-band write is immediately visible. What was never guaranteed is that your value won. The read-back of1below the written5000is a concurrent rollback, which is exactly the defect class #971 closed — a strictly-regressing write is now refused at the row (ON CONFLICT … WHERE target.version <= EXCLUDED.versionon Postgres, an atomic higher-version-winsAddOrUpdatein memory).Measured, deterministically. PR #979 established that running
MeshWeaver.Hosting.Monolith.TestwithmaxParallelThreads: 4reproduced this reliably — it is a lever, not a flake. Re-measured on current main with that lever applied: 498 passed, 0 failed, all fourStaleActivationSeedRollbackTestcases green.This also settles the contradiction in #979's own PR body, which stated in one section that the failures were cleared by
fix/945-resubscribe-reassert-rollbackand in another that Monolith still "reds 2 tests". The former is correct on current main, and #988 has since strengthened the same path further.Ruled out along the way: it is not a path collision (paths are GUID-suffixed).
Reopen if it recurs — but the reproduction lever above should be used to confirm, rather than waiting for a CI flake.
perf(test): parallelise Hosting.Monolith.Test — 314.7s -> 122.5s
perf(test): parallelise Hosting.Monolith.Test — 317.6s -> 110.2s (2.88x)