Bound Orleans cluster teardown so it stops starving the running test by rbuergi · Pull Request #1051 · Systemorph/MeshWeaver

rbuergi · 2026-08-09T21:18:28Z

The shard-0 Orleans flake is one cross-test interaction wearing six names. Across four runs it has failed as OrleansStaticRepoImportTest, OrleansMeshTests.HubWorksAfterDisposal, DelegationCompletionTest, OrleansMarkdownExportTest (twice, two different methods) and OrleansBrokenNodeTypeAccessTest — the last of those on #1045, whose entire diff adds Class="@Class" to four buttons in a .razor. None of them were anyone's diff.

Root cause

OrleansClusterDisposal hands each TestCluster's stop→dispose to IoPool.**Unbounded** and makes it hot immediately. Both of those are right on their own: teardown must not block the xUnit thread (awaiting it inline was the original deadlock this file was written to fix), and the next class should start at once.

With no bound, though, every cluster the assembly has built can be shutting down at once underneath the class currently running. Silo shutdown is not cheap. On a 4-vCPU runner those overlapping teardowns starve the live test's render, which then dies on its own 30s Timeout. The victim is whoever happens to be running when enough teardowns pile up — hence a different name every time, and hence it never correlated with any change.

Reproduced deterministically

Not a stress harness — just the runner's shape, via DOTNET_PROCESSOR_COUNT:

build	processors	result
unbounded	18 (native)	141/141 pass
unbounded	4	1 FAILED — 30s timeout
unbounded, victim class alone	4	5/5 pass in 5s
bounded(2)	4	141/141 pass
bounded(2)	2	141/141 pass

The third row is the one that settles it: the class that fails in the suite passes by itself under the same constraint, so it is the overlap and not the test.

The fix

IoPool(2) instead of IoPool.Unbounded for the disposal legs.

This is the framework's own prescription rather than a tuning knob — AGENTS.md: "concurrency bounding … channels through IIoPool" — and IoPool(int maxConcurrency) already exists. Teardown stays off the teardown thread, so the deadlock the background disposal exists to avoid stays avoided; it simply stops competing with the suite for the runner. Legs of different clusters never wait on one another, so the bound cannot deadlock the final drain.

No What's New entry: test infrastructure, no user-visible change.

🤖 Generated with Claude Code

The shard-0 Orleans flake was one cross-test interaction wearing six names. OrleansClusterDisposal hands each TestCluster's stop→dispose to IoPool.Unbounded and makes it hot immediately — correct, in that teardown must not block the xUnit thread (that was the original deadlock) and the next class should start at once. But with no bound, every cluster the assembly has built can be shutting down AT ONCE underneath the class currently running. Silo shutdown is not cheap; on a 4-vCPU runner those overlapping teardowns starve the live test's render, which then dies on its own 30s Timeout. The victim is simply whoever is running when enough teardowns pile up, which is why it was a different test every run and why it never looked like anyone's diff. Reproduced deterministically with DOTNET_PROCESSOR_COUNT=4 — the runner's shape, not a stress harness: unbounded, 18 cores (native) 141/141 pass unbounded, 4 processors 1 FAILED (30s timeout) the victim class ALONE, 4 5/5 pass in 5s → the overlap, not the test bounded(2), 4 processors 141/141 pass bounded(2), 2 processors 141/141 pass Bounding is the framework's own prescription — "concurrency bounding channels through IIoPool" — and IoPool(maxConcurrency) already exists. It keeps teardown off the teardown thread, so the deadlock the background disposal was written to avoid stays avoided, while stopping teardown from competing with the suite. Legs of different clusters never wait on each other, so the bound cannot deadlock the final drain. No What's New entry: test infrastructure, no user-visible change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

This PR addresses an Orleans test-suite flake caused by cross-test resource starvation by bounding concurrent Orleans cluster teardown work, while still keeping teardown off the xUnit teardown thread (avoiding the original deadlock risk).

Changes:

Introduces a bounded IIoPool (new IoPool(2)) as a shared gate for Orleans TestCluster teardown.
Routes teardown legs through the bounded pool instead of IoPool.Unbounded to prevent concurrent shutdowns from starving currently-running tests.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

    private static IObservable<Unit> RunVoid(Func<Task> leaf) =>
-        IoPool.Unbounded
+        DisposalPool
            .Run(_ => leaf().ContinueWith(static _ => Unit.Default, TaskScheduler.Default))
            .Catch(Observable.Return(Unit.Default));


github-actions · 2026-08-09T21:33:20Z