[rbuergi]

Refs Systemorph/MeshWeaver.Plugins#1605

The next mesh now starts only once the previous mesh's collectible NodeType contexts have really unloaded. The wait is a subscription to an observable.

This follows the maintainer's diagnosis (2026-09-11): "disposal being in progress while new instance starting". It also follows his design: "we need to wait (reactively, i.e. observable.Subscribe()) until all is really finished disposing".

What the dumps show

Six crash dumps were read for their managed state. The method is recorded in Doc/Architecture/DebuggingNativeCrashes, in the entry "the MANAGED view of sightings #11–#16".

Contexts mid-unload. In every FutuRe dump that could be read (#12–#16), 3 to 7 NodeAssemblyLoadContexts were _state = Unloading at the moment of death.

• In correcting release config #15, the next mesh was being built 12 ms after the previous DISPOSE_DONE: 5 contexts were Unloading and none were alive. The crashing thread was inside HostedHubsCollection.CreateHub, then MessageHubConfiguration.Build, then an Autofac resolve.

What the zeroed word belongs to. The zeroed MethodTable word sits in objects a hub builds. None is a collectible type, and none is a native wrapper:

• Autofac's child registry (ServiceRegistrationInfo, List<IComponentRegistration>);
• System.Text.Json polymorphic metadata;
• the TypeRegistry;
• persistence closures.

Where I measured reachability (#15, #16), the object is garbage: a BFS from every GC root never reaches it. It is left over from a hub that has already been disposed.

Nothing disposing at the instant of death. No managed thread was running disposal code. The finalizer thread was idle in WaitForFinalizerEvent. The unloads were pending in the GC.

#11 is a different branch. It is GitSync, with no collectible context at all, and an interior stale reference: dotnet/runtime#131267's shape.

Why DISPOSE_DONE was not "done"

AssemblyLoadContext.Unload() only requests an unload. The runtime holds the context through a strong handle and frees it later, on the finalizer thread, over subsequent collections. MeshTeardownSignal fires when teardown has requested those unloads, and the test bases wrote DISPOSE_DONE at that point. So xUnit built the next mesh over contexts that were still being torn down.

The fix

• CollectibleContextUnloads (new, MeshWeaver.Mesh.Contract, mesh-scoped singleton):
  • It records each retired context by its signal, never by reference.
  • AllCollected completes when every context retired before the subscription has been collected.
  • It errors when an unload was abandoned: the drain faulted, or Unload() threw.
  • It emits synchronously when nothing is pending.
• NodeAssemblyLoadContext.RetireInto attaches a finalizable sentinel that is referenced only by the context.
  • Unload() suppresses the context's own finalizer, so the sentinel carries the signal instead.
  • The runtime releases the context only when it destroys the LoaderAllocator. The sentinel can therefore be finalized only after the unload has really finished.
• CompilationCacheService retires every context it disposes (UnloadContext, Dispose) into the tracker. The tracker reaches it as an optional constructor parameter.
• Test bases. MonolithMeshTestBase and HubTestBase end teardown with CollectibleUnloadDrain.
  • It drives full collections, which a collectible unload needs and an idle test host never triggers on its own.
  • It then observes AllCollected.
  • A faulted unload fails the class, like a dirty teardown.
  • A retained context (still referenced after full collections free nothing) is reported as DISPOSE_ALC_RETAINED, and never waited on.

It follows "teardown lets work finish, never forced": nothing is cancelled and nothing is unloaded early. Teardown simply does not claim to be finished until the unload has finished.

What held the contexts: gcroot, and the second fix

Waiting was not enough on its own. In three real FutuRe runs, 84 teardowns ended DISPOSE_ALC_RETAINED and 81 ended DISPOSE_UNLOADS_COLLECTED. So in about half the fixtures, contexts survived three rounds of full collections.

How the holder was found.

• I took a heap dump the instant a teardown wrote RETAINED.
• I ran gcroot on the two unloading LoaderAllocators. The contexts themselves are always strong-held by the runtime's own handle, so rooting them answers nothing.
• Both LoaderAllocators have exactly one strong root, and it is the same: ReflectionEmitCachingMemberAccessor, the static cache in System.Text.Json.
• The chain from it runs CacheEntry → emitted Action<object,string> setter → DynamicMethod → DynamicScope → the collectible RuntimeType → LoaderAllocator.
• The other roots are handle type 10, HNDTYPE_WEAK_INTERIOR_POINTER, which is weak.

What this means. STJ shares its emitted accessors for 1 s after last use and evicts them on a 200 ms timer. An unload therefore finished whenever STJ's timer fired, usually while the next mesh was already starting. This is the same shape as the Autofac cache that ReflectionCacheEviction already purges.

The second fix. JsonMemberAccessorCacheEviction is registered on Unloading. It calls STJ's published MetadataUpdateHandler.ClearCache(Type[]?) hook, the one hot reload uses.

• Control: ATypeSerializedThroughSystemTextJson_IsCollectedAtTeardown_NotWhenStjsTimerFires. It also asserts that the hook exists.
• After the fix, over three FutuRe runs: 93 teardowns ended COLLECTED and 45 ended RETAINED.
• A second dump, taken at a RETAINED teardown with the eviction active, shows no strong root on any of the three contexts. Every root is a type-10 handle (HNDTYPE_WEAK_INTERIOR_POINTER, weak) pointing at the context's own statics, for example the compiler's <4>__ProfitByLoB method-group cache. So nothing held them after DisposeAsync returned.
• They were not slow. Allowing up to 20 no-progress rounds (40 full collections) left 30 teardowns still RETAINED, and nothing was collected between round 4 and round 20. Something live held them during the drain.
• A dump taken inside the drain names it: a stack slot of MonolithMeshTestBase.<DisposeAsync>d__87.MoveNext(), the very frame running the drain. The chain runs from the disposed MessageHub → Workspace → MeshDataSource → MeshContentTypeRegistry → DiscriminatorClaim → the collectible RuntimeType. The fixture's ServiceProvider field is a second route to the same graph.
• The third fix. The drain yields before collecting, and the test bases clear ServiceProvider first.

The progression, measured the same way each time (three FutuRe runs each):

state	collected	retained
waiting for "really unloaded" only	81	84
+ evicting STJ's accessor cache	93	45
+ the teardown releasing its own mesh first	126	12

The last 12 are reported and named, never waited on. Two further measurements settle them:

• No managed root. A second dump taken inside the drain, with both fixes active, shows no non-weak root on the remaining contexts.
• The cutoff is right. Letting the drain run 20 no-progress rounds instead of 3 collected 88 teardowns (32 with nothing to wait for, 56 in 2 rounds) and left 4 retained. None was collected at any round from 3 to 20, so three rounds is measured, not guessed.

The residual, 4–9 % of the teardowns that had anything to wait for, is held outside the managed heap, most likely by a native LoaderAllocator-to-LoaderAllocator reference or a runtime-internal one. I'm handing it to #4017.

For the #4017 retention work: on a long-lived portal, MeshContentTypeRegistry's discriminator claims hold superseded NodeType generations' RuntimeTypes for as long as the mesh lives.

Controls

RetiredContextCollectedSignalTest, clean Release -warnaserror build: 5 passed, 0 failed. The full MeshWeaver.Compiler.Pipeline.Test suite was run before the STJ control was added: 866 passed, 0 failed. MeshWeaver.Documentation.Test: 397 passed.

control	asserts	falsified by	red with
ATypeSerializedThroughSystemTextJson_IsCollectedAtTeardown_NotWhenStjsTimerFires	a context whose type went through STJ is collected at teardown, and STJ still publishes the hook	removing the Unloading += JsonMemberAccessorCacheEviction.EvictFor registration	…must be collectable the moment its unload is requested — not when STJ's 1 s eviction timer happens to fire (1 context(s) RETAINED after 3 round(s) freed nothing: DynamicNode_Json1605), but found False.

control	asserts	falsified by	red with
NextStart_RunsOnlyAfterTheRetiredContextIsCollected_AndTheContextIsCollected	a start sequenced on the signal does not run while the context is only Unload()-requested; the context IS collected; the start is released after collection	signalling at Unload() time (_retirement?.Collected() right after Unload())	Expected value to be false because Unload() has only been REQUESTED … but found True.
AFaultedUnload_ReleasesTheWaiterWithTheFault_NeverHangsIt	an Unloading handler that throws releases the waiter with that fault within budget	removing both Faulted(...) arms	Expected a InvalidOperationException because an abandoned unload must ERROR the signal … but found TimeoutException
WithNothingRetired_TheNextStartIsNotDelayed · ACollectedRetirement_LeavesNothingBehind_SoTheNextWaitIsImmediate	no delay when nothing is retired; nothing lingers after a collection	the cache holding each retired context (the tracker rooting what it waits for)	control 1 …must be collected (1 context(s) RETAINED after 3 round(s) freed nothing: DynamicNode_Signal1605), but found False. and control 4 — and the driver REPORTED the retention after 3 rounds instead of hanging

What this does NOT prove, stated plainly

• No deterministic crash repro. On macOS arm64, a minimal repro with no MeshWeaver code survived 16,112 overlapped iterations: collectible unload overlapping Autofac child scopes, STJ emit and a GC hammer. On linux-arm64 (10.0.12) it survived 11,289. The real MeshWeaver.FutuRe.Test survived 20 consecutive runs, 1,180 fixtures, on macOS arm64 without the fix. The CI rate is about 1 % of x64 runs, so none of these could be expected to crash. The dumps establish the overlap state at 5 of 5 FutuRe crashes. They do not name the thread that wrote the zero.
• Scope. This sequences a mesh instance after the previous one's unloads: test fixtures, and any host that tears a mesh down and builds another. In-process NodeType recompiles on a live portal still supersede a generation while other hubs run. That is not sequenced here, and it is flagged for reconciliation with the fix(#4003, #4013): only a PUBLISH supersedes a generation, and the script memo refuses per-generation files #4017 / fix(#4017 follow-up): a read of the store copy of a published build reuses that build's context #4029 retention work.

Coordination with #4017 / #4029

The mirror problem is contexts retained too long. This PR makes retention visible at every teardown (DISPOSE_ALC_RETAINED names the contexts) without failing on it, so it neither reds suites on the retention #4017 is fixing nor hides it. KernelScriptReferences.cs is untouched. CompilationCacheService.cs changes are surgical: the retire calls, the sentinel, and the two fault arms. Unload timing is unchanged. Nothing unloads earlier or later; teardown only waits for the unload to finish.

What's New: skipped — test-host teardown sequencing and an internal signal, with no user-visible effect.

Pairs-with: none — no public type or member is removed. CollectibleContextUnloads is a new public type, and the CompilationCacheService constructor parameter is on an internal class.
Implementers: none — no member is added to any public interface.
Mirror-sync: none — no i18n catalog key is added or changed.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Critical unload-tracking and teardown-sequencing issues remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR sequences new mesh startup until collectible contexts from the previous mesh have fully unloaded.

Changes:

• Adds sentinel-based unload tracking and fault propagation.
• Integrates unload draining into test teardown.
• Adds regression tests and updates crash-analysis guidance.

File summaries

File	Reviewed change
test/MeshWeaver.Hosting.Monolith.TestBase/MonolithMeshTestBase.cs	Adds collectible-unload draining to monolith teardown.
test/MeshWeaver.Fixture/HubTestBase.cs	Adds unload draining to hub-test teardown.
test/MeshWeaver.Fixture/CollectibleUnloadDrain.cs	Drives collection and reports unload outcomes.
test/MeshWeaver.Compiler.Pipeline.Test/RetiredContextCollectedSignalTest.cs	Tests collection, fault, and empty-tracker behavior.
src/MeshWeaver.Mesh.Contract/Threading/IoPoolServiceCollectionExtensions.cs	Registers the unload tracker.
src/MeshWeaver.Mesh.Contract/Threading/CollectibleContextUnloads.cs	Tracks collectible-context completion and faults.
src/MeshWeaver.Documentation/Data/Architecture/DebuggingNativeCrashes.md	Documents crash-dump findings and lifecycle conclusions.
src/MeshWeaver.Compiler.Pipeline/CompilationCacheService.cs	Retires contexts and reports unload failures.
.claude/skills/sigsegv/SKILL.md	Updates crash-triage guidance.

Review details

Suppressed comments (5)

src/MeshWeaver.Compiler.Pipeline/CompilationCacheService.cs:895

• Catch is upstream of the Subscribe callback, so an exception thrown by CompleteUnload() cannot be routed into this handler. This comment incorrectly says the arm also receives Unload() exceptions; the separate catch below is the only path that reports those failures, and conflating the two makes future error handling changes unsafe.

            {
                _logger?.LogError(ex,
                    "Drain signal for AssemblyLoadContext {ContextName} faulted — KEEPING its load "
                    + "context rather than unloading an assembly a scan may still be reading",

src/MeshWeaver.Mesh.Contract/Threading/CollectibleContextUnloads.cs:125

• These public state transitions let any consumer call Retirement.Collected() immediately (releasing AllCollected while no ALC was collected), or call Retire and never signal it (permanently wedging the wait). The tracker is intended to be driven only by its sentinel; keep the retirement transitions internal/opaque and expose a controlled adapter to the compiler instead of public mutators.

        public void Collected()
        {
            if (Interlocked.Exchange(ref collected, 1) != 0)
                return;

test/MeshWeaver.Fixture/CollectibleUnloadDrain.cs:82

• This is the safety-critical fallback that lets teardown continue with an outstanding context, but the added tests cover only collected and faulted unloads. Add a test that deliberately keeps a strong reference, drives the drain, asserts RetainedContexts contains that context, and verifies the call returns instead of waiting forever; otherwise a regression can either reintroduce the overlap or wedge teardown without a control.

        if (unloads.Faults.Count > 0)
            return new CollectibleUnloadOutcome(rounds, [], unloads.Faults[0]);
        if (unloads.Pending > 0)
            return new CollectibleUnloadOutcome(rounds, unloads.PendingContextNames, null);

test/MeshWeaver.Fixture/HubTestBase.cs:225

• HubTestBase never disposes the ServiceProvider (its base DisposeAsync() only tears down test logging), and disposing the mesh does not close the root hub's child scope. Consequently CompilationCacheService.Dispose() is not reached here, so no retired contexts are registered in collectibleUnloads and this wait returns immediately. The next HubTestBase mesh can still overlap the previous cache's collectible contexts; close the owning mesh scope before waiting.

            await base.DisposeAsync();
            Mesh = null!;
            var unloadOutcome = await CollectibleUnloadDrain.WaitUntilCollectedAsync(collectibleUnloads);

test/MeshWeaver.Fixture/HubTestBase.cs:226

• By this point base.DisposeAsync() has already called FileOutput.Dispose(). Logger was created with a factory returning that same helper, whose WriteLine is a no-op after disposal, so this Information record is silently dropped. The Retained outcome is therefore not reported for HubTestBase as claimed; write it to the teardown trace (or log before disposing the test output).

            var unloadOutcome = await CollectibleUnloadDrain.WaitUntilCollectedAsync(collectibleUnloads);
            Logger.LogInformation("[{DisposalId}] Collectible contexts: {Outcome}", disposalId, unloadOutcome);

• Files reviewed: 10/10 changed files
• Comments generated: 5
• Review effort level: Lite

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[rbuergi]

This is deliberate for now, and I have measured it rather than asserting it.

What a retention is. A context that is still referenced after rounds of full collections free nothing will not be freed by waiting. So it is reported, with names, as DISPOSE_ALC_RETAINED, and never waited on. Failing the class on it would make every suite red on a retention defect that is tracked separately (#4017 / #4029), before those holders are removed.

The holders are being removed one by one, by gcroot:

• The first holder was System.Text.Json's static accessor cache. It is now evicted on Unloading, in b1c8ef0.
• That took FutuRe from 84 to 45 retained teardowns over three runs. The next holder is being rooted now.

Once retention reaches zero, escalating Retained to a failure is a one-line change. I'd rather make it on evidence than now.

[rbuergi]

Follow-up with the measurements promised above. Every row is three FutuRe runs.

state	collected	retained
waiting for "really unloaded" only	81	84
+ STJ's accessor cache evicted on Unloading (b1c8ef0)	93	45
+ the teardown releases its own mesh before waiting (deef2f8)	126	12

The second holder turned out to be the teardown itself. A dump taken inside the drain found the only non-weak root in MonolithMeshTestBase.DisposeAsync's own MoveNext frame: the disposed MessageHub → … → MeshContentTypeRegistry.DiscriminatorClaim → the collectible type. Now the drain yields before it collects, and the test bases clear ServiceProvider first.

What is left:

• A second inside-the-drain dump shows no non-weak root on the remaining contexts.
• Letting the drain run 20 no-progress rounds instead of 3 collected nothing more. No teardown was collected at any round from 3 to 20. So three is the measured cutoff, not a guess.
• The residual (4–9 % of the teardowns that had anything to wait for) is held outside the managed heap, most likely by a native LoaderAllocator-to-LoaderAllocator reference.

So Retained stays a report, for this reason. Failing the class would red suites on a holder that neither this change nor a managed fix can release. The case is named in DISPOSE_ALC_RETAINED on every teardown and handed to the #4017 retention track. Once that residual reaches zero, escalating to a failure is a one-line change.

[github-actions]

Test Results (shard 0)

259 tests  ±0   259 ✅ ±0   2m 16s ⏱️ -5s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 8b38b7c. ± Comparison against base commit 8ddccad.

[github-actions]

Test Results (shard 3)

390 tests  ±0   390 ✅ ±0   42s ⏱️ -13s
  2 suites ±0     0 💤 ±0 
  2 files   ±0     0 ❌ ±0 

Results for commit 8b38b7c. ± Comparison against base commit 8ddccad.

[github-actions]

Test Results (shard 1)

508 tests  ±0   508 ✅ ±0   36s ⏱️ -13s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 8b38b7c. ± Comparison against base commit 8ddccad.

[github-actions]

Test Results (shard 2)

1 299 tests  +5   1 297 ✅ +5   3m 53s ⏱️ +4s
    3 suites ±0       2 💤 ±0 
    3 files   ±0       0 ❌ ±0 

Results for commit 8b38b7c. ± Comparison against base commit 8ddccad.

[github-actions]

Test Results (shard 4)

1 960 tests  ±0   1 960 ✅ ±0   4m 38s ⏱️ -3s
    4 suites ±0       0 💤 ±0 
    4 files   ±0       0 ❌ ±0 

Results for commit 8b38b7c. ± Comparison against base commit 8ddccad.

[github-actions]

Test Results (shard 5)

    6 files  ±0      6 suites  ±0   7m 15s ⏱️ +10s
2 746 tests ±0  2 554 ✅ ±0  192 💤 ±0  0 ❌ ±0 
2 750 runs  ±0  2 558 ✅ ±0  192 💤 ±0  0 ❌ ±0 

Results for commit 8b38b7c. ± Comparison against base commit 8ddccad.

[github-actions]

Test Results

   17 files  ±0     17 suites  ±0   19m 24s ⏱️ -17s
7 162 tests +5  6 968 ✅ +5  194 💤 ±0  0 ❌ ±0 
7 166 runs  +5  6 972 ✅ +5  194 💤 ±0  0 ❌ ±0 

Results for commit 8b38b7c. ± Comparison against base commit 8ddccad.