fix(Plugins#1605): the next mesh starts only after the previous one's collectible contexts REALLY unloaded

.claude/skills/sigsegv/SKILL.md

@@ -108,10 +108,24 @@ Both are SIGSEGV. They are different bugs and the dump distinguishes them in one
 field the faulting code happened to read. Sighting #10 faults at `si_addr = 0x4` and is the same bug.
 Match on *"a MethodTable word that is exactly zero"*, never on the literal address.
 
-The second one is the FutuRe family — **fifteen sightings, and the collectible-ALC hypothesis has been
-falsified three separate ways** (RIP is in file-backed runtime code; a freed `LoaderAllocator` yields
-a non-null *unmapped* pointer, never `0x0`; a free-list item has no ALC at all). Do not keep paying
-that hypothesis forward, and **do not "fix" it by disabling concurrent GC** — that was tried
+The second one is the FutuRe family. 🚨 **Corrected 2026-09-11: the "collectible-ALC hypothesis
+falsified three ways" line that stood here was unsound.** Its three arguments (RIP in file-backed
+runtime code; a freed `LoaderAllocator` yields a non-null *unmapped* pointer; a free-list item has no
+ALC) exclude only a freed collectible *MethodTable* being dereferenced — the zeroed word is the header
+of an ordinary default-context object — and its `alc=1` support counts `AssemblyLoadContext.All`, which
+drops a context the moment `Unload()` is called. Read for the MANAGED state (ClrMD, not native SOS),
+every retained FutuRe dump (#12–#16) held **3–7 `NodeAssemblyLoadContext`s mid-unload** while the next
+instance was built or run, and the zeroed object is the **garbage of a disposed hub** (Autofac child
+registry, STJ polymorphic metadata, `TypeRegistry`) — see DebuggingNativeCrashes.md, *"the MANAGED view
+of sightings #11–#16"*. The GitSync pair (#10/#11, `GetCodeInfo`) is a different branch: no collectible
+context at all, an interior stale reference. 🚨 **When an unload does not finish, gcroot the
+`LoaderAllocator`, never the context** (the runtime's own strong handle always holds an unloading context).
+Measured 2026-09-11: the one strong holder of FutuRe's unfinished unloads was **System.Text.Json's static
+`ReflectionEmitCachingMemberAccessor` cache** (a `DynamicMethod` accessor's scope holds the collectible
+type; 1 s sliding expiry on a 200 ms timer), so contexts were freed when STJ's timer fired — during the
+next mesh. Evicted now on `Unloading` (`JsonMemberAccessorCacheEviction`), like Autofac's. Handle type
+`(10)` in gcroot output is `HNDTYPE_WEAK_INTERIOR_POINTER` — weak, not a root. And **do not "fix" it by
+disabling concurrent GC** — that was tried
 (#1274), changed nothing measurable, and was removed. Read the instruction + registers, not the
 function name: the frame moved across `background_sweep` → `plan_phase` → `find_first_object` →
 `background_mark_simple1` (2026-09-03) while the fault did not. 🚨 **And sighting #10 (2026-09-06)
@@ -123,11 +137,13 @@ dereferences it. 🚨 **And the frame REVISITS — it is not a progression.** Si
 inside the JIT, the other is back in `background_sweep()+0xa61` with `si_addr = 0x0` on a dedicated
 BGC thread — same runtime binary, 33 hours apart, `[R15]` reading zero in both. So do not read #10's
 move out of `gc_heap` as the family migrating toward the LCG/serialization workload; that was a
-sample of one. 🚨 **Nor is it a property of one runtime build, and the zeroed block can be a LIVE
-object somebody POINTS AT.** Sighting #15 (2026-09-10) ran on **`10.0.12`**, libcoreclr build-id
-`79945f51…` — a different binary from #4–#14's `10.0.11` / `989b56df…` — so "wait for the next patch"
-is not a plan; and a whole-core scan for its cursor found the address in a field of a live,
-well-formed 96-byte heap object, not only on a free list.
+sample of one. 🚨 **Nor is it a property of one runtime build.** Sighting #15 (2026-09-10) ran on
+**`10.0.12`**, libcoreclr build-id `79945f51…` — a different binary from #4–#14's `10.0.11` /
+`989b56df…` — so "wait for the next patch" is not a plan. (Its "live, referenced" block was later shown
+to be garbage: the 96-byte referrer is itself reachable from no GC root.) 🚨 **Read a dump for who was
+ACTIVE, not only for the faulting frame:** the evidence of an overlap is on the other threads and in
+the ALC census (`AssemblyLoadContext._state` through the GC handles), never in the GC thread that
+trips over the result.
 Measured base rate on Plugins CI: **0.74 %** over 1,197 runs (2026-08-29 → 09-03,
 denominator = non-cancelled runs) and **1.17 %** over 343 runs (2026-09-06 → 09-08, denominator =
 runs whose portal-host shard reached a verdict) — **different denominators, so not a trend**; both

src/MeshWeaver.Compiler.Pipeline/CompilationCacheService.cs

@@ -9,6 +9,7 @@
 using MeshWeaver.Compiler;
 using MeshWeaver.Graph.Configuration;
 using MeshWeaver.Mesh.Persistence;
+using MeshWeaver.Mesh.Threading;
 using MeshWeaver.ServiceProvider;
 using Microsoft.Extensions.Logging;
 using Microsoft.Extensions.Options;
@@ -598,6 +599,38 @@ private sealed class NullLease : IDisposable
         public void Dispose() { }
     }
 
+    // 🚨 Plugins#1605 — the point at which this context has REALLY unloaded. Unload() only starts
+    // an unload (and calls GC.SuppressFinalize on the context, so a finalizer here would never run);
+    // the runtime keeps the context alive through a strong handle until it destroys the
+    // LoaderAllocator. So the signal is a separate sentinel referenced ONLY by this field: it
+    // becomes unreachable in the same collection as the context — which the runtime allows only
+    // after phase two of the unload — and its finalizer reports the retirement collected. Set once,
+    // at retirement: a context aliased under two cache keys is ONE generation and must be tracked
+    // once, or its second retirement would never complete.
+    private CollectibleContextUnloads.Retirement? _retirement;
+    private RetirementSentinel? _retirementSentinel;
+
+    /// <summary>
+    /// Records this context on <paramref name="unloads"/> as asked to unload, so the mesh can wait
+    /// until it has really been collected (<see cref="CollectibleContextUnloads.AllCollected"/>).
+    /// Call BEFORE <see cref="Dispose"/>; a second call is a no-op.
+    /// </summary>
+    internal void RetireInto(CollectibleContextUnloads unloads)
+    {
+        lock (_loadLock)
+        {
+            if (_retirement is not null)
+                return;
+            _retirement = unloads.Retire(Name ?? _nodeName);
+            _retirementSentinel = new RetirementSentinel(_retirement);
+        }
+    }
+
+    private sealed class RetirementSentinel(CollectibleContextUnloads.Retirement retirement)
+    {
+        ~RetirementSentinel() => retirement.Collected();
+    }
+
     public NodeAssemblyLoadContext(string nodeName, string? dllPath, ILogger? logger = null)
         : base(name: $"DynamicNode_{nodeName}", isCollectible: true)
     {
@@ -614,6 +647,11 @@ public NodeAssemblyLoadContext(string nodeName, string? dllPath, ILogger? logger
         // Autofac does this automatically for BeginLoadContextLifetimeScope; we manage the context
         // by hand, so we mirror it. Static handler ⇒ no self-reference that would defeat collection.
         Unloading += ReflectionCacheEviction.EvictFor;
+        // …and System.Text.Json's process-static member-accessor cache, the one strong root of the
+        // contexts a FutuRe teardown could not collect (Plugins#1605, gcroot): without this, a context
+        // whose types were serialised is freed whenever STJ's 1 s eviction timer fires — while the NEXT
+        // mesh is already starting. See JsonMemberAccessorCacheEviction.
+        Unloading += JsonMemberAccessorCacheEviction.EvictFor;
     }
 
     /// <summary>
@@ -856,6 +894,12 @@ public void Dispose()
                     "Drain signal for AssemblyLoadContext {ContextName} faulted — KEEPING its load "
                     + "context rather than unloading an assembly a scan may still be reading",
                     Name);
+                // Kept means this unload will never finish: say so to whoever waits for it. This arm
+                // also receives an Unload() that threw inside CompleteUnload (the drain subscription
+                // reaches the completed subject through Rx's SubscribeSafe); CompleteUnload has
+                // already reported that exact exception, and the first fault is the one signalled.
+                _retirement?.Faulted(new InvalidOperationException(
+                    $"AssemblyLoadContext {Name} was KEPT loaded: its unload faulted", ex));
                 return Observable.Empty<Unit>();
             })
             .Subscribe(_ => CompleteUnload());
@@ -875,7 +919,22 @@ private void CompleteUnload()
         _loadedAssembly = null;
 
         // Initiate unload outside the lock — the context is collected once all references release.
-        Unload();
+        // An Unload() that throws (an Unloading handler, raised first, faulting) leaves the context
+        // LOADED. This runs as the OnNext of the drain subscription — synchronously inside Dispose
+        // when the drain was already quiet, but on the thread of the LAST scan's pin release when it
+        // was not — so a rethrow would escape into that scan's Dispose (Copilot review, #4042). It is
+        // handled here instead: logged at Error, and reported to whoever waits for the unload, which
+        // fails the teardown that expected it (DISPOSE_UNLOAD_FAULTED).
+        try
+        {
+            Unload();
+        }
+        catch (Exception ex)
+        {
+            _logger?.LogError(ex,
+                "Unload() of AssemblyLoadContext {ContextName} threw — the context stays LOADED", Name);
+            _retirement?.Faulted(ex);
+        }
 
             // Diagnostic probe (opt-in, off by default): drive a collection right after the unload so a
             // use-after-unload dangling native pointer trips at THIS unload — naming the culprit node
@@ -921,9 +980,18 @@ private void CompleteUnload()
 /// </summary>
 internal class CompilationCacheService(
     IOptions<CompilationCacheOptions> options,
-    ILogger<CompilationCacheService> logger)
+    ILogger<CompilationCacheService> logger,
+    CollectibleContextUnloads? unloads = null)
     : ICompilationCacheService, IDisposable
 {
+    /// <summary>Records <paramref name="context"/> as retired on the mesh's tracker (if any) so the
+    /// mesh can wait until it has REALLY been collected — Plugins#1605.</summary>
+    private void Retire(NodeAssemblyLoadContext context)
+    {
+        if (unloads is not null)
+            context.RetireInto(unloads);
+    }
+
     private readonly CompilationCacheOptions _options = options.Value ?? new CompilationCacheOptions();
     private readonly ConcurrentDictionary<string, NodeAssemblyLoadContext> _loadContexts = new();
     private readonly ConcurrentDictionary<string, System.Collections.Immutable.ImmutableArray<string>> _probingDirs = new();
@@ -1492,6 +1560,7 @@ public void UnloadContext(string nodeName)
         if (_loadContexts.TryRemove(nodeName, out var context))
         {
             logger.LogDebug("Unloading AssemblyLoadContext for {NodeName}", nodeName);
+            Retire(context);
             context.Dispose();
         }
     }
@@ -1637,6 +1706,7 @@ public void Dispose()
             {
                 try
                 {
+                    Retire(context);
                     context.Dispose();
                 }
                 catch (Exception ex)

src/MeshWeaver.Compiler.Pipeline/JsonMemberAccessorCacheEviction.cs

@@ -0,0 +1,56 @@
+using System.Reflection;
+using System.Reflection.Metadata;
+using System.Runtime.Loader;
+using System.Text.Json;
+
+namespace MeshWeaver.Graph.Configuration;
+
+/// <summary>
+/// Empties System.Text.Json's PROCESS-STATIC member-accessor cache when a collectible node context
+/// starts unloading — the System.Text.Json twin of <c>ReflectionCacheEviction</c> (Autofac).
+///
+/// <para><b>Why (Plugins#1605, measured).</b> On CoreCLR, STJ builds property getters/setters and
+/// constructors as <see cref="System.Reflection.Emit.DynamicMethod"/>s and keeps them in a static
+/// cache with a 1-second sliding expiry, evicted by a 200 ms timer
+/// (<c>ReflectionEmitCachingMemberAccessor</c>). A dynamic method's scope holds the
+/// <see cref="System.RuntimeTypeHandle"/> of every type its IL touches, so serialising ONE instance of
+/// a NodeType-compiled type roots that type's <c>LoaderAllocator</c> from a static field. A heap dump
+/// taken at the end of a FutuRe teardown (gcroot, 2026-09-11) shows exactly that as the ONLY strong
+/// root of both contexts that teardown could not collect: <c>ReflectionEmitCachingMemberAccessor</c>
+/// → <c>Cache</c> → <c>CacheEntry</c> → <c>Action&lt;object, string&gt;</c> → <c>DynamicMethod</c>
+/// → <c>DynamicResolver</c> → <c>DynamicScope</c> → <c>RuntimeTypeHandle</c> → the collectible
+/// <c>RuntimeType</c> → <c>LoaderAllocator</c>. So an unload "completed" whenever STJ's timer happened
+/// to fire — typically a second later, while the NEXT mesh was already being built — which is the
+/// disposal-in-progress-while-the-next-instance-starts overlap every readable crash dump caught.</para>
+///
+/// <para><b>How.</b> Through STJ's own published hook, not its internals: the assembly declares a
+/// <see cref="MetadataUpdateHandlerAttribute"/> whose handler exposes the static
+/// <c>ClearCache(Type[]?)</c> the hot-reload agent calls after a metadata update. It clears the
+/// member-accessor cache (and the per-options caches STJ tracks only while hot reload is on). Clearing
+/// drops shared accessors for every type, not only the unloading one; the cost is a re-emit the next
+/// time a NEW options instance resolves a type — live options keep the accessors they already hold.</para>
+///
+/// <para>Runs inside <c>Unloading</c>, i.e. synchronously within <c>Unload()</c>, before the context
+/// is released — the same moment Autofac's cache is purged. Holds no state: the resolved method is an
+/// immutable lookup computed once (NoStaticState allows <c>static readonly</c> constants).</para>
+/// </summary>
+internal static class JsonMemberAccessorCacheEviction
+{
+    private static readonly MethodInfo? ClearCache = typeof(JsonSerializer).Assembly
+        .GetCustomAttributes<MetadataUpdateHandlerAttribute>()
+        .Select(attribute => attribute.HandlerType.GetMethod(
+            "ClearCache",
+            BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static,
+            [typeof(Type[])]))
+        .FirstOrDefault(method => method is not null);
+
+    /// <summary>
+    /// Whether this runtime's System.Text.Json publishes the hook. Pinned true by a control: if a
+    /// future STJ drops it, the eviction must fail a test, not silently stop happening.
+    /// </summary>
+    public static bool IsAvailable => ClearCache is not null;
+
+    /// <summary>The <c>Unloading</c> handler. Static, so it roots nothing.</summary>
+    public static void EvictFor(AssemblyLoadContext loadContext)
+        => ClearCache?.Invoke(null, [null]);
+}