fix(#2684): modules GC leaves the readiness path — runs after ApplicationStarted through IIoPool

memex/Memex.Portal.Shared/MemexConfiguration.cs

@@ -204,9 +204,18 @@ public TBuilder ConfigureMemexMesh(IConfiguration configuration, bool isDevelopm
             // Generations GC — delete only what NO activation entry references and nothing holds
             // open (skip-on-locked). Landing never deletes anything on a shared volume; this is
             // the one reclaim point. See ModuleLandingService.CollectGarbage.
-            var collected = ModuleLandingService.CollectGarbage(moduleRoot);
-            if (collected > 0)
-                Console.WriteLine($"[ModuleActivation] modules GC removed {collected} unreferenced generation(s)");
+            //
+            // 🚨 Registered, NEVER run here (#2684). It used to be a synchronous call at this exact
+            // spot — before the host listened — and on an Azure Files (CIFS) /data the
+            // rename-then-recursive-delete of orphaned generations is one SMB round-trip per file:
+            // minutes of uninterruptible IO (PID 1 in `Dsl` at wchan=wait_for_response), no
+            // listener on :8080, so the 300 s startup probe killed a pod the kernel could not even
+            // deliver the kill to, and the roll looped (memex-cloud, ci.6559). Housekeeping must
+            // not gate readiness: the hosted service runs the SAME pass, with the SAME semantics,
+            // after ApplicationStarted, through the file-system IIoPool. It stays ahead of the
+            // awaiting-setup early return below on purpose — an instance parked in setup still
+            // reclaims its garbage, exactly as the synchronous call did.
+            builder.ConfigureServices(services => services.AddModuleGenerationsGc(moduleRoot));
             var persistedActivation = ModuleActivationSidecar.Read(moduleRoot,
                 msg => Console.Error.WriteLine($"[ModuleActivation] {msg}"));
             var effectiveModules = ModuleActivationBoot.ComputeEffectiveModuleEntries(

src/MeshWeaver.Documentation/Data/Architecture/Modules.md

@@ -219,12 +219,12 @@ deployment is ever running an old bundle against a new platform. Inventing a shi
 is the one thing that must not happen: it mints a SECOND type identity and reintroduces the `as`/`is`
 trap-door that reads as a silent null.
 
-### 🚨 An ACTIVATED entry with no bytes — the boot-GC race (#2303)
+### 🚨 An ACTIVATED entry with no bytes — the GC race (#2303)
 
 The "Missing DLL" skip above is the SYMPTOM; #2303 traced one concrete way an entry ends up
-pointing at nothing: a race between `ModuleLandingService.CollectGarbage` (run at every pod's boot,
-before its own module set is computed) and a landing happening on a DIFFERENT replica at the same
-moment.
+pointing at nothing: a race between `ModuleLandingService.CollectGarbage` (run once per pod start,
+after `ApplicationStarted` — see the readiness section below) and a landing happening on a
+DIFFERENT replica at the same moment.
 
 A landing is two writes on the shared `/data` volume, deliberately ordered bytes-then-entry: it
 `Directory.Move`s the new generation into place, THEN writes the sidecar entry that names it
@@ -286,6 +286,30 @@ that had landed long ago, on both prods at once:
    The pin is protection, not a gate: a boot that cannot copy warns loudly and falls back to the
    shared path.
 
+### 🚨 GC is OFF the readiness path (#2684)
+
+Where the pass runs is as load-bearing as what it deletes. It used to run synchronously in the
+portal's boot path — before the host listened — and on an Azure Files (CIFS) `/data` the
+rename-then-recursive-delete of orphaned generations is one SMB round-trip per file: minutes of
+uninterruptible IO for a handful of directories. Rollout time thereby became a function of how much
+garbage the previous generation left on a network volume, which is unbounded and invisible until
+the probe kills the pod: memex-cloud's roll to ci.6559 sat as PID 1 in `Dsl` at
+`wchan=wait_for_response`, never bound :8080, blew the 300 s startup probe — whose kill cannot land
+on a process parked in uninterruptible IO — and looped, wedging the whole `helm upgrade`. Raising
+the probe budget would only move the cliff.
+
+Reclaiming orphans is housekeeping: valid at any time, needed by nothing the portal serves. So the
+pass now runs from `ModuleGenerationsGcHostedService`, registered by the same boot path that used
+to call it: `StartAsync` only registers an `ApplicationStarted` callback (it can never delay the
+listener), the callback schedules `CollectGarbage` on the file-system `IIoPool`, and the pass
+observes the pool's cancellation between directories so a mesh teardown never waits out a slow
+unlink. Nothing about the pass gates `/health` or `/alive`, and nothing about its SEMANTICS
+changed: same rules, same grace window, same atomic `.trash-*` rename — and the reference set is
+re-read from the per-module sidecar files at run time, so a post-start pass sees a set at least as
+fresh as the boot-time pass did. The running process is immune to its own reclaim because it loads
+store-landed generations from the process-local pin (above), never the shared tree — the same
+property that already protected it from a SIBLING pod's pass.
+
 Why a sidecar file and not a mesh node: the list is consumed before any storage provider, hub, or
 connection string exists, and it must move with the DLLs it describes — the landing service writes
 both in one operation onto the same volume, so they cannot drift apart.

src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-30-module-cleanup-no-longer-delays-a-restart.md

@@ -0,0 +1,20 @@
+---
+Name: Module cleanup no longer delays a restart
+Category: Fix
+Description: Cleaning up leftover module files now happens after the portal is already serving, so a restart can never hang on slow storage doing housekeeping.
+Icon: Sparkle
+Order: -20260830
+---
+
+# Module cleanup no longer delays a restart
+
+When the portal starts, it tidies up module files that older versions left behind. On deployments
+whose data lives on network storage, that tidying can take minutes — and it used to run *before*
+the portal started answering requests. A restart could therefore look stuck for so long that the
+platform gave up, killed the starting portal, and tried again, in a loop: an update that never
+completed, caused entirely by cleanup work nobody was waiting for.
+
+The cleanup now runs right after the portal is up and serving. Restarts and updates come up as fast
+as the portal itself, regardless of how much there is to tidy — and the tidying still happens, with
+exactly the same care as before: nothing in use is ever touched, and anything a busy moment skips
+is picked up on a later pass.

src/MeshWeaver.PluginCatalog/ModuleGenerationsGcHostedService.cs

@@ -0,0 +1,174 @@
+using System.Reactive.Subjects;
+using MeshWeaver.Mesh.Threading;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Hosting;
+using Microsoft.Extensions.Logging;
+
+namespace MeshWeaver.PluginCatalog;
+
+/// <summary>
+/// Runs the <c>modules/</c> generations GC (<see cref="ModuleLandingService.CollectGarbage(string,
+/// ILogger?, TimeSpan?, DateTime?, CancellationToken)"/>) OFF the readiness path — after
+/// <see cref="IHostApplicationLifetime.ApplicationStarted"/>, through the file-system
+/// <see cref="IIoPool"/> (#2684).
+///
+/// <para>🚨 <b>Why the boot path is the wrong place.</b> The pass used to run synchronously in
+/// <c>ConfigureMemexMesh</c>, before the host listened. Reclaiming orphaned generations is one SMB
+/// round-trip per file on an Azure Files (CIFS) <c>/data</c> — minutes of uninterruptible IO for a
+/// handful of leftover directories — and it reclaims nothing the portal needs in order to SERVE.
+/// Rollout time therefore became a function of how much garbage the previous generation left on a
+/// network volume: memex-cloud's roll to ci.6559 sat as PID 1 in <c>Dsl</c> at
+/// <c>wchan=wait_for_response</c> deleting a <c>.trash-*</c> generation, never bound :8080, blew
+/// the 300 s startup probe (whose kill cannot land on a process parked in uninterruptible IO), and
+/// looped. Raising the probe budget would only move the cliff; the GC has no business in front of
+/// the listener at all.</para>
+///
+/// <para><b>The semantics do not change — only the WHEN.</b> Same pass, same rules: delete only
+/// what no activation entry references and nothing holds open (skip-on-locked), landing never
+/// deletes, the atomic <c>.trash-*</c> rename (#2509), the #2303 grace window, fail-closed on an
+/// unreadable reference set. The reference set is re-read from the per-module sidecar files at run
+/// time, so a post-start pass sees a set at least as fresh as the boot-time pass did — and the
+/// running process is immune to its own reclaim because boot loads store-landed generations from a
+/// process-local pinned copy (<see cref="ModuleGenerationPin"/>), never the shared tree.</para>
+///
+/// <para><b>Lifecycle.</b> <see cref="StartAsync"/> only registers the
+/// <c>ApplicationStarted</c> callback — it can never delay the listener, and nothing here gates
+/// <c>/health</c> or <c>/alive</c>. The callback schedules the pass on the pool and returns; the
+/// blocking work runs on the pool's limited-concurrency scheduler with the pool's cancellation
+/// linked in, so a mesh teardown cancels a sweep still crawling a slow volume instead of waiting
+/// on it (the token is observed between directories — the unit of atomic removal).</para>
+/// </summary>
+public sealed class ModuleGenerationsGcHostedService : IHostedService, IDisposable
+{
+    /// <summary>The pass itself — a test seam. Production is
+    /// <see cref="ModuleLandingService.CollectGarbage(string, ILogger?, TimeSpan?, DateTime?,
+    /// CancellationToken)"/>; a test substitutes a collector it can block or count, which is how
+    /// "readiness never waits on a stalled sweep" is provable without a real CIFS mount.</summary>
+    public delegate int Collector(string moduleRoot, ILogger? logger, CancellationToken cancellationToken);
+
+    private readonly string moduleRoot;
+    private readonly IHostApplicationLifetime lifetime;
+    private readonly IIoPool pool;
+    private readonly ILogger<ModuleGenerationsGcHostedService>? logger;
+    private readonly Collector collect;
+    // One-shot terminal state: the count once the pass finished, or its error. AsyncSubject so a
+    // subscriber arriving after the sweep (a test, a diagnostic) still gets the outcome.
+    private readonly AsyncSubject<int> completed = new();
+    private IDisposable? startedRegistration;
+    private IDisposable? sweep;
+
+    /// <summary>Creates the service.</summary>
+    /// <param name="moduleRoot">The deployment root the <c>modules/</c> tree lives under — the
+    /// SAME resolved root the boot path computes effective modules from (<see cref="ModuleRoot"/>),
+    /// never <c>AppContext.BaseDirectory</c> directly.</param>
+    /// <param name="lifetime">The host lifetime whose <c>ApplicationStarted</c> gates the pass —
+    /// it fires only once every hosted service (Kestrel's listener included) has started.</param>
+    /// <param name="pool">The bounded IO pool the blocking filesystem pass runs through.</param>
+    /// <param name="logger">Diagnostics — the removal count is reported exactly as the boot-path
+    /// call reported it.</param>
+    /// <param name="collect">Test seam for the pass; see <see cref="Collector"/>.</param>
+    public ModuleGenerationsGcHostedService(
+        string moduleRoot,
+        IHostApplicationLifetime lifetime,
+        IIoPool pool,
+        ILogger<ModuleGenerationsGcHostedService>? logger = null,
+        Collector? collect = null)
+    {
+        this.moduleRoot = moduleRoot;
+        this.lifetime = lifetime;
+        this.pool = pool;
+        this.logger = logger;
+        this.collect = collect ?? ((root, log, ct) =>
+            ModuleLandingService.CollectGarbage(root, log, cancellationToken: ct));
+    }
+
+    /// <summary>
+    /// The pass's terminal outcome: emits the removed-directory count once the sweep finished,
+    /// errors when it faulted, and stays silent while it has not run. Tests and diagnostics wait
+    /// on this — readiness never does.
+    /// </summary>
+    public IObservable<int> Completed => completed;
+
+    /// <inheritdoc />
+    public Task StartAsync(CancellationToken cancellationToken)
+    {
+        // 🚨 Register only — no filesystem IO before the host listens (#2684). ApplicationStarted
+        // fires after EVERY hosted service has started, i.e. strictly after the listener is bound.
+        startedRegistration = lifetime.ApplicationStarted.Register(KickSweep);
+        return Task.CompletedTask;
+    }
+
+    private void KickSweep()
+    {
+        // InvokeBlocking: the pass is synchronous, blocking filesystem IO — it runs on the pool's
+        // limited-concurrency scheduler, with the pool's own cancellation linked in, so it can
+        // neither starve the ThreadPool nor outlive a teardown drain (the collector observes the
+        // token between directories).
+        sweep = pool.InvokeBlocking(ct => collect(moduleRoot, logger, ct))
+            .Subscribe(
+                removedCount =>
+                {
+                    if (removedCount > 0)
+                        logger?.LogInformation(
+                            "[ModuleActivation] modules GC removed {Count} unreferenced generation(s)",
+                            removedCount);
+                    completed.OnNext(removedCount);
+                    completed.OnCompleted();
+                },
+                ex =>
+                {
+                    // A faulted pass reclaims nothing and breaks nothing: every state it can leave
+                    // behind (an intact orphan, a .trash-* remainder) is one a later pass — the
+                    // next pod start — already plans for. Surfaced, never swallowed.
+                    logger?.LogWarning(ex,
+                        "[ModuleActivation] modules GC pass over {ModuleRoot} faulted — nothing "
+                        + "was collected; the next pod start plans it again.", moduleRoot);
+                    completed.OnError(ex);
+                });
+    }
+
+    /// <inheritdoc />
+    public Task StopAsync(CancellationToken cancellationToken)
+    {
+        Dispose();
+        return Task.CompletedTask;
+    }
+
+    /// <inheritdoc />
+    public void Dispose()
+    {
+        startedRegistration?.Dispose();
+        startedRegistration = null;
+        // Unsubscribing cancels the pooled leaf's linked token — a sweep still crawling a slow
+        // volume stops at its next between-directories check instead of parking the drain. The
+        // AsyncSubject is deliberately NOT disposed: a leaf mid-completion may still be delivering
+        // its terminal notification on a pool thread, and OnNext into a disposed subject throws
+        // where nothing can observe it. The subject holds no resources; it dies with the service.
+        sweep?.Dispose();
+        sweep = null;
+    }
+}
+
+/// <summary>Registration for the post-start <c>modules/</c> generations GC.</summary>
+public static class ModuleGenerationsGcExtensions
+{
+    /// <summary>
+    /// Registers <see cref="ModuleGenerationsGcHostedService"/> over <paramref name="moduleRoot"/>.
+    /// Two-registration idiom: the <c>IHostedService</c> forward is what STARTS it; the concrete
+    /// singleton is resolvable for tests and diagnostics (<see
+    /// cref="ModuleGenerationsGcHostedService.Completed"/>).
+    /// </summary>
+    /// <param name="services">The host's service collection.</param>
+    /// <param name="moduleRoot">The resolved module root — the same one the boot path computed its
+    /// effective module set from.</param>
+    public static IServiceCollection AddModuleGenerationsGc(
+        this IServiceCollection services, string moduleRoot)
+        => services
+            .AddSingleton(sp => new ModuleGenerationsGcHostedService(
+                moduleRoot,
+                sp.GetRequiredService<IHostApplicationLifetime>(),
+                sp.GetService<IoPoolRegistry>()?.Get(IoPoolNames.FileSystem) ?? IoPool.Unbounded,
+                sp.GetService<ILogger<ModuleGenerationsGcHostedService>>()))
+            .AddSingleton<IHostedService>(sp =>
+                sp.GetRequiredService<ModuleGenerationsGcHostedService>());
+}