Module health check activates from the current generation's own factory (#6766)

src/MeshWeaver.Documentation/Data/Architecture/LiveModuleUpdate.md

@@ -229,6 +229,32 @@ did not add, a keyed registration, a class-typed platform service the module imp
 implementation that also implements another platform interface (a proxy would hide it), an open generic
 the module implements.
 
+### A module's health check follows the swap (MeshWeaver#6766)
+
+A module's `AddHealthChecks().AddCheck<TCheck>()` reaches the root as `ModuleHealthCheckOptionsForwarder`,
+and the root's `DefaultHealthCheckService` activates the check. The root reads
+`IOptions<HealthCheckServiceOptions>` ONCE per process. The forwarder used to copy the registration that
+the boot generation's configuration added, and to rebind only its PROVIDER to the module's current
+scope. The factory it kept was therefore generation N's `GetServiceOrCreateInstance<TCheck>`: `TCheck`,
+and every type its constructor takes, were N's types. After a live swap, N+1's container was asked for N's
+dependency type, which it does not hold. Measured on a portal: every `/health` threw `Unable to resolve
+service for type 'MeshWeaver.AI.ProviderCatalogConvergence' while attempting to activate
+'MeshWeaver.AI.ProviderCatalogHealthCheck'` out of the health middleware. The kept factory also pinned
+generation N's collectible context.
+
+The root's registration now names no module type. It keeps the check's name, tags, failure status,
+timeout, delay and period. At every activation its factory takes ONE snapshot of the module's current
+generation, runs that generation's own factory for the check of that name, and activates it in that
+generation's container. Each generation's checks are read once, from its own configuration run against an
+empty options instance, and are held in a table keyed weakly by the generation, so a retired generation is
+never pinned. A module that is no longer held (retired by an uninstall, or withdrawn at boot), or a current
+generation that no longer declares the check, gives a NAMED `Degraded` result for that check, never an
+exception: the feature is absent, and the probe says which feature and why.
+`ModuleHealthCheckActivatesInModuleScopeTest` swaps a module live and retires it, running the check
+through the root's `HealthCheckService` each time. On the base, the swap case throws the measured shape
+(`Unable to resolve service for type '…Beat' while attempting to activate '…ProbeCheck'`) and the
+retire case throws `Module … holds no service scope`.
+
 ### Built before published (MeshWeaver#6542)
 
 🚨 **The root only ever READS a module's instances.** Every PROCESS-LIFETIME instance the root can

src/MeshWeaver.Mesh.Contract/ModuleContexts.cs

@@ -207,6 +207,15 @@ public object ResolveModuleService(string module, int index, Autofac.ILifetimeSc
     public IServiceProvider? ModuleScope(string module) =>
         Current(module)?.Services?.Scope(Root);
 
+    /// <summary>The scope of the generation <paramref name="services"/> belong to — a snapshot the caller
+    /// already took, so the scope and whatever else it reads off that snapshot are the SAME generation's —
+    /// or null once that generation's services are disposed (MeshWeaver#6766).</summary>
+    internal IServiceProvider? TryScopeOf(ModuleServices services) => services.TryScope(Root);
+
+    /// <summary>Registration <paramref name="index"/> of the generation <paramref name="services"/> belong
+    /// to — a snapshot the caller already took, never a second read of <see cref="Current"/>.</summary>
+    internal object ResolveIn(ModuleServices services, int index) => services.Resolve(Root, index);
+
     /// <summary>The current generations, with <paramref name="stage"/>'s generations in place of the
     /// current ones of the same name while it is open — what a STAGED generation's container sees.</summary>
     internal IReadOnlyCollection<ModuleGeneration> GenerationsSeenFrom(ModuleSwapStage? stage) =>

src/MeshWeaver.Mesh.Contract/ModuleServiceForwarders.cs

@@ -1,5 +1,7 @@
+using System.Collections.Concurrent;
 using System.Collections.Immutable;
 using System.Reflection;
+using System.Runtime.CompilerServices;
 using MeshWeaver.ServiceProvider;
 using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.Diagnostics.HealthChecks;
@@ -168,39 +170,102 @@ public Task StopAsync(CancellationToken cancellationToken) =>
 /// 'MeshWeaver.SelfUpdate.Aks.FleetWatchHeartbeat' while attempting to activate
 /// 'MeshWeaver.SelfUpdate.Aks.FleetWatchHealthCheck'</c>) and the control image never passed acceptance.</para>
 ///
-/// <para>So this forwarder runs the module's CURRENT generation's configuration and rebinds every
-/// registration it added to activate in that module's CURRENT scope — the same scope the check's
-/// dependencies (and the hosted service that feeds them) are served from, read at every activation so a
-/// live swap is followed.</para>
+/// <para>So the root holds, for every check the module declares, a registration that names NO module
+/// type — its name, tags, failure status, timeout, delay and period — and whose factory, at every
+/// activation, takes ONE snapshot of the module's CURRENT generation and runs THAT generation's own
+/// factory for the check of that name in THAT generation's container.</para>
+///
+/// <para>🚨 <b>Why the factory is the current generation's, not the boot one's</b> (MeshWeaver#6766).
+/// <c>IOptions&lt;HealthCheckServiceOptions&gt;</c> is built once per process, so a factory captured
+/// while it was built is generation N's <c>GetServiceOrCreateInstance&lt;TCheck&gt;</c> — <c>TCheck</c>
+/// is N's type, and so are its constructor's parameter types. Rebinding only the PROVIDER to the
+/// current scope then asked generation N+1's container, after a live swap, for N's dependency type,
+/// which it does not hold: <c>Unable to resolve service for type 'MeshWeaver.AI.ProviderCatalogConvergence'
+/// while attempting to activate 'MeshWeaver.AI.ProviderCatalogHealthCheck'</c> on every <c>/health</c>
+/// until the pod restarted — and the captured factory pinned generation N's collectible context.</para>
+///
+/// <para>A module that is no longer held (retired by an uninstall, or withdrawn at boot), or a current
+/// generation that no longer declares the check, is a NAMED <see cref="HealthStatus.Degraded"/> verdict
+/// for that check — never an exception out of the health middleware: the feature is absent, and the
+/// probe says which and why.</para>
 ///
 /// <para>🚨 It is an <see cref="IConfigureNamedOptions{TOptions}"/>, as <see cref="ModuleConfigureOptionsForwarder{TOptions}"/>
 /// is, and applies the module's setup by name exactly as <see cref="OptionsFactory{TOptions}"/> would: a module's
 /// <c>Configure&lt;HealthCheckServiceOptions&gt;("name", …)</c> must reach <c>IOptionsMonitor.Get("name")</c>, and a
-/// setup that is not named reaches the default name only (MeshWeaver#6749).</para>
+/// setup that is not named reaches the default name only (MeshWeaver#6749). The per-generation snapshot below is
+/// therefore taken PER OPTIONS NAME.</para>
 /// </summary>
 internal sealed class ModuleHealthCheckOptionsForwarder(ModuleContexts contexts, string module, int index)
     : IConfigureNamedOptions<HealthCheckServiceOptions>
 {
+    // The checks each GENERATION declares, per options name, by check name — keyed weakly by that
+    // generation's services, so a retired generation is never pinned by this table (an ephemeron: the
+    // value lives only while the key does). Mesh-scoped instance state on a singleton, never static
+    // (NoStaticState).
+    private readonly ConditionalWeakTable<ModuleServices, ConcurrentDictionary<string, ImmutableDictionary<string, HealthCheckRegistration>>> declared = new();
+
     public void Configure(HealthCheckServiceOptions options) => Configure(Options.DefaultName, options);
 
     public void Configure(string? name, HealthCheckServiceOptions options)
     {
-        var target = contexts.ResolveModuleService(module, index);
-        var before = options.Registrations.ToHashSet(ReferenceEqualityComparer.Instance);
-        if (target is IConfigureNamedOptions<HealthCheckServiceOptions> named)
-            named.Configure(name, options);
-        else if (name == Options.DefaultName)
-            ((IConfigureOptions<HealthCheckServiceOptions>)target).Configure(options);
-        else
+        var optionsName = name ?? Options.DefaultName;
+        if (contexts.Current(module)?.Services is not { } services)
+        {
+            // Withdrawn before the options were first read: its checks cannot even be named. One
+            // named verdict stands for them (the withdrawal itself is an IncompatibleModule record) —
+            // on the default name only, where a setup that cannot say its name lands.
+            if (optionsName == Options.DefaultName)
+                options.Registrations.Add(new HealthCheckRegistration(module,
+                    _ => Absent($"module {module} is not held by this mesh's module registry — it was withdrawn, so its health checks cannot run"),
+                    HealthStatus.Degraded, null));
             return;
-        foreach (var registration in options.Registrations.Where(r => !before.Contains(r)).ToArray())
+        }
+        foreach (var check in Declared(services, optionsName).Values.OrderBy(r => r.Name, StringComparer.Ordinal))
         {
-            var activate = registration.Factory;
-            registration.Factory = _ => activate(contexts.ModuleScope(module)
-                ?? throw new InvalidOperationException(
-                    $"Module {module} holds no service scope — its health check '{registration.Name}' cannot be activated."));
+            var checkName = check.Name;
+            options.Registrations.Add(new HealthCheckRegistration(checkName, _ => Activate(optionsName, checkName), check.FailureStatus, check.Tags, check.Timeout)
+            {
+                Delay = check.Delay,
+                Period = check.Period,
+            });
         }
     }
+
+    // ONE snapshot of the current generation: the factory and the container are the same generation's.
+    private IHealthCheck Activate(string optionsName, string name)
+    {
+        if (contexts.Current(module)?.Services is not { } services
+            || contexts.TryScopeOf(services) is not { } scope)
+            return Absent($"module {module} is no longer held by this mesh's module registry (retired or withdrawn) — its health check '{name}' cannot run");
+        return Declared(services, optionsName).TryGetValue(name, out var check)
+            ? check.Factory(scope)
+            : Absent($"the current generation of module {module} no longer declares the health check '{name}'");
+    }
+
+    // What `services`' own configuration adds to an EMPTY options instance of `optionsName`: that
+    // generation's checks, with that generation's factories — applied by name exactly as the options
+    // factory would (#6749). Run once per generation and name.
+    private ImmutableDictionary<string, HealthCheckRegistration> Declared(ModuleServices services, string optionsName) =>
+        declared.GetValue(services, _ => new ConcurrentDictionary<string, ImmutableDictionary<string, HealthCheckRegistration>>(StringComparer.Ordinal))
+            .GetOrAdd(optionsName, n =>
+            {
+                var probe = new HealthCheckServiceOptions();
+                var target = contexts.ResolveIn(services, index);
+                if (target is IConfigureNamedOptions<HealthCheckServiceOptions> named)
+                    named.Configure(n, probe);
+                else if (n == Options.DefaultName)
+                    ((IConfigureOptions<HealthCheckServiceOptions>)target).Configure(probe);
+                return probe.Registrations.ToImmutableDictionary(r => r.Name, StringComparer.Ordinal);
+            });
+
+    private static IHealthCheck Absent(string why) => new AbsentModuleHealthCheck(why);
+
+    /// <summary>The verdict for a check whose module (or whose declaration) is gone.</summary>
+    private sealed class AbsentModuleHealthCheck(string why) : IHealthCheck
+    {
+        public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default) =>
+            Task.FromResult(new HealthCheckResult(HealthStatus.Degraded, why));
+    }
 }
 
 /// <summary>

src/MeshWeaver.Mesh.Contract/ModuleServices.cs

@@ -354,6 +354,15 @@ or ModuleServiceRoute.Contributed
         }
     }
 
+    /// <summary>As <see cref="Scope"/>, or null once this generation's services are disposed (a live
+    /// uninstall, a swap's retirement, a boot withdrawal) — the disposal is an answer here, not a fault.</summary>
+    /// <param name="root">The mesh's root provider.</param>
+    internal IServiceProvider? TryScope(IServiceProvider root)
+    {
+        lock (gate)
+            return disposed != 0 ? null : Scope(root);
+    }
+
     /// <summary>The live swap this generation was staged in, or null (a generation loaded at boot).</summary>
     internal ModuleSwapStage? Stage { get; set; }