[rbuergi]

Refs #6556 #6557 (sev:H production regression from #6545, holds the memex roll on image 3.1.10611)

Root cause

#6545 ("built before published") built every shared instance of a module container at boot and before a live swap commits, including each scoped registration (built at the container's root). A scoped instance belongs to a circuit or a request. The one built at boot ran against scoped state that no circuit had initialised. Radzen's AddRadzenComponents() registers DialogService as scoped, and its constructor subscribes to NavigationManager.LocationChanged. The module container resolves NavigationManager in a root child scope (RootResolutionScope), where RemoteNavigationManager is uninitialised, so the constructor threw 'RemoteNavigationManager' has not been initialized. BuildAllBeforePublish then withdrew the whole Radzen view pack (#6556). The withdrawn module's transient InstalledModuleAssembly registration then threw "no longer held by this mesh's module registry" inside the pre-warm sweep's enumeration, and the pod refused readiness (#6557).

Fix

ModuleServiceProvider.BuildRootReachableSingletons (it replaces BuildSharedInstances, which was added by #6545 the same day and has no callers outside core) pre-builds only process-lifetime singletons that the root can reach:

• The four Module containers: a root singleton reaching a module service while a module singleton needs a root singleton deadlocks (lock order root/module inverted) #6542 paths, by route. Proxy covers the proxy invoke and the health-check and options forwarders. Current is the forwarder for class-typed services. ModuleOwned and Contributed are reached through the ModuleOwnedRootSource stand-ins. A Private registration is built only when a built registration takes it. Hosted registrations are excluded, as before.
• Singletons only (RootScopeLifetime, shared). A scoped registration is never pre-built.
• No singleton whose constructor graph reaches a scoped registration. ScopedDependencyWalk detects this structurally from the registrations, not from type names. It reads every public constructor's parameters of a reflection-activated registration and follows them through transients and the module's own singletons. It also follows them into the root's registrations for types that name no collectible assembly; it never asks the root about a collectible type, because asking caches the type there and pins its context. A root singleton ends the walk. A delegate-activated registration names no dependencies, so it is still built.

The #6542 deadlock guarantee is kept. A root singleton that takes a scoped module service is a captive dependency, not a path by which the root reaches a module. Every root-reachable singleton is still built before publish.

Tests (MeshWeaver.Compiler.Pipeline.Test, Release)

• New: AModuleWithCircuitBoundServicesLoadsAtBootTest. It uses the Radzen shape against a root-scoped stand-in (CircuitBoundState) that throws exactly as RemoteNavigationManager does. The module has a scoped module class, a scoped platform class (the Current route), a singleton that takes circuit state, and a singleton that takes that singleton. The module is published, nothing is reported incompatible, and it serves.
• New: ModuleRootServicesSwapTest.AnUpdateWithCircuitBoundServices_SwapsLive. The same shape on a live swap; the swap goes Live.
• Both new tests are red against current main, with the production sentence: 'CircuitBoundState' has not been initialized. (while building … CircuitDialog).
• Negative controls stay green. In AModuleWhoseSingletonCannotBeBuiltAtBootTest, a root-reachable singleton whose constructor throws still withdraws the module and reports it. AnUpdateWhoseSingletonCannotBeBuilt_IsRefusedBeforeItCommits_AndNKeepsServing still refuses the swap.
• The Module containers: a root singleton reaching a module service while a module singleton needs a root singleton deadlocks (lock order root/module inverted) #6542 repro RootAndModuleSingletonsBuildInOneLockOrderTest is green (3/3, including its deadlocking negative control).
• Full MeshWeaver.Compiler.Pipeline.Test run: 1410/1410 passed, Release, locally. The touched projects build with -c Release -warnaserror with 0 warnings and 0 errors.

Not in this PR (#6557's latent half)

A module withdrawn at boot for a genuine reason still leaves its InstalledModuleAssembly transient in the root, which throws once the boot assembly is collected. Moving that registration into an Autofac registration source was tried and rejected: hosts that build with MS DI's BuildServiceProvider() ignore sources, and ConfigureModuleActivationTest lost every held module. This is documented as open in Doc/Architecture/LiveModuleUpdate → "Built before published". With this fix Radzen is no longer withdrawn, so it no longer reaches that path.

Deploy

Once an image carrying this fix is built, the memex roll proceeds on the next self-update Roll. No recycle is needed: the defect is in boot-time module activation, and a new pod is a fresh process.

Pairs-with: none — removes ModuleServiceProvider.BuildSharedInstances, added by #6545 earlier today; no satellite or in-mesh caller (the only callers were core's ModuleServices and one core test, both ported here)
Mirror-sync: none — no i18n catalog change

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Scoped services still resolve through the fixed module scope, while factories and service locators bypass detection and can reintroduce activation failures or deadlocks.

3 open findings

• Skipped root-reachable singleton can recreate root-module lock cycle · New
• Module resolution ignores caller scope for circuit-bound services · New
• Delegate singleton factories can resolve scoped state during pre-build · New

What changed in this PR

Narrows module pre-building to root-reachable singletons, aiming to avoid activating circuit-scoped services during boot or live swaps.

Changes:

• Adds structural scoped-dependency detection.
• Adds boot and live-swap regression tests.
• Documents the revised activation behavior and remaining issue.

File	Description
src/​MeshWeaver.ServiceProvider/​ModuleServiceProvider.cs	Implements filtered singleton pre-building.
src/​MeshWeaver.Mesh.Contract/​ModuleServices.cs	Selects root-reachable module routes.
test/​.../​RootAndModuleSingletonsBuildInOneLockOrderTest.cs	Updates lock-order coverage.
test/​.../​ModuleRootServicesSwapTest.cs	Adds live-swap coverage.
test/​.../​AModuleWhoseSingletonCannotBeBuiltAtBootTest.cs	Adds boot-time circuit-service coverage.
src/​MeshWeaver.Documentation/​Data/​Architecture/​LiveModuleUpdate.md	Documents the activation model.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[rbuergi]

You're right that this registration is left to its first resolve, and that a lazy first build of it can meet the #6542 lock cycle. That happens if a root singleton's constructor resolves it through Current/ModuleOwned while another thread is building it for the first time. It isn't confined to this skip, though. Every SCOPED module registration has the same shape: ModuleServices.Resolve serves it from the one module container (Scope(root)), so it is shared at that container's root and built lazily under its lock.

Rejecting the captive registration outright would withdraw any module that has one, which is the boot-time blast radius this PR removes. Building it would run circuit state nobody initialised, which is the #6556 failure itself. Neither closes the cycle for scoped registrations.

The root cause is that a module's scoped registrations resolve in the module container's root instead of a scope tied to the caller. I've filed it with the code path and what is and isn't established: rbuergi/Feedback/module-scoped-services-not-circuit-scoped (control instance bug triage). Fixing that makes the scoped-reaching singleton a real captive dependency that can be refused by name, and it removes this residual for both shapes together. This PR stays limited to the production blocker: boot no longer withdraws a module over circuit-bound services. No change in this PR.

[rbuergi]

Confirmed from the code. Current → ModuleContexts.ResolveModuleService → ModuleServices.Resolve always resolves from the generation's one Scope(root), and ModuleOwned resolves from the same container. So a module's scoped registration, and its scoped root dependency resolved through the fallback, is not circuit-scoped.

This PR only stops that registration being built at BOOT, where the build withdrew the whole module and refused the pod's readiness (the held production roll). It doesn't make the eventual resolve circuit-correct, and it doesn't claim to. The tests assert publication and swap, not a circuit resolve. The architectural gap predates #6542: before it, the lazy resolve took the same path.

Propagating the caller's scope into module resolution is a change to every forwarder and to ModuleContexts, so it is filed as its own root-cause item with the code path and what has not been measured on a live portal: rbuergi/Feedback/module-scoped-services-not-circuit-scoped (control instance bug triage). No change in this PR.

[rbuergi]

A delegate-activated singleton can resolve scoped state inside its factory, and the structural walk can't see into a delegate. Treating every delegate-backed root-reachable singleton as unsafe would leave most module registrations (options, factories, sp => … registrations) to a lazy first build. That reopens #6542's root/module lock cycle for all of them to protect against a shape no known module has.

So the trade-off is deliberate: a delegate singleton is built before publish, as on main since #6542. A factory that does reach uninitialised circuit state throws at boot, and the module is withdrawn and reported by name (IncompatibleModule, /health). That surfaces a captive registration; it doesn't hide one.

The real cure for scoped state reached from module services is resolving module scoped registrations in the caller's scope. That is filed with the code path: rbuergi/Feedback/module-scoped-services-not-circuit-scoped (control instance bug triage). No change in this PR.

[github-actions]

Test Results

    20 files   -   4      20 suites   - 4   54m 17s ⏱️ - 2m 39s
11 753 tests  - 305  11 753 ✅  - 111  0 💤  - 193  0 ❌  - 1 
11 767 runs   - 305  11 767 ✅  - 111  0 💤  - 193  0 ❌  - 1 

Results for commit 6d4c31d. ± Comparison against base commit c702582.

This pull request removes 347 and adds 18 tests. Note that renamed tests count towards both.

   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---, expected: True)
   --- End of inner exception stack trace ---, isDenial: True)
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
 ---> (Inner Exception #1) System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known<---
…

Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "84ceef4a-8f37-40a8-856d-1b3c87f8d19f")
Memex.Portal.Shared.Test.SessionDenialIsAnAnswerTest ‑ OnlyAVerdictReadsAsADenial(shape: "the same verdict nested, as a late denial dispatch"···, failure: System.InvalidOperationException: write failed
 ---> System.UnauthorizedAccessException: Access denied
   --- End of inner exception stack trace ---, isDenial: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's FIRST member", fault: System.AggregateException: One or more errors occurred. (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's SECOND member", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall nested in an aggregate inside a wrapper", fault: System.InvalidOperationException: outer
 ---> System.AggregateException: One or more errors occurred. (Value does not fall within the expected range.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.ArgumentException: Value does not fall within the expected range.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---

   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall wrapped as InnerException", fault: System.InvalidOperationException: outer
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "an aggregate of unrelated faults", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Value does not fall within the expected range.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
, expected: False)
MeshWeaver.Graph.Test.AModuleWithCircuitBoundServicesLoadsAtBootTest ‑ TheModuleIsPublished_AndServes
MeshWeaver.Graph.Test.ModuleRootServicesSwapTest ‑ AnUpdateWithCircuitBoundServices_SwapsLive
MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest ‑ Classifies(shape: "a MIXED aggregate — one transient branch, one genu"···, exception: System.AggregateException: One or more errors occurred. (Failed to connect to 10.42.18.4:5432) (source B is misconfigured)
 ---> MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
, expected: False)
…

♻️ This comment has been updated with latest results.