Repository navigation
Init faulting on a disposed DI scope is shutdown, not a FAILED hub (#2444) - #2604
Merged
Merged
Conversation
During hub initialization, a BuildupAction resolving from the hub's
ServiceProvider could observe ObjectDisposedException ('this LifetimeScope
(or one of its parent scopes) has already been disposed') and
HandleInitialize's .Catch put the hub in the permanent FAILED state, with
an Error-level 'initialization failed' log — for a hub that was in fact
being torn down (mesh/... data-source hubs, memex/memex-cloud pods,
2026-08-23..26, incident aaf0ffe1d2556dc6).
Root cause: the existing recognized-shutdown branch keys on IsShuttingDown
(own disposalStarted OR the ancestor-Dispose CloseCreation cascade). An
OUT-OF-BAND scope teardown — the host's root container disposed on a
Host.StartAsync abort or pod shutdown, or an ancestor Autofac scope
disposing this hub's child scope — never runs any hub Dispose first:
Autofac's Disposable.Dispose flips the scope's disposed flag BEFORE its
disposer reaches the tracked hub instance, so a BuildupAction resolving
inside that window sees a dead scope while IsShuttingDown is still false.
The 2026-08-26 occurrences line up exactly with the host-start aborts of
issue #2443: mesh hubs were mid-init when Host.StartAsync failed and the
root container was torn down.
Fix at the single seam every BuildupAction shares: HandleInitialize's
catch now also recognizes 'the fault chain carries ObjectDisposedException
AND the hub's own scope no longer resolves' (probed against the hub's
already-materialized IMessageHub registration) as a shutdown outcome —
Debug log, no FAILED residue, gate opened so the imminent disposal flows.
A disposed scope is proof the hub is going away: the hub instance is a
tracked component of that same scope, so its Dispose is already queued in
the very disposer that killed the scope. An ObjectDisposedException from
an UNRELATED disposed dependency (scope alive) still faults the init.
Deterministic repro: ScopeDisposedDuringInitializationTest disposes the
hub's own scope out-of-band while a BuildupAction is parked, using a
sentinel disposable created after the hub (reverse-creation-order disposal
runs it inside the exact window: scope flag set, hub Dispose not yet run).
RED before the fix (InitializationError records the ODE), GREEN after.
Full MeshWeaver.Messaging.Hub.Test suite: 306/306 passed (Release).
Closes #2444
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Improves hub-initialization failure classification so an ObjectDisposedException caused by out-of-band DI scope teardown during initialization is treated as a normal shutdown outcome (no FAILED residue, no error-level “init failed” log), with a deterministic regression test and a What’s New entry.
Changes:
- Extend
MessageHub.HandleInitializeshutdown recognition to include disposed-scope teardown detection. - Add
ScopeDisposedDuringInitializationTestto deterministically reproduce and pin the incident window. - Add a What’s New entry documenting the user-visible effect (no stuck FAILED hubs when startup races shutdown).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| test/MeshWeaver.Messaging.Hub.Test/ScopeDisposedDuringInitializationTest.cs | Adds a deterministic repro covering scope-disposed-during-init classification (#2444). |
| src/MeshWeaver.Messaging.Hub/MessageHub.cs | Updates init .Catch classification to treat disposed-scope ObjectDisposedException as shutdown. |
| src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-28-hub-init-scope-teardown.md | Documents the fix as a user-visible reliability improvement (Category: Fix). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
Contributor
Contributor
Contributor
Contributor
Contributor
Contributor
From review on #2604: IsTerminatedByScopeTeardown walked only InnerException, so an ObjectDisposedException carried by an AggregateException at any index but 0 was invisible to it. Reactive pipelines produce exactly that — a Merge that saw a real error alongside a teardown hands back an aggregate whose ordering nobody controls — so a shutdown would have been classified as an init FAILURE some of the time and as shutdown the rest, from one code path. HubDisposingException.IsHubDisposal already had a tree walk. Rather than copy a second one, both now share ExceptionChain — they ask different questions about the same graph, and a walker fixed in one place had already left the other wrong. 🚨 Extracting it exposed a defect in the ORIGINAL shape, which the shared version would have inherited unchanged. A for-loop over InnerException that ALSO recurses into InnerExceptions re-walks the same chain once per position, because AggregateException.InnerException IS InnerExceptions[0]. On nested aggregates that is exponential: 200 of them pinned a core at 100% CPU indefinitely — found by the new pathological-graph test on its first run, which is why it now asserts a WALL-CLOCK bound rather than mere termination. A walker that eventually returns the right answer is still a defect, and a depth cap could never have fixed it: the blow-up is in breadth of re-walking, not depth. The traversal is now iterative over an explicit stack with a reference-identity visited set and a 256-node budget — linear in the graph, bounded regardless of shape, and terminating on a CYCLIC graph, which the recursive version could not do at all. Exception graphs are caller-supplied data; this is the difference between a classification and a hang. 8 ExceptionChain tests (17 ms, against a core pinned indefinitely), plus HubDisposalFailureClassification, ScopeDisposedDuringInitialization and DeletedAddressNackClassification green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2444 (incident
aaf0ffe1d2556dc6).Root cause
HandleInitialize's recognized-shutdown branch keys onIsShuttingDown— owndisposalStartedOR the ancestor-DisposeCloseCreationcascade. An out-of-band scope teardown never trips either: when the host's root container is disposed (aHost.StartAsyncabort, pod shutdown) or an ancestor Autofac scope disposes this hub's child scope, no hubDispose()has run yet — Autofac'sDisposable.Disposeflips the scope's disposed flag before its disposer reaches the tracked hub instance. A BuildupAction resolving inside that window (prod:MeshDataSourceExtensions.SubscribeToOwnDeletionresolving the workspace onmesh/…hubs) throwsObjectDisposedExceptionwhileIsShuttingDownis still false → misclassified as init failure → permanent FAILED state + Error-level log for routine teardown. The 2026-08-26 occurrences line up to the second with #2443'sHost.StartAsyncaborts on the same pod.Fix — at the one seam every BuildupAction shares
The catch now also recognizes fault chain carries
ObjectDisposedExceptionAND the hub's own scope no longer resolves (probed by resolving the hub's own already-materializedIMessageHubregistration — cheap on a live scope, throws ODE on a dead one) as a shutdown outcome: Debug log, no FAILED residue, gate opened so the imminent disposal flows. A disposed scope is proof the hub is going away — the hub instance is a tracked component of that same scope, so itsDispose()is already queued in the very disposer that killed the scope; no zombie is possible. An ODE from an unrelated disposed dependency (scope alive) still faults the init, andInitializationErrorSurfacedTestkeeps pinning that side.Not a per-call-site try/catch in
MeshDataSource(that would be N band-aids for one classification defect), and no retry/re-resolve loop.Relation to draft #2438 (scope ownership): complementary, not overlapping — #2438 has the parent close the child's scope strictly after
DisposalCompleted, when no init can be in flight; this PR fixes the classification when a scope dies out-of-band mid-init.Deterministic repro
ScopeDisposedDuringInitializationTest: a sentinelIDisposableregistered in the hub's own scope and resolved by the parked BuildupAction (created after the hub ⇒ reverse-creation-order disposal runs it inside the incident window: scope flag set, hubDisposenot yet run); its disposal wakes the BuildupAction synchronously on the disposing thread, whose resolve then throws exactly the incident's ODE withIsShuttingDown == false. No sleeps, no races.InitializationErrorrecords the ODE — the exact incident.DisposeDuringInitializationTest/InitializationErrorSurfacedTeststay green.Verification
dotnet build -c Release -warnaserror(--no-incrementalfor the final src build): MeshWeaver.Messaging.Hub + test project, 0 warnings / 0 errors.MeshWeaver.Messaging.Hub.Testsuite, Release: 306/306 passed (1 m 18 s, fresh .trx).WhatsNewEntryIntegrityTestrun against a rebuilt DLL verified to embed the new entry: passed.🤖 Generated with Claude Code