Webhook inbox delivery fails with unhandled ObjectDisposedException when portal MessageHub is shutting down #5136

Activity

1. systemorph

   added

   bugSomething isn't workingSomething isn't working

   on Sep 21, 2026
2. systemorph commented on Sep 21, 2026

   

   systemorphBot

   on Sep 21, 2026

   ContributorAuthor

   More actions

   Still occurring: 1 more since the last update (5 total).

   • Last seen: 2026-09-21 10:24:24Z
   • Pods: memex-portal-deployment-5998d5fddf-9sfsv

   Recent log lines

   2026-09-21 09:56:24Z memex-portal-deployment-5998d5fddf-9sfsv fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
         An unhandled exception has occurred while executing the request.
         System.ObjectDisposedException: Hub portal/nodeops-wpWb1fSJPUacoVxY6LnltQ is shutting down — cannot register new response subject for pbxE-hzl3kKLKsVYeShjGw.
         Object name: 'MessageHub'.
            at Memex.Portal.Shared.Api.WebhookInboxEndpoints.Deliver(String target, HttpRequest request, IMessageHub hub, IConfiguration config, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs:line 87
            at Microsoft.AspNetCore.Http.RequestDelegateFactory.ExecuteTaskResult[T](Task`1 task, HttpContext httpContext)
            at Memex.Portal.Shared.Authentication.OnboardingMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/OnboardingMiddleware.cs:line 166
            at Memex.Portal.Shared.Authentication.VirtualUserMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/VirtualUserMiddleware.cs:line 78
            at MeshWeaver.Hosting.AspNetCore.Portal.UserContextMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.AspNetCore/Portal/UserContextMiddleware.cs:line 277
            at Memex.Portal.Shared.Seo.PublicSite.<>c__DisplayClass7_0.<<UsePublicHostRedirect>b__0>d.MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Seo/PublicSite.cs:line 121
         --- End of stack trace from previous location ---
            at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
            at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
            at MeshWeaver.Hosting.AspNetCore.MeshModuleStaticAssetExtensions.<>c__DisplayClass6_0.<<UseMeshModuleStaticAssets>b__1>d.MoveNext() in /home/ru…[truncated]
   2026-09-21 09:56:24Z memex-portal-deployment-5998d5fddf-9sfsv fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
         An unhandled exception has occurred while executing the request.
         System.ObjectDisposedException: Hub portal/nodeops-wpWb1fSJPUacoVxY6LnltQ is shutting down — cannot register new response subject for jKO3a6DXqUSr0BW1VSFY5A.
         Object name: 'MessageHub'.
            at Memex.Portal.Shared.Api.WebhookInboxEndpoints.Deliver(String target, HttpRequest request, IMessageHub hub, IConfiguration config, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs:line 87
            at Microsoft.AspNetCore.Http.RequestDelegateFactory.ExecuteTaskResult[T](Task`1 task, HttpContext httpContext)
            at Memex.Portal.Shared.Authentication.OnboardingMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/OnboardingMiddleware.cs:line 166
            at Memex.Portal.Shared.Authentication.VirtualUserMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/VirtualUserMiddleware.cs:line 78
            at MeshWeaver.Hosting.AspNetCore.Portal.UserContextMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.AspNetCore/Portal/UserContextMiddleware.cs:line 277
            at Memex.Portal.Shared.Seo.PublicSite.<>c__DisplayClass7_0.<<UsePublicHostRedirect>b__0>d.MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Seo/PublicSite.cs:line 121
         --- End of stack trace from previous location ---
            at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
            at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
            at MeshWeaver.Hosting.AspNetCore.MeshModuleStaticAssetExtensions.<>c__DisplayClass6_0.<<UseMeshModuleStaticAssets>b__1>d.MoveNext() in /home/ru…[truncated]
   2026-09-21 10:24:24Z memex-portal-deployment-5998d5fddf-9sfsv fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
         An unhandled exception has occurred while executing the request.
         System.ObjectDisposedException: Hub portal/nodeops-wpWb1fSJPUacoVxY6LnltQ is shutting down — cannot register new response subject for 4ZLMfgxbpE6yNxRonoKEZg.
         Object name: 'MessageHub'.
            at Memex.Portal.Shared.Api.WebhookInboxEndpoints.Deliver(String target, HttpRequest request, IMessageHub hub, IConfiguration config, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs:line 87
            at Microsoft.AspNetCore.Http.RequestDelegateFactory.ExecuteTaskResult[T](Task`1 task, HttpContext httpContext)
            at Memex.Portal.Shared.Authentication.OnboardingMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/OnboardingMiddleware.cs:line 166
            at Memex.Portal.Shared.Authentication.VirtualUserMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Authentication/VirtualUserMiddleware.cs:line 78
            at MeshWeaver.Hosting.AspNetCore.Portal.UserContextMiddleware.InvokeAsync(HttpContext context) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.AspNetCore/Portal/UserContextMiddleware.cs:line 277
            at Memex.Portal.Shared.Seo.PublicSite.<>c__DisplayClass7_0.<<UsePublicHostRedirect>b__0>d.MoveNext() in /home/runner/work/MeshWeaver/MeshWeaver/memex/Memex.Portal.Shared/Seo/PublicSite.cs:line 121
         --- End of stack trace from previous location ---
            at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
            at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
            at MeshWeaver.Hosting.AspNetCore.MeshModuleStaticAssetExtensions.<>c__DisplayClass6_0.<<UseMeshModuleStaticAssets>b__1>d.MoveNext() in /home/ru…[truncated]
   
3. rbuergi

   added

   area:hostingCore subsystem: hosting - deployments, rolls, the control planeCore subsystem: hosting - deployments, rolls, the control plane

   feature:graceful-shutdownFeature: graceful shutdownFeature: graceful shutdown

   sev:MMedium - secondary path broken, or easy workaroundMedium - secondary path broken, or easy workaround

   on Sep 21, 2026
4. rbuergi commented on Sep 21, 2026

   

   rbuergi

   on Sep 21, 2026

   Contributor

   More actions

   Cluster triage: this symptom has five distinct roots

   Reviewed together with #1540 #1547 #1548 #3243 #3593 #2833 #2480 #5064 #4859 #5136 #4847 #4883 #4888 #4961 — every one of them reports ObjectDisposedException with Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException() on top, or a teardown that did not finish. They are not one bug. The grouping, the discriminators and the evidence are committed as Doc/Architecture/DisposedScopeAndDyingHubs (added in #5159).

   Root R2 — identified, and deliberately NOT fixed yet. The two obvious fixes are both unsafe today.

   The mechanism is confirmed in code. HostedHubsCollection.GetHubWithOutcome answers a registry hit
   HostedHubOutcome.Available unconditionally, while a hosted hub leaves that registry only through a
   RegisterForDisposal callback that runs inside MessageHub.DisposeImpl — several phases after
   Dispose() set IsDisposing, and after the hub already refuses new work (GetOrAddResponseSubject
   answers exactly the ObjectDisposedException in this report from RunLevel >= ShutDown). Between those
   two points the lookup reports a hub that can serve nothing.

   The window is not bounded, which is what makes your evidence read the way it does: a teardown that
   wedges never reaches the callback at all (#4883 measured one sync/ hub holding its own ShutDown turn
   for 96+ seconds and still counting). Four deliveries to the same portal/nodeops-… identity across 55
   minutes on a pod that was otherwise serving is not a pod drain — the Kubernetes grace ceiling ends
   those — it is one hub that stopped working and was never evicted. The caller reaches it through
   NodeOperationExecutionHub, i.e. GetHostedHub(…, HostedHubCreation.Always).

   Why I did not ship either fix:

   • Refusing the dying hub (null + HostShuttingDown) reads correct and is not. 38 call sites use
     the two-argument GetHostedHub, which is documented to return non-null for Always and is
     dereferenced accordingly — several assign it straight into a non-nullable field. Refusing converts an
     attributable ObjectDisposedException into an unattributable NullReferenceException in code that
     works today, on the everyday recycle path.
   • Retire-and-replace (mint a fresh hub, keep the old one in the disposal join) is the self-healing
     answer and needs three removals to be value-conditional first, so a predecessor's teardown cannot
     erase its successor. Two are addressed; the third is the blocker:
     1. the hosted-hub registry entry — removed by KEY (see the caveat below);
     2. the routing service's local route — was removed by KEY in both MonolithRoutingService and
        OrleansRoutingService; fixed in fix(mesh): resolve while the scope is alive — the delete pipeline's own continuations (#5064) #5159 (a registration now removes what it
        registered, subscriptionReady included);
     3. the pod-hub cluster claim — and this one is not merely by-key, it is actively harmful.
        IPodHubGrain.Detach stamps a terminal Released tombstone on the address for ten minutes,
        which the router reads as NotFound + TargetUnserved — the verdict that triggers owner-side
        stream eviction — and a successor's Attach clears it only if it runs afterwards. So a
        predecessor's Detach landing after a successor's Attach would get the live hub evicted as a
        corpse. Strictly worse than the 500 this issue reports. That is a grain-contract change with a
        cross-repo surface and needs its own scope call.

   🚨 And a lead that may make all of the above moot, filed into triage rather than asserted here:
   HostedHubsCollection.Add — the only method that registers both the disposal-time registry removal
   and CloseScopeWhenDisposed — appears unreferenced in src/. The one insertion path in use, the
   per-address Lazy factory in GetHubWithOutcome, assigns the entry directly and does neither. If that
   is real rather than a call site my sweep missed, a hosted hub never leaves the registry at all and
   its scope is never closed — a simpler and stronger root for this issue than the phase window above, plus
   a per-hub scope leak. Not established: it contradicts the documented recycle flow and the
   scope-closing tests, so verify before building on it.

   Full write-up, including the three prerequisites and the discriminators that separate this from the
   other four roots: Doc/Architecture/DisposedScopeAndDyingHubs.
5. rbuergi commented on Sep 21, 2026

   

   rbuergi

   on Sep 21, 2026

   Contributor

   More actions

   Correction to my comment above, measured rather than estimated: the count of call sites that would begin receiving null if the lookup refused a dying hub is 25, not 38. The larger figure came from a grep that also counted comments, doc mentions and the declarations themselves.

   The argument is unchanged — 25 sites consume the null-forgiving two-argument GetHostedHub — and it is now specific instead of summarised: six of them assign the result straight into a non-nullable field or return it from a method with a non-nullable return type (Activity, PortalApplication, SessionHubFactory, PartitionStorageRouter, MeshNodeStreamCache, StaticRepoImporter). Those are where refusal would turn an attributable ObjectDisposedException into an unattributable NullReferenceException.

   Also aligned: this issue's evidence is four occurrences at three distinct times (09:01, 09:49, 09:56) spanning 55 minutes; the doc page said "three deliveries" and now matches the evidence table here. Corrected in Doc/Architecture/DisposedScopeAndDyingHubs by commit b978623488 on #5159.
6. rbuergi commented on Sep 22, 2026

   

   rbuergi

   on Sep 22, 2026

   Contributor

   More actions

   Fixed at the lookup, and the three removals are now all value-matched — #5199.

   The two obvious fixes this thread ruled out stay ruled out. What landed is the third one the page named, once its prerequisites were in:

   • HostedHubsCollection.GetHubWithOutcome no longer answers a registered hub at RunLevel >= ShutDown to an Always caller. It retires it (value-matched removal, so a successor a concurrent lookup registered is untouched), mints a successor through the ordinary single-flight creation, and keeps the corpse in the collection's disposal join until its own DisposalCompleted — its scope is a child of the owner's, and the owner must not close it under a registrant walk still running. One Warning line, [HOSTED-RETIRE], names the address and phase; the ordinary path never reaches it.
   • The bound is ShutDown, not IsDisposing, on purpose: below ShutDown the hub is still draining accepted work and the intake's transient ShuttingDown NACK is the documented answer; a successor minted there would overlap accepted writes on the first activation. At ShutDown nothing accepted remains — the measured shape here (cannot register new response subject) is exactly that phase.
   • Prerequisite 3: PodHubGrain.Detach leaves the claim alone when the silo still holds a live local route for the address — RegisterStream's disposal removes its route before it releases the claim, so a live route at that point can only be a successor's. No grain-contract change. Two more key-only removals found in the sweep are gone too: MonolithRoutingService.UnregisterStream(address) (armed on every route-created hub, in the same registrant walk) and the podHubClaimSettled seam.

   Controls, measured (xUnit v3 host run directly; the hub dll's mtime confirmed after the source's on each side):

   test	fixed	control
   AHubAtShutDownIsNotHandedOutTest (parks the ShutDown turn inside a reactive dispose action, asserts the corpse is still registered)	total 2, failed 0	unconditional registry hit restored: total 2, failed 2 — both on "Did not expect value to refer to the same instance", i.e. the lookup handed back the corpse
   PodHubStaleReleaseTest (real two-silo cluster)	total 2, failed 0	route check removed from Detach: total 2, failed 1 — the successor's delivery threw PodHubNotHereException … was RELEASED by its owner; the honest-release case passed on both sides, as it must

   What this does not fix: the wedge itself. A hub parked at ShutDown for 55 minutes is #4883's shape; this lets callers past it and names it, it does not free the turn. That stays on #4883.

   Closing on the merge: the fix is compiled core code, it ships in the next image, and this thread carries no post-fix recurrence. Reopen if a cannot register new response subject fault from a hosted-hub address recurs on an image carrying this commit without a [HOSTED-RETIRE] line for that address preceding it — that would mean a path that hands out a corpse this lookup does not see.
7. rbuergi

   closed this as completedin #5199on Sep 22, 2026
8. rbuergi

   added a commit that references this issue on Sep 23, 2026

   fix(compile): resolve the settle's survivor per use, never hold it (r…

   ...

   36a8ada
9. rbuergi

   added a commit that references this issue on Nov 5, 2026

   fix(orleans): the release-in-flight marker is a set of per-registrati…

   ...

   f5783b5