Repository navigation
RecycleCore posts DisposeRequest from the root mesh hub instead of the NodeOperationIssuingHub seam #4464
Description
What is failing
MeshOperations.RecycleCore posts its DisposeRequest directly on the root mesh hub (the DI-injected IMessageHub, which IS the router), making the router an end of a work delivery. This is the defect class that puts node-hub lifecycle work on the router's action block and competes with routing — the same shape as the 2026-06-11 prod portal wedge and issues #1113/#1121/#1136/#1140.
Probable cause (high confidence)
The RecycleCore lambda at MeshOperations.cs:4091 builds and posts the DisposeRequest without first hopping onto MeshExtensions.NodeOperationIssuingHub(). The origin-site stack names it unambiguously, and this is exactly the caller class the architecture doc calls out (a mesh-singleton operations service holding the root hub). The ratchet guard (RouterAsNodeOperationOriginRatchetGuard) only keys on node-CRUD request types — DisposeRequest is not among them — so this site was invisible to the guard and only the origin detector caught it. The recycle path was recently reworked to defer behind running installs (see Doc/WhatsNew/2026-09-13-recycling-a-node-waits-for-an-install-writing-under-it), so the un-hopped post may well have come in with that change.
Impact
One occurrence on one pod so far — low because the origin site reports only once per (role, type) per hub lifetime. It is not routine noise: every recycle issued through the operations surface posts this way, and the underlying risk is router starvation under a burst of recycles. The fix is a no-op for correctness (the seam is the identity function for any non-mesh hub).
Where to look
MeshWeaver.AI.MeshOperations,RecycleCore— the lambda at MeshOperations.cs:4091 that posts theDisposeRequest.- Fix: issue the
DisposeRequestviahub.NodeOperationIssuingHub()before posting.MeshServiceis the reference shape (hops both issuing hub and target). - Rulebook: Doc/Architecture/RouterTrafficDetection.
- Consider extending the ratchet to cover
DisposeRequestso this site cannot regress.
Evidence
| Fingerprint | fea87cd10adfd05b |
| Category | MeshWeaver.Messaging.MessageHub |
| Severity | Error |
| Top frame | MeshWeaver.AI.MeshOperations+<>c__DisplayClass95_0.<RecycleCore>b__3 (MeshOperations.cs:4091) |
| Namespace | memex |
| Pods | memex-portal-deployment-7cb6684584-ztqz8 |
| Occurrences | 1 |
| First seen | 2026-09-16 01:10:13Z |
| Last seen | 2026-09-16 01:10:13Z |
Recent log lines
2026-09-16 01:10:13Z memex-portal-deployment-7cb6684584-ztqz8 fail: MeshWeaver.Messaging.MessageHub[0]
ROUTER_TRAFFIC ORIGIN: DisposeRequest was POSTED with the mesh hub as sender (sender: mesh/Mw6ZXCJcUEmJTXAXoeuGxw, target: Hosting/Triage/ci-failure/systemorph-meshweaver-35041090763). The mesh hub is the ROUTER and must not be an end of a work delivery — hop off it with MeshExtensions.NodeOperationIssuingHub() (writes) or MeshExtensions.ReadIssuingHub() (reads) before posting. Reported once per role+type for this hub. Call site:
at MeshWeaver.AI.MeshOperations+<>c__DisplayClass95_0.<RecycleCore>b__3 (MeshOperations.cs:4091)
at System.Reactive.Linq.ObservableImpl.Select`2+Selector+_.OnNext
at System.Reactive.Linq.ObservableImpl.Timeout`1+Relative+_.OnNext
at System.Reactive.Linq.ObservableImpl.Take`1+Count+_.OnNext
at System.Reactive.SafeObserver`1+WrappingSafeObserver.OnNext
at MeshWeaver.Messaging.AccessContextCaptureExtensions+RestoringObserver`1.OnNext (AccessContextCaptureExtensions.cs:210)
at System.Reactive.Subjects.FastImmediateObserver`1.EnsureActive
at System.Reactive.Subjects.ReplaySubject`1+ReplayBase.Subscribe
at System.ObservableExtensions.SubscribeSafe
at System.Reactive.Linq.ObservableImpl.Select`2+Selector.Run
at System.Reactive.Producer`2.SubscribeRaw
at System.ObservableExtensions.SubscribeSafe
Opened automatically from Admin/_LogIncident/fea87cd10adfd05b. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 16, 2026
Duplicate of #4463 — the same incident was filed twice, in the same second.
Both issues carry the identical title, the identical body, and the identical source incident
Admin/_LogIncident/fea87cd10adfd05b, both created at2026-09-16T01:11:58Z. Closing this one and keeping #4463, which has the lower number.🚨 Worth a look on its own: the filer is supposed to fold a recurrence onto the existing issue by fingerprint — that is the whole mechanism — and here it created two issues for one fingerprint at the same instant. That looks like a concurrency gap in the file-or-fold step (two pods, or two passes, both finding no existing issue and both creating one) rather than anything about the defect being reported. Same family as Systemorph/MeshWeaver.Plugins#1796, which is already about incident identity not doing what the folding assumes.
The defect itself is tracked on #4463.
rbuergi commentedon Sep 17, 2026
Still a duplicate of #4463, and #4463 is fixed on
main. The new occurrence is from an image thatpredates that fix — not a regression.
#4477 ("issue the recycle teardown off the router, and derive the lifecycle guard's denominator",
merged 2026-09-16 07:59Z) hopped
RecycleCore'sDisposeRequestontohub.NodeOperationIssuingHub()and closed #4463. It is on
maintoday, and it is pinned at RUNTIME as well as in source —RouterTrafficOnNodeCreateFromTheRootHubTest.AnOperatorRecycleIssuedFromTheRootMeshHub_NeverPostsTheTeardownAsTheRouterdrives
new MeshOperations(Mesh).Recycle(path)from the ROOT hub and requires the detector to staysilent (reverted in rehearsal it reproduces this issue's line byte for byte, down to
<RecycleCore>b__3).Why it fired again on 2026-09-17 12:57:30Z. Neither portal is running an image that contains the
fix (measured just now against
/api/version, ancestry checked withgit merge-base --is-ancestor):3385e18fb7)?afde4eabe0740ff658f3dae8a3073c33a2b3ea02c84c6c05503228860df03c4a8b596e497e6d218cThe older line in this thread (2026-09-16 01:10:13Z) also predates the merge by ~7 hours, so both
samples are pre-fix by construction.
Re-closing as a duplicate of #4463. What is actually outstanding is operational: roll the portals
onto a current image.
Not the same for the three siblings filed in that burst — #4614 / #4615 / #4617 are one live
defect (the router as a stream SUBSCRIBER, which no seam adopted so far reaches) and are being fixed.
rbuergi commentedon Sep 17, 2026
🚩 Expect this to recur until the portals roll, and read those recurrences as delivery lag rather than as a regression. The fix (#4477, merged 2026-09-16 07:59Z, tracked on #4463) is on
main, but memex.systemorph.com runsafde4ea(2026-09-15 21:50Z) and memex.meshweaver.cloud runsc84c6c0(2026-09-12 09:30Z) — neither contains it. Every occurrence reported from those images is pre-fix code doing what it did before. Staying closed is deliberate: the code question is answered, and the outstanding item is operational, not a defect to track here.rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-09-17)Reopened — this fault is happening again.
Still occurring: 1 more since the last update (2 total).
memex-portal-deployment-7cb6684584-ztqz8,memex-portal-deployment-6c967c7c4c-kdff5Recent log lines