Repository navigation
RoutingGrain leaks a dispatch slot when a route leg to a portal destination never terminates (episode 7c973d7f never drained) #5134
Description
What is failing
A RoutingGrain episode on the memex portal silo crossed the reporting threshold (64 route dispatches in flight) and, judging by the retained logs, never drained: no cleared line and no higher-episode line for activation 7c973d7f follows the crossing. The queue shape at the crossing — 2 destinations queued with a deepest per-destination queue of 41 — is a head-of-line blockage on a single destination (the portal/nodeops-wpWb1fSJPUacoVxY6LnltQ hub), not spread-out load.
Probable cause
A route leg dispatching to that nodeops destination never terminated, so its slot was held indefinitely and the in-flight count never fell below half the threshold — the log's own episode-stamp guidance says exactly this pattern means a slot leak rather than a busy silo. Medium-high confidence on the slot-never-released part; the underlying reason the leg hangs (destination hub unresponsive vs. cancellation path not releasing the slot) is not determinable from the log line alone. Note this component has prior history of exactly this defect class: the 2026-08-12 fix ("Routing back-pressure alerts now name what they actually saw", Doc/WhatsNew/2026-08-12-routing-alerts-name-what-they-actually-saw) already fixed one slot leak where a delivery cancelled at shutdown never reported completion — worth checking whether other cancellation/exception paths still bypass slot release. There are also repeated [ROUTE] Directed delivery ... was refused lines for portal hub destinations in the same log stream (see e.g. Ops/Logs/memex-1789416067209272236-6b6b44-klrt7) from the 2026-09-14 burst — a refused delivery that leaves its slot behind would produce precisely this signature.
Impact
Single occurrence on a single pod (memex-portal-deployment-5998d5fddf-75t7b) at 2026-09-21 09:48 UTC — a stuck routing slot on one activation, degrading mesh delivery to that destination until the grain is recycled. The broader fingerprint family (RoutingGrain back-pressure) fires many times a day across pods, so a real leak here has cumulative impact, but this specific episode alone is a contained degradation, not an outage.
Where to look
MeshWeaver.Hosting.Orleans.RoutingGrain — the slot bookkeeping from dispatch to leg termination, especially the cancellation/refusal paths (delivery refused by a destination hub, delivery cancelled, silo shutdown), comparing against the already-fixed shutdown path from the 2026-08-12 change. Reproduce by refusing deliveries from a destination hub and verifying the in-flight count returns to zero.
Evidence
| Fingerprint | 8021dd7d44ea6640 |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex |
| Pods | memex-portal-deployment-5998d5fddf-75t7b |
| Occurrences | 1 |
| First seen | 2026-09-21 09:48:24Z |
| Last seen | 2026-09-21 09:48:24Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-21 09:48:24Z memex-portal-deployment-5998d5fddf-75t7b crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [7c973d7f#1 started 2026-09-21T09:48:24.8371415Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 2, deepest per-destination queue 41, routing pool subscribing 0. Latest dispatch target portal/nodeops-wpWb1fSJPUacoVxY6LnltQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/8021dd7d44ea6640. Recurrences are folded into this issue rather than opening new ones.
Activity
systemorph commentedon Sep 21, 2026
Still occurring: 1 more since the last update (2 total).
memex-portal-deployment-5998d5fddf-75t7bRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph commentedon Sep 21, 2026
Still occurring: 1 more since the last update (3 total).
memex-portal-deployment-5998d5fddf-75t7bRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Sep 21, 2026
Closes as duplicate of #5009.
Reason: same defect —
RoutingGraindispatch slots saturate at the 64-in-flight reporting threshold with legs head-of-line blocked behind a single destination. Grouped on the discriminator the log line itself defines, not on the title: a deepest per-destination queue of 1 or more means legs are blocked behind a leg. This ticket measured deepest per-destination queue 41 of a 64-slot budget over 2 queued destination(s) — so the great majority of the budget was held behind one leg, which is #5009's subject and not the empty-queue load shape tracked separately at #5018.Evidence: namespace
memex, 1 occurrence(s), 2026-09-21 09:48:24Z → 2026-09-21 09:48:24Z, episode stamp(s)7c973d7f#1, latest dispatch target(s)portal/nodeops-wpWb1fSJPUacoVxY6LnltQ. Fingerprint8021dd7d44ea6640.Unique evidence — migrated to #5009: destination
portal/nodeops-wpWb1fSJPUacoVxY6LnltQ— the nodeops sibling of thecache/id in #5129–#5133. Episode7c973d7f#1is followed by neither a clear nor a higher episode, which by the log site's own rule means the leg never terminated and its slot leaked, rather than the silo being busy. This is the cleanest slot-leak sample in the flood.fix(routing): the saturation report prints the gauge that can see the…