RoutingGrain repeatedly hits 64-dispatch back-pressure threshold on memex-portal pods, with one episode head-of-line blocked on a single cache destination

What is failing

The RoutingGrain on the memex-portal deployment repeatedly crossed its 64-dispatch back-pressure threshold within a ten-minute window (three episodes across both pods). One episode is a real blockage signal; the other two are not:

Pod ndnxt, activation c0cc8549#3: 62 of 64 in-flight legs queued behind a single destination (destinations queued 1, deepest per-destination queue 62). That is head-of-line blocking on one leg — everything was waiting on the delivery to cache/vyK5n1IlsUqe7j8OHpW-Sg (or whatever that leg itself was waiting on).
Pod gx6z6, activation fcf31578, episodes [Snyk] Upgrade react-dropzone from 12.0.5 to 12.1.0 #9 and [Snyk] Upgrade @microsoft/signalr from 3.1.9 to 3.1.31 #10 ten minutes apart, both with destinations queued 0, deepest 0 — nothing was waiting on anything. Per the episode-stamp semantics, a higher episode on the same activation means each prior episode drained, so these are recurring short bursts, not stuck legs. With zero queuing and the slot-held-until-termination rule (including the unbounded ThreadPool wait), these crossings look like a CPU-starved or burst-loaded silo, not a routing defect.

Probable cause (moderate confidence, split evidence)

The ndnxt episode points at one destination's leg stalling long enough to park 62 others behind it — consistent with a slow/blocked pod hub under cache/… rather than a router bug.
The gx6z6 episodes are almost certainly load: episodes 9 and 10 on the same activation means this grain crosses the threshold regularly and recovers each time. Cause unclear beyond load; the evidence does not support a single root cause, and the two pods may not share one.

No episode shows the leaked-slot signature (no episode ever followed by a higher one on the same activation, no evidence here yet — the window is only ten minutes).

Impact

Low-to-moderate: 3 occurrences in 10 minutes, both portal pods affected, episodes self-draining. This is a recurring performance/burst pattern, not an outage — but the ndnxt head-of-line episode indicates delivery latency spikes of ~full backlog depth for whatever shares that destination.

Where to look

MeshWeaver.Hosting.Orleans.RoutingGrain (log category RoutingGrain[0]), specifically the leg-termination/slot-release path and per-destination queue.
For the ndnxt episode: the pod hub at cache/vyK5n1IlsUqe7j8OHpW-Sg and why its leg held 62 followers.
Background: Routing back-pressure alerts now name what they actually saw and Background services no longer run their work on the mesh router.
Check CPU/thread-pool saturation on memex-portal-deployment-69956b6dbc-* around 13:15–13:26 UTC before treating this as a router defect.

Evidence


Fingerprint	`eb29857024b3f0ed`
Category	`MeshWeaver.Hosting.Orleans.RoutingGrain`
Severity	Critical
Namespace	`memex-cloud`
Pods	`memex-portal-deployment-69956b6dbc-ndnxt`, `memex-portal-deployment-69956b6dbc-gx6z6`
Occurrences	3
First seen	2026-09-19 13:15:55Z
Last seen	2026-09-19 13:26:19Z

Recent log lines

2026-09-19 13:15:55Z memex-portal-deployment-69956b6dbc-ndnxt crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [c0cc8549#3 started 2026-09-19T13:15:55.1602190Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/vyK5n1IlsUqe7j8OHpW-Sg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 13:15:55Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [fcf31578#9 started 2026-09-19T13:15:55.4562498Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Reinsurance/Counterparty — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 13:26:19Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [fcf31578#10 started 2026-09-19T13:26:19.7170042Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Signature/Desk — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

_{Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/eb29857024b3f0ed. Recurrences are folded into this issue rather than opening new ones.}

Still occurring: 3 more since the last update (6 total).

Last seen: 2026-09-19 13:46:58Z
Pods: memex-portal-deployment-69956b6dbc-ndnxt, memex-portal-deployment-69956b6dbc-gx6z6

Recent log lines

2026-09-19 13:46:56Z memex-portal-deployment-69956b6dbc-ndnxt crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [c0cc8549#4 started 2026-09-19T13:46:56.6199221Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target RolePlay/Story — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 13:46:57Z memex-portal-deployment-69956b6dbc-ndnxt crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [c0cc8549#5 started 2026-09-19T13:46:57.1189116Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Hosting/ModuleInventory/_Activity/compile-state — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 13:46:58Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [fcf31578#11 started 2026-09-19T13:46:58.1159976Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/12xX8OXQIEmdwvf_ZZ8LjA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

_{Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Still occurring: 3 more since the last update (6 total).

Last seen: 2026-09-19 13:46:58Z
Pods: memex-portal-deployment-69956b6dbc-ndnxt, memex-portal-deployment-69956b6dbc-gx6z6

Recent log lines

2026-09-19 13:46:56Z memex-portal-deployment-69956b6dbc-ndnxt crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [c0cc8549#4 started 2026-09-19T13:46:56.6199221Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target RolePlay/Story — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 13:46:57Z memex-portal-deployment-69956b6dbc-ndnxt crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [c0cc8549#5 started 2026-09-19T13:46:57.1189116Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Hosting/ModuleInventory/_Activity/compile-state — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 13:46:58Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [fcf31578#11 started 2026-09-19T13:46:58.1159976Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/12xX8OXQIEmdwvf_ZZ8LjA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

_{Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Closing as a duplicate of #4798.

This issue and 46 others were filed today by one log site — MeshWeaver.Hosting.Orleans.RoutingGrain's back-pressure line — because the Orleans activation id in that line was not masked before the fingerprint was taken. LogLineParser.HexBlob required sixteen hex characters; an activation id is eight, so it fell through to the number masker, which stripped its digits and left its letters behind as literal text. An activation id changes on every grain recycle and every pod roll, and memex-cloud rolled repeatedly today, so each activation minted a new fingerprint, a new incident and a new issue.

These are not 47 defects. Under correct masking all 47 normalize to one message and would have been one incident. Verified two of them against the control instance's own records: Admin/_LogIncident/5b6a59648c91fd85 and eb29857024b3f0ed differ only in cfb5d6 vs c0cc8 — the dispatch target is already {path} in both, and every count is already {n}.

Fixed in Systemorph/MeshWeaver.Plugins#2165 (merged), which masks a run of ≥ 8 hex characters carrying both a digit and an a–f letter. The fault itself — routing back-pressure at the 64-dispatch threshold — remains open and is tracked on #4798, which carries the longest occurrence history.

Reopen if you believe this one is a distinct fault rather than another activation of the same line.

Reopened — this fault is happening again.

Still occurring: 30 more since the last update (36 total).

Last seen: 2026-09-19 15:41:43Z
Pods: memex-portal-deployment-69956b6dbc-ndnxt, memex-portal-deployment-69956b6dbc-gx6z6, memex-portal-deployment-69956b6dbc-s246c

Recent log lines

2026-09-19 15:41:38Z memex-portal-deployment-69956b6dbc-s246c crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [cfb5d696#6 started 2026-09-19T15:41:38.5714172Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 1. Latest dispatch target Ifrs17/IfrsVariable — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 15:41:38Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [fcf31578#36 started 2026-09-19T15:41:38.5855837Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 1. Latest dispatch target cache/YORFhwiyqEyTP9ESF-NUNg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 15:41:43Z memex-portal-deployment-69956b6dbc-s246c crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [cfb5d696#7 started 2026-09-19T15:41:43.5702598Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Ifrs17/RawVariable — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

_{Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Closes as duplicate of #5009 — Same head-of-line shape: 62 of 64 legs queued behind one cache/* destination. Its destination cache/vyK5n1IlsUqe7j8OHpW-Sg is migrated to the canonical's destination table (§1) — it is one of twelve, which is the evidence that this is not one bad hub.

Navigation Menu

Repository navigation

RoutingGrain repeatedly hits 64-dispatch back-pressure threshold on memex-portal pods, with one episode head-of-line blocked on a single cache destination #4897

Evidence

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

RoutingGrain repeatedly hits 64-dispatch back-pressure threshold on memex-portal pods, with one episode head-of-line blocked on a single cache destination #4897

Description

Evidence

Activity

systemorph commented on Sep 19, 2026

systemorph commented on Sep 19, 2026

rbuergi commented on Sep 19, 2026

systemorph commented on Sep 19, 2026

rbuergi commented on Sep 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions