RoutingGrain dispatch pool saturates with all 64 slots in flight; one destination holds 62 queued legs (head-of-line block), fleet-wide today

What is failing

The Orleans routing grain's dispatch pool saturates: route dispatch slots are held from dispatch until each leg terminates, and all 64 slots go in flight at once, crossing the reporting threshold. Both portal pods hit it within the same 28 seconds, and today's log is full of sibling back-pressure episodes (dozens of separate incidents between 11:41 and 13:17 UTC) plus OutOfMemoryException route dispatch faults at 11:51 — this is one systemic saturation event, not an isolated burst.

Probable cause

Moderate-to-high confidence that at least one leg is head-of-line blocked on a single destination: the third sample (activation fcf31578, episode #14) reports a deepest per-destination queue of 62 of 64 while only 1 destination is queued — nearly the whole in-flight budget is stacked behind one leg targeting cache/12xX8.... The same activation's earlier episode #13 showed deepest queue 0 and drained (a higher episode followed), so #13 was load; #14 has the shape of a stuck or extremely slow destination. The coincident OutOfMemoryException incidents on the route dispatch path (Admin/_LogIncident/0feb3c48c04cc4f1, 9539b56b6384fadb, ac4d85f3e9dce876) suggest the silo was already memory/CPU-starved, and the log's own guidance notes a slot is held through the unbounded ThreadPool wait before a leg's timeouts start — a starved silo can hold all 64 slots with nothing logically stuck. Which of these two dominates is not decidable from this incident alone; checking whether episode #14 was ever followed by a higher episode on the same activation (slot leak) vs. a drain would settle it.

Impact

Message routing through the affected activation stalls at saturation; delivery latency for everything routed behind the blocked leg grows. Only 3 occurrences on 2 pods in this incident, but the sheer number of sibling episodes today across the fleet makes the aggregate impact real: routing capacity, not a single tenant, is the casualty. Critical severity is justified by the fleet-wide pattern, not by this burst alone.

Where to look

The emitting component is the Orleans routing grain (MeshWeaver.Hosting.Orleans.RoutingGrain, the [ROUTE] back-pressure log site). Start from the leg dispatch/slot-management code there — specifically (1) whether the per-destination queue of ~62 legs for a cache/... target indicates a single slow consumer, (2) whether the ThreadPool-wait-before-timeout window lets starvation pin all 64 slots, and (3) whether episode #14 on activation fcf31578 was followed by a higher episode or a clear (slot leak vs. load). The OOM incidents from 11:51 should be treated as the likely upstream trigger.

Fingerprinting note

The masking rule folds the activation id and episode number into {n}, so every activation produces a separate incident at this site — hence dozens of near-identical incidents today. Consider masking the activation id out entirely so one routing saturation event files one ticket; a human will need to close ~30 near-duplicates by hand otherwise.

Evidence


Fingerprint	`a5a6d58b5ec03919`
Category	`MeshWeaver.Hosting.Orleans.RoutingGrain`
Severity	Critical
Namespace	`memex-cloud`
Pods	`memex-portal-deployment-69956b6dbc-gx6z6`, `memex-portal-deployment-69956b6dbc-ndnxt`
Occurrences	3
First seen	2026-09-19 14:03:19Z
Last seen	2026-09-19 14:03:47Z

Recent log lines

2026-09-19 14:03:19Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [fcf31578#13 started 2026-09-19T14:03:19.7093744Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Essentials/Email — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 14:03:47Z memex-portal-deployment-69956b6dbc-ndnxt crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [c0cc8549#7 started 2026-09-19T14:03:47.8235150Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Hosting/_Entitlements/_Policy — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 14:03:47Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [fcf31578#14 started 2026-09-19T14:03:47.8315643Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/12xX8OXQIEmdwvf_ZZ8LjA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

_{Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/a5a6d58b5ec03919. Recurrences are folded into this issue rather than opening new ones.}

Still occurring: 3 more since the last update (6 total).

Last seen: 2026-09-19 14:19:27Z
Pods: memex-portal-deployment-69956b6dbc-gx6z6, memex-portal-deployment-69956b6dbc-ndnxt

Recent log lines

2026-09-19 14:19:27Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [fcf31578#17 started 2026-09-19T14:19:27.6361802Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Store/Tier — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 14:19:27Z memex-portal-deployment-69956b6dbc-ndnxt crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [c0cc8549#10 started 2026-09-19T14:19:27.6730308Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/vyK5n1IlsUqe7j8OHpW-Sg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 14:19:27Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [fcf31578#18 started 2026-09-19T14:19:27.9093312Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Underwriting/Submission — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

_{Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Closing as a duplicate of #4798.

This issue and 46 others were filed today by one log site — MeshWeaver.Hosting.Orleans.RoutingGrain's back-pressure line — because the Orleans activation id in that line was not masked before the fingerprint was taken. LogLineParser.HexBlob required sixteen hex characters; an activation id is eight, so it fell through to the number masker, which stripped its digits and left its letters behind as literal text. An activation id changes on every grain recycle and every pod roll, and memex-cloud rolled repeatedly today, so each activation minted a new fingerprint, a new incident and a new issue.

These are not 47 defects. Under correct masking all 47 normalize to one message and would have been one incident. Verified two of them against the control instance's own records: Admin/_LogIncident/5b6a59648c91fd85 and eb29857024b3f0ed differ only in cfb5d6 vs c0cc8 — the dispatch target is already {path} in both, and every count is already {n}.

Fixed in Systemorph/MeshWeaver.Plugins#2165 (merged), which masks a run of ≥ 8 hex characters carrying both a digit and an a–f letter. The fault itself — routing back-pressure at the 64-dispatch threshold — remains open and is tracked on #4798, which carries the longest occurrence history.

Reopen if you believe this one is a distinct fault rather than another activation of the same line.

Reopened — this fault is happening again.

Still occurring: 11 more since the last update (17 total).

Last seen: 2026-09-19 15:52:19Z
Pods: memex-portal-deployment-69956b6dbc-gx6z6, memex-portal-deployment-69956b6dbc-ndnxt

Recent log lines

2026-09-19 15:28:19Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [fcf31578#31 started 2026-09-19T15:28:19.7068309Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target ClaudeCode/_Issue/1785 — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 15:30:19Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [fcf31578#32 started 2026-09-19T15:30:19.7002380Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Reinsurance/ModelledReinsuranceSection — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 15:52:19Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [fcf31578#38 started 2026-09-19T15:52:19.7136653Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target ILS/Tranche — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

_{Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Closes as duplicate of #5009 — Same shape behind cache/12xX8OXQIEmdwvf_ZZ8LjA; destination migrated to §1 of the canonical. No other reading in it that the canonical lacks.

Navigation Menu

Repository navigation

RoutingGrain dispatch pool saturates with all 64 slots in flight; one destination holds 62 queued legs (head-of-line block), fleet-wide today #4899

Evidence

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

RoutingGrain dispatch pool saturates with all 64 slots in flight; one destination holds 62 queued legs (head-of-line block), fleet-wide today #4899

Description

Evidence

Activity

systemorph commented on Sep 19, 2026

rbuergi commented on Sep 19, 2026

systemorph commented on Sep 19, 2026

rbuergi commented on Sep 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions