Repository navigation
Routing saturation episode on portal pod: head-of-line channel queue (depth 12) with ThreadPool slot starvation during Ifrs17 dispatch #5277
Description
What is failing
The portal silo's RoutingGrain crossed its saturation threshold (64 in-flight route dispatches) during an Ifrs17 stream dispatch burst. This is the single most-ticketed log site in the platform and the line is a gauge, not a bound — but this particular sample carries two routing-relevant signatures, not just load.
Probable cause — medium confidence
The sample shows a genuine head-of-line blocking signature (deepest per-channel queue 12 — frames of one stream stacking behind one another in the ordered (destination, stream) channel) combined with a ThreadPool thread shortage (waiting for a pool slot 15 while routing pool subscribing 0 — work the pool has accepted but that has not reached the gate, per IIoPool.CurrentlyWaiting). The oldest leg (18.2 s, named dispatch → Ifrs17, delivery ipk1vCA-…) is seconds old, not minutes, so this reads as a slow-but-draining episode rather than a leaked slot. Rule the process out first: check the same pod/window for an OOM or a LocalSiloHealthMonitor thread-pool stall — if present, this line is a symptom of that, not a routing defect.
Impact
Single occurrence on one portal pod over a 5-hour window — a transient saturation episode, not an outage. Stream deliveries to the Ifrs17 destination were delayed by the channel head-of-line wait during the episode; no evidence of lost or leaked slots.
Where to look
- Reading a Routing Saturation Report — the field-by-field reading this verdict follows
- Ordered Route Channels — the
(destination, stream)channel key and the head-of-line verdict - Controlled I/O Pooling —
IIoPool,CurrentInFlightvsCurrentlyWaiting RoutingGrain.SaturationThresholdgates only this log call; the in-flight count of 64 is a latch artefact, not a cap
Known-incident note
This log site has many sibling incidents (≥20 routing back-pressure incidents in Admin/_LogIncident at triage time) and a documented consolidation history (~87 filings folded into 3 tickets). If a standing ticket for head-of-line on Ifrs17 stream channels exists, this is likely a duplicate datapoint for it rather than a new defect.
Evidence
| Fingerprint | 38a33956262a9554 |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-5c89c7f5f9-sddb5 |
| Occurrences | 1 |
| First seen | 2026-09-22 12:44:25Z |
| Last seen | 2026-09-22 12:44:25Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-22 12:44:25Z memex-portal-deployment-5c89c7f5f9-sddb5 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [5bdfaaf2#6 started 2026-09-22T12:44:25.8225946Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 9 over 5 stream destination(s), deepest per-channel queue 12, routing pool subscribing 0, waiting for a pool slot 15; oldest leg in flight 18234 ms — dispatch → Ifrs17 (delivery ipk1vCA-I0mQF8H19I2MrQ). Latest dispatch target portal/EEb_9nhrQGguYeGDpB2Kkn1V-62htLU60NlMmkqL5rs — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so like…[truncated]
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/38a33956262a9554. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 22, 2026
Closing as a duplicate of #5118 — the fingerprint-split flood.
Why this is a duplicate. Every one of these comes from the same log site —
[ROUTE] Routing back-pressure, categoryMeshWeaver.Hosting.Orleans.RoutingGrain— split into one issue per grain activation. The 8-hex activation id survived normalization (a0e2984b→a0e2{n}b):HexBlobrequired 16+ chars, andNumber()then masked only the digit-run preceded by a digit, leaving a per-activation residue in the hash.Measured over 101 verbatim bursts drawn from 29 issues, covering 42 activation ids: the old parser yields 41 distinct fingerprints, the current parser yields 1.
The fix is already merged in MeshWeaver.Plugins — #2165 (
MixedHexId), #2167, #2168 (positional episode/target masking) and #2218 (the portal re-parses raw samples, instead of hashing text a stale watcher already masked).It is not live yet. The control instance is sealed on an older set because the Plugins Portal image rebuild workflow fails at its GitHub App token step —
actions: writeis not granted to the installation — so core's rebuild is never dispatched. Until that is granted, new split issues will keep arriving, and an issue whose activation is still alive may reopen here.Not closed by this. The underlying routing condition — cache-destination head-of-line blocking versus ThreadPool starvation — is a real defect and stays open in #5118. The normaliser itself is tracked in #5270 and #5271.
rbuergi commentedon Sep 22, 2026
Why this is its own issue, and what stops the next one — posted on each RoutingGrain back-pressure incident issue filed 2026-09-22; not a disposition of this one.
This incident's fingerprint was split from its siblings by two values the incident normaliser did not mask — the delivery id and a single-segment destination in the line's
oldest leg in flight … — dispatch → <address> (delivery <id>)field — not by the episode stamp, which was already masked. Fixed in Systemorph/MeshWeaver.Plugins#2286 (merged 2026-09-22T21:46Z asb922f070fc): over the 146 back-pressure samples quoted in the open core issues (43 filed fingerprints), the identity now yields 3 (one per leg kind, plus the pre-#5151 wording). It reaches production with the portal roll; existing incidents re-address onto their successors throughLogIncidentCorpusMigration.What the crossing itself means is a separate question and is not answered by the fix above. A line reading high
waiting for a pool slotwithrouting pool subscribing 0is legs the ThreadPool had not yet run (the pool's cap is 256 and no permit is held — see #5261 (comment)); that starvation root is being investigated separately (family: #5018, #4886). A line with a non-zerodeepest per-channel queueon acache/…destination and an old oldest leg is #5009's falsifier and should be raised there.Disposition stays with triage: a symptom gauge can invert the causal arrow, so this issue's own sample may carry evidence its siblings do not.