Repository navigation
Slow dispatch leg to Manufacturing import activity holds a routing slot and backs up same-stream legs past the 64-dispatch threshold #5703
Description
What is failing
The Orleans RoutingGrain (MeshWeaver.Hosting.Orleans) crossed its 64-dispatch back-pressure threshold in a ~43 s burst on the memex portal. On two pods this was pure load — episode counters rose monotonically (#12→#14) with every leg under 60 ms, so those episodes drained. On the third pod the same dispatch leg (delivery FtDKJDuwBUWf_MqcA6BkNQ → Manufacturing/_Activity/import-520c9070a542c55a-…) aged 7.0 s → 9.6 s across two consecutive crossings while both pool gauges (subscribing 0, waiting for a pool slot 0→1) say the leg is past its subscribe and the wait is downstream I/O, not a thread or pool shortage.
Probable cause
Moderate confidence (~50%): a single stream-dispatch leg to a Manufacturing import activity grain stalled in downstream I/O for at least ~10 s, holding its pool slot (slots are held dispatch-to-termination, including unbounded pre-timeout waits) while same-stream legs queued behind it — per-channel depth 52→56, ordered channels 1→2, classic head-of-line within one channel per issue #5009's channel model. Two data points cannot distinguish a slow-but-alive leg from a slot that never returns; if no higher-episode or clear line ever followed on activation 2fa0fbe4, treat it as a leaked slot and investigate with #5018's gauge semantics in mind.
Impact
Momentary, self-limiting: 24 lines in 43 s across 3 pods, no further recurrences on this fingerprint since 2026-09-24 06:25 (two days ago). Routing was throttled only while the threshold was crossed; at least two pods absorbed the load cleanly.
Where to look
MeshWeaver.Hosting.Orleans.RoutingGrain — the dispatch path for Manufacturing/_Activity/import-* destinations, and the leg-termination/slot-release code. The RoutingGrain source was not found in the mesh, so no direct link; this is one of a large family of same-site incidents on 2026-09-24 (many sibling _LogIncident nodes) — a human should correlate before digging. Related prior work referenced by the log site itself: #5009 (per-channel split), #5018 (pool gauge semantics).
Evidence
| Fingerprint | fbd27b4e342e917c |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex |
| Pods | memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-76cf776847-9p7z4, memex-portal-deployment-76cf776847-5hflx |
| Occurrences | 29 |
| First seen | 2026-09-24 06:24:56Z |
| Last seen | 2026-09-24 06:27:01Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-24 06:25:29Z memex-portal-deployment-7596dd76bb-cbbj8 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [33f1d1f4#12 started 2026-09-24T06:25:29.0240853Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 2 over 2 stream destination(s), deepest per-channel queue 39, routing pool subscribing 0, waiting for a pool slot 0; oldest leg in flight 43 ms — stream-routed → import/rqEHlOPNSkCV1G34WXuQ9A (delivery 1BcX4kA050-NB9_DY_ns9Q). Latest dispatch target import/rqEHlOPNSkCV1G34WXuQ9A — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so …[truncated]
2026-09-24 06:25:36Z memex-portal-deployment-7596dd76bb-cbbj8 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [33f1d1f4#13 started 2026-09-24T06:25:36.0448227Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 2 over 2 stream destination(s), deepest per-channel queue 45, routing pool subscribing 0, waiting for a pool slot 0; oldest leg in flight 58 ms — stream-routed → import/rqEHlOPNSkCV1G34WXuQ9A (delivery hmm9Dv_Tykyri07pnh81JQ). Latest dispatch target import/jll3sSZfx061jLrYC3JaUA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so …[truncated]
2026-09-24 06:25:36Z memex-portal-deployment-56fbdcd48f-sz5st crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [2fa0fbe4#1 started 2026-09-24T06:25:36.0539164Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 1 over 1 stream destination(s), deepest per-channel queue 52, routing pool subscribing 0, waiting for a pool slot 0; oldest leg in flight 7020 ms — dispatch → Manufacturing/_Activity/import-520c9070a542c55a-20260924062515042-475933a2 (delivery FtDKJDuwBUWf_MqcA6BkNQ). Latest dispatch target portal/reads-jll3sSZfx061jLrYC3JaUA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anyth…[truncated]
2026-09-24 06:25:38Z memex-portal-deployment-56fbdcd48f-sz5st crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [2fa0fbe4#2 started 2026-09-24T06:25:38.6247363Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 2 over 2 stream destination(s), deepest per-channel queue 56, routing pool subscribing 0, waiting for a pool slot 1; oldest leg in flight 9591 ms — dispatch → Manufacturing/_Activity/import-520c9070a542c55a-20260924062515042-475933a2 (delivery FtDKJDuwBUWf_MqcA6BkNQ). Latest dispatch target portal/reads-jll3sSZfx061jLrYC3JaUA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anyth…[truncated]
2026-09-24 06:25:39Z memex-portal-deployment-7596dd76bb-cbbj8 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [33f1d1f4#14 started 2026-09-24T06:25:39.0158456Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 2 over 2 stream destination(s), deepest per-channel queue 60, routing pool subscribing 0, waiting for a pool slot 0; oldest leg in flight 28 ms — stream-routed → import/jll3sSZfx061jLrYC3JaUA (delivery qkQKBi0VJUKIDN2_YFDivA). Latest dispatch target import/jll3sSZfx061jLrYC3JaUA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so …[truncated]
2026-09-24 06:25:22Z memex-portal-deployment-76cf776847-9p7z4 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [cf7a46bc#13 started 2026-09-24T06:25:22.7104950Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 42 over 1 stream destination(s), deepest per-channel queue 1, routing pool subscribing 1, waiting for a pool slot 0; oldest leg in flight 5 ms — stream-routed → cache/rfD6RFrASE-1774FxdvuZA (delivery kVti59mbv0O4n2djxyEJPw). Latest dispatch target cache/rfD6RFrASE-1774FxdvuZA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so lik…[truncated]
2026-09-24 06:25:22Z memex-portal-deployment-76cf776847-5hflx crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [25890ef7#9 started 2026-09-24T06:25:22.9752512Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 39 over 1 stream destination(s), deepest per-channel queue 1, routing pool subscribing 1, waiting for a pool slot 0; oldest leg in flight 12 ms — stream-routed → cache/rfD6RFrASE-1774FxdvuZA (delivery SZ6PjQh8o02fH93ev3d8-A). Latest dispatch target cache/rfD6RFrASE-1774FxdvuZA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so lik…[truncated]
2026-09-24 06:25:23Z memex-portal-deployment-76cf776847-5hflx crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [25890ef7#10 started 2026-09-24T06:25:23.0479320Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 45 over 1 stream destination(s), deepest per-channel queue 1, routing pool subscribing 0, waiting for a pool slot 17; oldest leg in flight 12 ms — stream-routed → cache/rfD6RFrASE-1774FxdvuZA (delivery dNARH7-7p0qHiurt1k0esQ). Latest dispatch target cache/rfD6RFrASE-1774FxdvuZA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so l…[truncated]
2026-09-24 06:27:01Z memex-portal-deployment-76cf776847-9p7z4 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [cf7a46bc#14 started 2026-09-24T06:27:01.1426613Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 46 over 1 stream destination(s), deepest per-channel queue 1, routing pool subscribing 0, waiting for a pool slot 24; oldest leg in flight 25 ms — stream-routed → cache/MkvakB3S60a4rdhuPh7WLA (delivery D9e3gmUKWU2wg8RUf_bgnA). Latest dispatch target cache/MkvakB3S60a4rdhuPh7WLA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so l…[truncated]
2026-09-24 06:27:01Z memex-portal-deployment-76cf776847-5hflx crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [25890ef7#12 started 2026-09-24T06:27:01.3896363Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 52 over 1 stream destination(s), deepest per-channel queue 1, routing pool subscribing 0, waiting for a pool slot 24; oldest leg in flight 21 ms — stream-routed → cache/MkvakB3S60a4rdhuPh7WLA (delivery tNMoa_Nay0WsS3kzO6Q0IQ). Latest dispatch target cache/MkvakB3S60a4rdhuPh7WLA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so l…[truncated]
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/fbd27b4e342e917c. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 8b02af1dbffbfd6d: other fingerprints of this site fold in here as comments rather than opening tickets of their own.
Activity
systemorph commentedon Sep 26, 2026
Still occurring: 2 more since the last update (31 total).
memex-portal-deployment-7596dd76bb-cbbj8,memex-portal-deployment-56fbdcd48f-sz5st,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-76cf776847-5hflxRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph commentedon Sep 26, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
8b02af1dbffbfd6d(MeshWeaver.Hosting.Orleans.RoutingGrain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.981bb5aa56136a698b02af1dbffbfd6dMeshWeaver.Hosting.Orleans.RoutingGrain[ROUTE] Routing back-pressure [{n}f1d1f4#{n} started {time}]: {n} route dispatches in flight (reporting threshold {n}); ordered channels queued {n} over {n} stream destination(s), deepest per-channel queue {n}, routing pool subscribing {n}, waiting for a pool slot {n}; oldest leg in flight {n} ms — dispatch → {path} (delivery ydjs4Kv8B0OLHsFCgnn9hg). Latest dispatch target {path} — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of {n} or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); {n} means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #{n} those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #{n}. '{value}' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read {n} there; '{value}' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream {path} 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N channels sharing the backlog it is ~{path} whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.memexmemex-portal-deployment-7596dd76bb-cbbj8,memex-portal-deployment-56fbdcd48f-sz5stAdmin/_LogIncident/981bb5aa56136a69What triage wrote about this fingerprint
What is failing
The Orleans
RoutingGrain(MeshWeaver.Hosting.Orleans) crossed its 64-dispatch back-pressure threshold twice in a ~7 s window on the memex portal, on two different pods. Both lines show legs past their subscribe (subscribing 0/1,waiting for a pool slot 0), so the constraint is downstream, not a thread or pool shortage. This is the same log site, same 2026-09-24 ~06:25 burst, as already-filed issue #5703 (incidentfbd27b4e342e917c) and part of a large family of sibling[ROUTE]incidents from that morning — likely a duplicate; a human may close it as such.Probable cause
Moderate confidence (~50%): transient overload during a portal-wide routing burst on 2026-09-24 ~06:25Z. On the first pod (
33f1d1f4#15, pod 7596dd76bb) the oldest leg is 2.9 s with deepest per-channel queue 29 over 6 queued legs — young legs, load the silo was absorbing. On the second pod (2fa0fbe4#3, pod 56fbdcd48f) the oldest leg (deliveryYur2gUnikEmhb_dyvdN5nA→Store/_GitSync) aged 12.8 s with depth 40 over 8 queued legs — head-of-line blocking within one channel per the #5009 channel model, but with only a single line from this activation we cannot tell a slow-but-alive leg from a leaked slot (a higher episode or clear line never appeared in this incident's evidence). Cause beyond "the burst at 06:25" is unclear; evidence points at downstream I/O on a GitSync/import-adjacent dispatch.Impact
Minimal and transient: 2 occurrences on 2 pods over 7 s, no recurrences on this fingerprint since 2026-09-24 06:25 (two days ago). Routing was momentarily throttled while the threshold was crossed; nothing indicates a persistent leak.
Where to look
MeshWeaver.Hosting.Orleans.RoutingGrain— the dispatch path forStore/_GitSyncandHosting/Issuedestinations, and the leg-termination/slot-release code. The RoutingGrain source was not found in the mesh, so no direct link. Start from issue #5703, which covers the same burst with richer evidence, and correlate the sibling_LogIncidentnodes from 2026-09-24 before digging. Related prior work named by the log site itself: #5009 (per-channel split), #5018 (pool gauge semantics).Still occurring: 2 more since the last update (2 total).
memex-portal-deployment-7596dd76bb-cbbj8,memex-portal-deployment-56fbdcd48f-sz5stRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph commentedon Sep 26, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
8b02af1dbffbfd6d(MeshWeaver.Hosting.Orleans.RoutingGrain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.0beb991b4f0f46458b02af1dbffbfd6dMeshWeaver.Hosting.Orleans.RoutingGrain[ROUTE] Routing back-pressure [b5{n}b9{n}#{n} started {time}]: {n} route dispatches in flight (reporting threshold {n}); ordered channels queued {n} over {n} stream destination(s), deepest per-channel queue {n}, routing pool subscribing {n}, waiting for a pool slot {n}; oldest leg in flight {n} ms — dispatch → {path} (delivery WCgdS0XeaUaWlXOMU5gxuA). Latest dispatch target {path} — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of {n} or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); {n} means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #{n} those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #{n}. '{value}' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read {n} there; '{value}' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream {path} 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N channels sharing the backlog it is ~{path} whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.memexmemex-portal-deployment-7596dd76bb-cbbj8Admin/_LogIncident/0beb991b4f0f4645What triage wrote about this fingerprint
What is failing
The Orleans
RoutingGrain(MeshWeaver.Hosting.Orleans) crossed its 64-dispatch back-pressure threshold twice in 15 s on a single memex portal pod. Both lines are load, not a defect: the oldest leg in flight is 283 ms in the first crossing and 4 ms in the second — every leg young, i.e. dispatches the silo was absorbing. The first crossing shows 43 legswaiting for a pool slotwithsubscribing 0(pre-subscribe pool wait, per the issue #5018 gauge semantics), all queued over one stream destination with a per-channel depth of 1 — a fan-out of independent legs, not head-of-line. The second shows a deepest per-channel queue of 43 (head-of-line within one channel per the issue #5009 channel model) but with a 4 ms oldest leg, so the backlog was draining as fast as it formed.Probable cause
High confidence (~80%) this is a transient load spike — a burst of route dispatches (to a
Crm/Opportunitypublic-access destination and animport/*activity) momentarily exceeded the routing pool's 64 slots; the second line is a NACK-labelled crossing with a different activation id (8256909evsb560b916), and both episodes started fresh (#1). No leg aged past 300 ms, so there is no evidence of a leaked slot or stalled downstream I/O. Two lines 15 s apart on one pod two days ago, with no recurrence on this fingerprint since, support a one-off burst.Impact
Minimal: 2 occurrences, 1 pod, 15 s window, self-limited. Routing was throttled only while the threshold was crossed; no user-visible failure is evidenced.
Where to look
MeshWeaver.Hosting.Orleans.RoutingGrain— the dispatch pool sizing and back-pressure threshold (64). If this site recurs frequently under normal traffic, the pool size or threshold may deserve tuning rather than a code fix. This is one of a family of same-site back-pressure incidents on 2026-09-24 (siblingfbd27b4e342e917c→ issue #5703 covers a genuinely stalled Manufacturing import leg from the same burst; the present fingerprint is the benign load side of that same event). Prior referenced work: #5009 (per-channel split), #5018 (pool gauge semantics).Still occurring: 2 more since the last update (2 total).
memex-portal-deployment-7596dd76bb-cbbj8Recent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph commentedon Sep 26, 2026
Still occurring: 1 more since the last update (3 total).
memex-portal-deployment-7596dd76bb-cbbj8Recent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph commentedon Sep 26, 2026
Still occurring: 79 more since the last update (81 total).
memex-portal-deployment-7596dd76bb-cbbj8,memex-portal-deployment-56fbdcd48f-sz5st,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-8498855cbf-vchtf,memex-portal-deployment-6f9d6848f8-d9jwpRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph commentedon Sep 26, 2026
Still occurring: 164 more since the last update (195 total).
memex-portal-deployment-7596dd76bb-cbbj8,memex-portal-deployment-56fbdcd48f-sz5st,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-76cf776847-5hflx,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-8498855cbf-vchtf,memex-portal-deployment-75cd8dc74c-pckmh,memex-portal-deployment-6f9d6848f8-qskmnRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph commentedon Sep 26, 2026
Still occurring: 8 more since the last update (203 total).
memex-portal-deployment-7596dd76bb-cbbj8,memex-portal-deployment-56fbdcd48f-sz5st,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-76cf776847-5hflx,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-8498855cbf-vchtf,memex-portal-deployment-75cd8dc74c-pckmh,memex-portal-deployment-6f9d6848f8-qskmn,memex-portal-deployment-f8dd949d7-sggqz,memex-portal-deployment-f8dd949d7-b9kg8,memex-portal-deployment-959f6754d-x87flRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph commentedon Sep 26, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
8b02af1dbffbfd6d(MeshWeaver.Hosting.Orleans.RoutingGrain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.bab5b3e412f85ea58b02af1dbffbfd6dMeshWeaver.Hosting.Orleans.RoutingGrain[ROUTE] Routing back-pressure [{n}f0{n}c#{n} started {time}]: {n} route dispatches in flight (reporting threshold {n}); ordered channels queued {n} over {n} stream destination(s), deepest per-channel queue {n}, routing pool subscribing {n}, waiting for a pool slot {n}; oldest leg in flight {n} ms — dispatch → Markdown (delivery X3X3nKN1NECc5C8{n}qhoeMw). Latest dispatch target rsalzmann — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of {n} or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); {n} means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #{n} those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #{n}. '{value}' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read {n} there; '{value}' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream {path} 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N channels sharing the backlog it is ~{path} whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.memexmemex-portal-deployment-56fbdcd48f-7cd4hAdmin/_LogIncident/bab5b3e412f85ea5What triage wrote about this fingerprint
What is failing. The log-watch incident filer's message normaliser does not mask the per-episode identity that
MeshWeaver.Hosting.Orleans.RoutingGrainembeds in its Critical back-pressure line — the bracketed[activationId#episode started ISO-timestamp]group. Each threshold crossing therefore gets its own fingerprint and its own Critical incident node, instead of recurrences folding onto one incident with a growing occurrence count.Probable cause (high confidence, corroborated by measurement). The RoutingGrain line composes an 8-hex activation id, an episode counter and a tick-precision timestamp directly into the message text (call site:
src/MeshWeaver.Hosting.Orleans/RoutingGrain.cs, methodReportSaturation, per the investigation in rbuergi/Feedback/route-backpressure-fingerprint-splits-20260921-2236). The normaliser has no pattern for that bracketed group. This is not hypothesis-only: core issue 5018's consolidation comments record seven separate GitHub issues (5017, 5019–5022, 5060, 5018) all carrying the same fingerprint 54ecfdaea23fd110 for the same incident window, and roughly 87 bot-filed tickets for this one log site between 2026-09-19 and 2026-09-21, of which 48 were hand-folded into core issue 5009 and 24 into 5018. The Admin incident partition currently holds 50+ distinct "Routing back-pressure" incident nodes for the same site — this ticket is one of them. Note the normaliser fix, not the log line, is the right lever: the episode stamp is load-bearing diagnosis (it tells a single line whether an episode drained or the grain was recycled).Impact. Observability noise, not a user-facing outage: a real slot-leak signal is buried among dozens of near-identical Criticals, and every crossing costs a triage round. The specific crossing behind this incident reads as absorbed load, not a leak: 64 dispatches in flight at threshold 64, 16 queued over 1 stream destination, deepest per-channel queue 2, both pool gauges at 0, oldest leg 1170 ms — all legs young, wait downstream I/O. One occurrence on one pod over the window.
Where to look. The incident filer's message normaliser in MeshWeaver.Plugins — add a masking pattern for the
[activation#episode started timestamp]group so recurrences fold onto one incident and the occurrence count becomes the signal. Related core issues: 5118 (the alert-flood verdict; the threshold-raise/demote band-aids were refused there), 5018 (load verdict, carries the seven-issues-one-fingerprint measurement), 5009 (head-of-line verdict, fixed in core PR 5151, awaiting production verification).Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-56fbdcd48f-7cd4hRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph commentedon Sep 26, 2026
Still occurring: 115 more since the last update (318 total).
memex-portal-deployment-7596dd76bb-cbbj8,memex-portal-deployment-56fbdcd48f-sz5st,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-76cf776847-5hflx,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-8498855cbf-vchtf,memex-portal-deployment-75cd8dc74c-pckmh,memex-portal-deployment-6f9d6848f8-qskmn,memex-portal-deployment-f8dd949d7-sggqz,memex-portal-deployment-f8dd949d7-b9kg8,memex-portal-deployment-959f6754d-x87fl,memex-portal-deployment-745745db7c-b2gjn,memex-portal-deployment-56fbdcd48f-7cd4h,memex-portal-deployment-589ff8f895-ghgjt,memex-portal-deployment-589ff8f895-vn49b,memex-portal-deployment-56fbdcd48f-cbms9Recent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph commentedon Sep 26, 2026
Still occurring: 5 more since the last update (145 total).
memex-portal-deployment-7596dd76bb-cbbj8,memex-portal-deployment-56fbdcd48f-sz5st,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-8498855cbf-vchtf,memex-portal-deployment-6f9d6848f8-d9jwp,memex-portal-deployment-f8dd949d7-b9kg8,memex-portal-deployment-589ff8f895-ghgjt,memex-portal-deployment-745745db7c-b2gjn,memex-portal-deployment-56fbdcd48f-7cd4h,memex-portal-deployment-56fbdcd48f-cbms9Recent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph commentedon Sep 26, 2026
Still occurring: 68 more since the last update (386 total).
memex-portal-deployment-7596dd76bb-cbbj8,memex-portal-deployment-56fbdcd48f-sz5st,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-76cf776847-5hflx,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-8498855cbf-vchtf,memex-portal-deployment-75cd8dc74c-pckmh,memex-portal-deployment-6f9d6848f8-qskmn,memex-portal-deployment-f8dd949d7-sggqz,memex-portal-deployment-f8dd949d7-b9kg8,memex-portal-deployment-959f6754d-x87fl,memex-portal-deployment-745745db7c-b2gjn,memex-portal-deployment-56fbdcd48f-7cd4h,memex-portal-deployment-589ff8f895-ghgjt,memex-portal-deployment-589ff8f895-vn49b,memex-portal-deployment-56fbdcd48f-cbms9Recent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph commentedon Sep 26, 2026
Still occurring: 7 more since the last update (152 total).
memex-portal-deployment-7596dd76bb-cbbj8,memex-portal-deployment-56fbdcd48f-sz5st,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-8498855cbf-vchtf,memex-portal-deployment-6f9d6848f8-d9jwp,memex-portal-deployment-f8dd949d7-b9kg8,memex-portal-deployment-589ff8f895-ghgjt,memex-portal-deployment-745745db7c-b2gjn,memex-portal-deployment-56fbdcd48f-7cd4h,memex-portal-deployment-56fbdcd48f-cbms9Recent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph commentedon Sep 26, 2026
Still occurring: 1 more since the last update (387 total).
memex-portal-deployment-7596dd76bb-cbbj8,memex-portal-deployment-56fbdcd48f-sz5st,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-76cf776847-5hflx,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-8498855cbf-vchtf,memex-portal-deployment-75cd8dc74c-pckmh,memex-portal-deployment-6f9d6848f8-qskmn,memex-portal-deployment-f8dd949d7-sggqz,memex-portal-deployment-f8dd949d7-b9kg8,memex-portal-deployment-959f6754d-x87fl,memex-portal-deployment-745745db7c-b2gjn,memex-portal-deployment-56fbdcd48f-7cd4h,memex-portal-deployment-589ff8f895-ghgjt,memex-portal-deployment-589ff8f895-vn49b,memex-portal-deployment-56fbdcd48f-cbms9Recent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.81 remaining items
systemorph commentedon Oct 8, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
8b02af1dbffbfd6d(MeshWeaver.Hosting.Orleans.RoutingGrain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.5172aa25805c49fd8b02af1dbffbfd6dMeshWeaver.Hosting.Orleans.RoutingGrain[ROUTE] Cannot deliver the NotFound DeliveryFailure for fo2zye6hv5kjayw8{n}g6qcq to sender {path}: no pod-hub activation claims that address AND its stream has no live subscriber, and a NACK has no NACK of its own. The sender will wait out its own request budget — the original failure was: No {id} found at '{value}'. Closest ancestor is '{value}' (remainder='{value}').. {n} earlier undeliverable NACK(s) to this sender since the last such line were logged at Debug.memex-cloudmemex-portal-deployment-6b756548d4-dzn5qAdmin/_LogIncident/5172aa25805c49fdWhat triage wrote about this fingerprint
What is failing
Portal clients on memex-cloud are sending requests addressed to the node
Admin/Settings/Registration— the Settings ▸ Instance registration page introduced with the 2026-08-30 consent-gate feature. The server-side lookup fails (No node found at 'Admin/Settings/Registration'. Closest ancestor is 'Admin'), and the resulting NotFound NACK cannot be delivered back to the sender either (its pod-hub/stream has no live subscriber), so each sender hangs until its request budget expires instead of failing fast.Probable cause
Medium confidence: the client surfaces and addresses a settings node that is not provisioned on this deployment. Every sample in this burst — and in the sibling incidents from the same minute (
Admin/_LogIncident/bc4355c0689e6aa5,1f54d036176b8a51,aad942a75a0e3dcd,bcdff7ccae5e5daa,38b033584af2ee9d, all podmemex-portal-deployment-6b756548d4-dzn5q) — names the identical missing target, and a mesh search confirms noAdmin/Settings/Registrationnode exists. Either the registration panel's node is not created/installed on memex-cloud (packaging/provisioning bug), or the Settings menu offers the entry in a deployment where it does not apply. The RoutingGrain's own handling is consistent with its design; the defect is the missing node, plus the client-side cost of waiting out the request budget on an undeliverable NACK.Impact
User-visible failure of the Instance registration settings page for portal users: each request fails (or appears to hang until timeout). The feature this page gates — an admin accepting privacy/terms before the installation registers — cannot be completed on this portal. Observed as a short burst (~6 distinct sender sessions within a minute) on a single pod, not a fleet-wide outage.
Where to look
Admin/Settings/Registration.Observability note: this log site's message embeds the per-request id, so every burst at it mints a new fingerprint/incident — the same class of problem reported in
Feedback/_Submissions/fb-986b28181a7b692c2358e81f31236aa3("RoutingGrain back-pressure incidents split per episode stamp"). Consider masking the request id in the[ROUTE]undeliverable-NACK message so these fold into one incident.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-6b756548d4-dzn5qRecent log lines
systemorph commentedon Oct 8, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
8b02af1dbffbfd6d(MeshWeaver.Hosting.Orleans.RoutingGrain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.bc4355c0689e6aa58b02af1dbffbfd6dMeshWeaver.Hosting.Orleans.RoutingGrain[ROUTE] Cannot deliver the NotFound DeliveryFailure for znbyal0mveouqk8hipy6{n} to sender {path}: no pod-hub activation claims that address AND its stream has no live subscriber, and a NACK has no NACK of its own. The sender will wait out its own request budget — the original failure was: No {id} found at '{value}'. Closest ancestor is '{value}' (remainder='{value}').. {n} earlier undeliverable NACK(s) to this sender since the last such line were logged at Debug.memex-cloudmemex-portal-deployment-6b756548d4-dzn5qAdmin/_LogIncident/bc4355c0689e6aa5What triage wrote about this fingerprint
What is failing
A request in the
memex-clouddeployment targetedAdmin/Settings/Registration, a node that does not exist (closest ancestor isAdmin). The mesh correctly resolved aNotFounddelivery failure for it — but the RoutingGrain could not deliver that failure back to the sender (node/ds1elf9s6xspwllxgiqu8, portal podmemex-portal-deployment-6b756548d4-dzn5q): no pod-hub activation claims that address, its stream has no live subscriber, and a NACK has no NACK of its own. The system's documented fallback is to let the sender wait out its full request budget instead of failing fast.Probable cause
Two stacked problems, with different confidence:
Admin/Settings/Registrationdoes not exist in the mesh. This is the concrete event that minted theNotFoundfailure.Impact
Single occurrence, single pod, first seen at 2026-10-08 06:26 UTC. For the affected user this means a hung request that consumes the entire request budget instead of a fast 404-style answer. Low volume so far, but any pod restart / hub deactivation while a client request is in flight can hit the same dead-end, so the impact is user-visible hangs under a plausible race, not data corruption.
Where to look
MeshWeaver.Hosting.Orleans.RoutingGrain— the NACK/delivery-failure routing path, specifically the branch logged with[ROUTE] Cannot deliver ... DeliveryFailure ... no pod-hub activation claims that address AND its stream has no live subscriber(the framework source for this component is not in the readable mesh; start from the log categoryMeshWeaver.Hosting.Orleans.RoutingGrain).Admin/Settings/Registration— verify whether that address is a stale/erroneous link, or a node that failed to activate.Related, not duplicates
Feedback/_Submissions/fb-986b28181a7b692c2358e81f31236aa3/rbuergi/Feedback/route-backpressure-fingerprint-splits-20260921-2236— about RoutingGrain ticket fingerprint splitting, a different symptom at the same component.MeshWeaver.Messaging.DeliveryFailureException: No node found at '...'entries underOps/Logssince 2026-09-11 across pods — those are the deliverable failures; this incident is specifically the delivery of the failure itself failing.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-6b756548d4-dzn5qRecent log lines
systemorph commentedon Oct 8, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
8b02af1dbffbfd6d(MeshWeaver.Hosting.Orleans.RoutingGrain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.1f54d036176b8a518b02af1dbffbfd6dMeshWeaver.Hosting.Orleans.RoutingGrain[ROUTE] Cannot deliver the NotFound DeliveryFailure for g3{n}lnlhzr4vs3ms6 to sender {path}: no pod-hub activation claims that address AND its stream has no live subscriber, and a NACK has no NACK of its own. The sender will wait out its own request budget — the original failure was: No {id} found at '{value}'. Closest ancestor is '{value}' (remainder='{value}').. {n} earlier undeliverable NACK(s) to this sender since the last such line were logged at Debug.memex-cloudmemex-portal-deployment-6b756548d4-dzn5qAdmin/_LogIncident/1f54d036176b8a51What triage wrote about this fingerprint
What is failing
A portal client session on memex-cloud issued a request addressed to
rbuergi/Instance— a node that does not exist in the mesh (closest ancestorrbuergi, the user's space; remainderInstance). The server correctly produced a NotFound delivery failure, but the RoutingGrain could not deliver the NACK back to the sender (no pod-hub activation claims the sender's address, its stream has no live subscriber, and a NACK has no NACK of its own), so the sender waits out its full request budget instead of failing fast.Probable cause
Low-to-medium confidence. Something constructed an
Instancepath relative to the user's space that was never created — the user's pinned paths includeInstances(plural) and the workspace contains aninstanceskill doc (rbuergi/Skill/instance), so this may be a stale bookmark/pinned-path resolution or an agent mis-resolving a reference to a per-userInstancenode. The RoutingGrain's own logging is consistent with its design; the defect is the bad client-side address, compounded by the undeliverable-NACK hang.Impact
Minimal by the numbers: 1 occurrence, 1 pod (
memex-portal-deployment-6b756548d4-dzn5q), a single sender session within one burst. Likely one failed or hung client interaction. Not an outage.Where to look
rbuergi/Instance(sender session id is in the evidence below) — check pinned-path/bookmark resolution and any per-userInstancenode convention.Related
Admin/Settings/Registrationnode — the consent-gate settings page. This incident is the one outlier of that burst with a different missing target, so it is not a duplicate of Slow dispatch leg to Manufacturing import activity holds a routing slot and backs up same-stream legs past the 64-dispatch threshold #5703, but the NACK-delivery mechanism and the observability concern are shared.[ROUTE]undeliverable-NACK message so same-class bursts fold into one incident (same suggestion noted on Slow dispatch leg to Manufacturing import activity holds a routing slot and backs up same-stream legs past the 64-dispatch threshold #5703).Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-6b756548d4-dzn5qRecent log lines
systemorph commentedon Oct 8, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
8b02af1dbffbfd6d(MeshWeaver.Hosting.Orleans.RoutingGrain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.38b033584af2ee9d8b02af1dbffbfd6dMeshWeaver.Hosting.Orleans.RoutingGrain[ROUTE] Cannot deliver the NotFound DeliveryFailure for i3g6{n}r3{n}oo5ao8{n}c3mknu to sender {path}: no pod-hub activation claims that address AND its stream has no live subscriber, and a NACK has no NACK of its own. The sender will wait out its own request budget — the original failure was: No {id} found at '{value}'. Closest ancestor is '{value}' (remainder='{value}').. {n} earlier undeliverable NACK(s) to this sender since the last such line were logged at Debug.memex-cloudmemex-portal-deployment-6b756548d4-dzn5qAdmin/_LogIncident/38b033584af2ee9dWhat triage wrote about this fingerprint
What is failing
When the Orleans routing grain answers a cross-pod request with a NotFound
DeliveryFailure, it sometimes cannot deliver that NACK at all: no pod-hub activation claims the sender's address and the sender's stream has no live subscriber. The NACK is dropped, and the distant sender waits out its entire request budget for an answer the router had already computed — here a NotFound forrbuergi/Instance, a node that does not exist in therbuergipartition.Probable cause (medium confidence)
This is the documented cross-process residue of the refused-reply seams: Refused replies during teardown built in-process carriers for an undeliverable NACK (
IUndeliverableReplySink,MessageHub.TryDeliverNackInProcess, the intake-gate rule) and then explicitly declares the gap this incident lands in — "a reply that crossed a process boundary" is not covered, because the waiter lives in another process where no in-process registry can see it. A cross-pod NACK therefore has no carrier of last resort. Why this particular sender was unreachable is not decidable from one occurrence: either its hub disposed between request and NACK (a benign teardown race, the cross-process analogue of the #3303/#4072 family) or its pod-hub claim was lost on a membership re-partition (The Pod-Hub Claim Must Be Re-Asserted — a claim asserted once into a re-partitioned directory is lost silently). The claim-reassertion fix should make the second impossible, so a recurrence of this fingerprint discriminates between the two.Impact
Single occurrence on a single pod: one caller burned one request budget for an already-computed verdict, then recovered by timeout. Systemically, any cross-pod request whose NACK is minted after its sender's pod-hub is gone pays the same full-budget wait — and the same log site fired for several distinct senders within the same minute (sibling incidents
Admin/_LogIncident/5172aa25805c49fdandAdmin/_LogIncident/bcdff7ccae5e5daa, both NotFound forAdmin/Settings/Registration, a path that also does not exist), so the site is not a one-off. Secondary-path latency degradation, not data loss — sev:M, re-rule if the fingerprint recurs or a caller is shown to hang past its budget.Where to look
MeshWeaver.Hosting.Orleans.RoutingGrain— the undeliverable-NACK site; where a durable cross-pod NACK carrier, or a sender-unreachable classification the requester can re-arm on (theShuttingDown-style verdict, so a budget-long silence becomes an immediate retry), would have to land.rbuergi/Instance) — whatever issued it holds a stale or wrong path; worth a separate look if the same target recurs. Related in-process work: NackReachesTheWaiterDuringTeardownTest recurrences eject unrelated PRs from the merge queue — OwnerDisposing NACK verdict does not reach the armed watch after DisposeHostedHubs #6064.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-6b756548d4-dzn5qRecent log lines
systemorph commentedon Oct 8, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
8b02af1dbffbfd6d(MeshWeaver.Hosting.Orleans.RoutingGrain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.38b033584af2ee9d8b02af1dbffbfd6dMeshWeaver.Hosting.Orleans.RoutingGrain[ROUTE] Cannot deliver the NotFound DeliveryFailure for i3g6{n}r3{n}oo5ao8{n}c3mknu to sender {path}: no pod-hub activation claims that address AND its stream has no live subscriber, and a NACK has no NACK of its own. The sender will wait out its own request budget — the original failure was: No {id} found at '{value}'. Closest ancestor is '{value}' (remainder='{value}').. {n} earlier undeliverable NACK(s) to this sender since the last such line were logged at Debug.memex-cloudmemex-portal-deployment-6b756548d4-dzn5qAdmin/_LogIncident/38b033584af2ee9dWhat triage wrote about this fingerprint
What is failing
When the Orleans routing grain answers a cross-pod request with a NotFound
DeliveryFailure, it sometimes cannot deliver that NACK at all: no pod-hub activation claims the sender's address and the sender's stream has no live subscriber. The NACK is dropped, and the distant sender waits out its entire request budget for an answer the router had already computed — here a NotFound forrbuergi/Instance, a node that does not exist in therbuergipartition.Probable cause (medium confidence)
This is the documented cross-process residue of the refused-reply seams: Refused replies during teardown built in-process carriers for an undeliverable NACK (
IUndeliverableReplySink,MessageHub.TryDeliverNackInProcess, the intake-gate rule) and then explicitly declares the gap this incident lands in — "a reply that crossed a process boundary" is not covered, because the waiter lives in another process where no in-process registry can see it. A cross-pod NACK therefore has no carrier of last resort. Why this particular sender was unreachable is not decidable from one occurrence: either its hub disposed between request and NACK (a benign teardown race, the cross-process analogue of the #3303/#4072 family) or its pod-hub claim was lost on a membership re-partition (The Pod-Hub Claim Must Be Re-Asserted — a claim asserted once into a re-partitioned directory is lost silently). The claim-reassertion fix should make the second impossible, so a recurrence of this fingerprint discriminates between the two.Impact
Single occurrence on a single pod: one caller burned one request budget for an already-computed verdict, then recovered by timeout. Systemically, any cross-pod request whose NACK is minted after its sender's pod-hub is gone pays the same full-budget wait — and the same log site fired for several distinct senders within the same minute (sibling incidents
Admin/_LogIncident/5172aa25805c49fdandAdmin/_LogIncident/bcdff7ccae5e5daa, both NotFound forAdmin/Settings/Registration, a path that also does not exist), so the site is not a one-off. Secondary-path latency degradation, not data loss — sev:M, re-rule if the fingerprint recurs or a caller is shown to hang past its budget.Where to look
MeshWeaver.Hosting.Orleans.RoutingGrain— the undeliverable-NACK site; where a durable cross-pod NACK carrier, or a sender-unreachable classification the requester can re-arm on (theShuttingDown-style verdict, so a budget-long silence becomes an immediate retry), would have to land.rbuergi/Instance) — whatever issued it holds a stale or wrong path; worth a separate look if the same target recurs. Related in-process work: NackReachesTheWaiterDuringTeardownTest recurrences eject unrelated PRs from the merge queue — OwnerDisposing NACK verdict does not reach the armed watch after DisposeHostedHubs #6064.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-6b756548d4-dzn5qRecent log lines
systemorph commentedon Oct 8, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
8b02af1dbffbfd6d(MeshWeaver.Hosting.Orleans.RoutingGrain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.aad942a75a0e3dcd8b02af1dbffbfd6dMeshWeaver.Hosting.Orleans.RoutingGrain[ROUTE] Cannot deliver the NotFound DeliveryFailure for sbi1k8aqw8cdg1j1v8dbk to sender {path}: no pod-hub activation claims that address AND its stream has no live subscriber, and a NACK has no NACK of its own. The sender will wait out its own request budget — the original failure was: No {id} found at '{value}'. Closest ancestor is '{value}' (remainder='{value}').. {n} earlier undeliverable NACK(s) to this sender since the last such line were logged at Debug.memex-cloudmemex-portal-deployment-6b756548d4-dzn5qAdmin/_LogIncident/aad942a75a0e3dcdWhat triage wrote about this fingerprint
What is failing. The router correctly resolved a request to
Admin/Settings/Registrationas a missing node (closest ancestorAdmin, remainderSettings/Registration) and produced aNotFoundDeliveryFailure— but its NACK leg then hit the terminal undeliverable state: no pod-hub activation claims the sender's address (node/ck8ac38ogmuwnjmw7xsaf) and the sender's stream has no live subscriber. The sender therefore never learned its target was missing and will wait out its entire request budget in silence — the documented "silence is a wedge" symptom (Error Propagation & Wedges) recurring on the pod-hub transport.Probable cause (medium confidence — mechanism clear, trigger not). Two distinct questions, and only one is answerable from this sample:
node/ck8ac38ogmuwnjmw7xsafwas genuinely gone (environment), or alive with its registration momentarily absent (defect) — cannot be established from one occurrence. Note that anode/-prefixed hub address is exactly the shape a hosted client-process hub takes, and per Pod-Hub Delivery — the Transport Swap such hubs never attach a pod-hub grain, so the directed NACK leg can only ever reach them via the stream — which had no live subscriber here.LogUndeliverableNackterminal arm is a full stop — no bounded re-probe, no way to tell the waiting sender its verdict is gone, nothing to bound the wedge. This is the same design gap already ticketed as Slow dispatch leg to Manufacturing import activity holds a routing slot and backs up same-stream legs past the 64-dispatch threshold #5703 for aFailed-verdict NACK to a client-process sender.A secondary, separate question worth a look: why the original request targeted
Admin/Settings/Registrationat all — that path does not exist in the mesh (nothing underAdmin/Settingsresolves), so a caller — likely a portal UI view or a stale client reference — is asking for a node that isn't there. That caller got no answer either.Impact. One occurrence, one pod (
memex-cloud, 2026-10-08). The immediate cost is one sender waiting out its full budget instead of receiving an immediateNotFound. The class-level cost is larger: any request whose failure NACK cannot be delivered wedges the sender for its full budget with nothing logged above Debug on its side. Judged sev:M — a clear defect with an easy workaround (the sender's own budget eventually expires; the underlying request was already unanswerable), on a secondary path.Likely duplicate — reference before investigating. This is the same log site (site fold
8b02af1dbffbfd6d) and the same terminalLogUndeliverableNackarm as Systemorph/MeshWeaver#5703 (filed 2026-10-05 from incidentAdmin/_LogIncident/c960356bbdd8b7b9). This instance differs only in the original verdict (NotFoundfor a missing node vsFailedfor a pod-hub delivery timeout) and the sender shape (node/…hub vs a PR-triage thread path) — plausibly the same root defect, and a human can likely fold this into #5703 in seconds.Where to look.
MeshWeaver.Hosting.Orleans.RoutingGrain— thePostFailure/LogUndeliverableNackterminal arm insrc/MeshWeaver.Hosting.Orleans/RoutingGrain.cs(path as named in #5703; I could not read the source in the mesh): what, if anything, bounds the sender's wait after the NACK is declared undeliverable. Then the registration side: when does a hostednode/…client-process hub's stream subscription lapse relative to a pending request's lifetime (Pod-Hub Delivery — the Transport Swap documents the client-process residual). For the missing target: find the caller that constructs a request toAdmin/Settings/Registrationand correct or remove it.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-6b756548d4-dzn5qRecent log lines
systemorph commentedon Oct 8, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
8b02af1dbffbfd6d(MeshWeaver.Hosting.Orleans.RoutingGrain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.bcdff7ccae5e5daa8b02af1dbffbfd6dMeshWeaver.Hosting.Orleans.RoutingGrain[ROUTE] Cannot deliver the NotFound DeliveryFailure for f2{n}mid7{n}vz5hel6{n}jh4d0t to sender {path}: no pod-hub activation claims that address AND its stream has no live subscriber, and a NACK has no NACK of its own. The sender will wait out its own request budget — the original failure was: No {id} found at '{value}'. Closest ancestor is '{value}' (remainder='{value}').. {n} earlier undeliverable NACK(s) to this sender since the last such line were logged at Debug.memex-cloudmemex-portal-deployment-6b756548d4-dzn5qAdmin/_LogIncident/bcdff7ccae5e5daaWhat triage wrote about this fingerprint
What is failing
Two things are visible at one log site in the
memex-cloudportal:Admin/Settings/Registration— a node that does not exist in this mesh (closest ancestor isAdmin). The router correctly answers these withNotFoundDeliveryFailures (NACKs).RoutingGrainlogs the undeliverable failure atfaillevel and drops it — the sender, if it still existed, would wait out its full request budget.Probable cause — medium confidence on both halves:
Admin/Settings/Registrationafter the installation/registration flow changed (see the "An installation asks before it registers" change note). Three different sender node ids issued these requests within seconds of each other, which looks like a page load or view initialization rather than one stray call.RoutingGrain's reply leg for senders that are gone by reply time. The senders here had already lost their pod-hub, so the drop itself is likely benign — there is no waiter left to hang — but the condition is logged at Critical for what may be an expected disconnect race.Impact
Judged from the numbers: a one-off small burst, a single portal pod, and the first occurrence of this fingerprint. No outage, and no live request demonstrably hung. A sibling incident at the same log site in the same window shows the same shape for a different missing path (
rbuergi/Instance, seeAdmin/_LogIncident/38b033584af2ee9d), so the NACK dead-end generalizes, but neither burst looks like live-traffic loss.Where to look
RoutingGrainin theMeshWeaver.Hosting.Orleanssources (log categoryMeshWeaver.Hosting.Orleans.RoutingGrain, message "Cannot deliver the NotFound DeliveryFailure"). Routing placement and pod-hub reachability context: The Routing Entry Point.Admin/Settings/Registration: the Admin/Settings views and the installation-registration flow (change note). A client-side reference to a node that no longer exists is the actually-fixable defect here.Already known?
RoutingGrainturning transient pod-hub conditions into terminalDeliveryFailures (same component, different failure mode).{n}digit masking, so the same log site split instead of folding. Same mechanism class as the RoutingGrain back-pressure line documented inFeedback/_Submissions/fb-986b28181a7b692c2358e81f31236aa3— a masking case for this line likely belongs alongside it.Severity:
sev:L— a rare edge case (first occurrence), with no demonstrated live-traffic loss; the request budget bounds any hang.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-6b756548d4-dzn5qRecent log lines
fix(#5703): name the stamp what it is — DISPATCHED to the pool, not r…
rbuergi commentedon Oct 8, 2026
Fix for the verdict half: #6324, open.
Defect. The newest shape here (fingerprint
2ca53b1d1670888b, 2026-10-05 20:00Z) reported the oldest leg at 124 863 ms and 138 280 ms, with the deepest queue at 61–62 on the single channelcache/L1PlUIhTo0WIzSJqWuyx1Q [HOTBn-NpckOVgw-t3NcqWg], and logged it at Critical. That Critical misread its own measurement.RoutingGrain.RouteMessage, stream branch, calledquiescence?.Track(...)beforeorderedDispatcher.Enqueue.LegSelfBoundbounds a leg after it gets its turn.What #6324 changes.
RoutingQuiescence.TrackQueuedandOrderedRouteDispatcher'sonLegStartedcallback now stamp a leg's dispatch: the moment its channel hands it to the routing pool.AQueuedLegIsNotALeakTestfails (201.9 ms vs 0.56 ms since dispatch).What it does NOT fix, so this stays open.
HOTBn-…) produced 60+ frames faster than one cross-siloIPodHubGrain.Deliverper frame could drain them. That producer is still unidentified.Cannot deliver the NotFound DeliveryFailurefingerprints fromRoutingGrain, through 2026-10-08 06:26Z on memex-cloud, folded in because they share the site key (category + EventId 0 + exception type). Those need their own EventIds before this issue's count means anything. fix(#1549): every NodeType release-failure class logs under its own EventId #6326 applies the same remedy to the NodeType release site (Per-node hubs stopped answering on memex-cloud for 2h20m — the NodeType release burst is the symptom that surfaced it #1549).Merge pull request #6324 from Systemorph/fix/5703-queued-leg-age-is-n…
rbuergi commentedon Oct 9, 2026
#6324 is live on memex, and today's back-pressure has a named producer.
c573a41138) is in memex.systemorph.com's core8b791643f1, generationcfb58787b, booted 11:16–11:45Z.Ops/Actions/logs-sevM-a4315-20261009-q14(11:25Z → 15:05Z, cut at 200) holds 100 back-pressure episodes. Each prints the dispatched/queued split:oldest leg dispatched 3 ms … oldest leg queued behind its own channel … none queued.waiting for a pool slot 62, and their dispatch targets areLinkedIn/Learning/_Activity/…and…/Release/…, which are delete legs. The commit posted every leaf of a subtree at once. fix(#6351): a recursive delete's commit fans out through one bounded lane; a leaf's refusal keeps its reason #6357 gives that fan-out one bounded lane (see Bulk recursive delete aborts mid-subtree when a leaf commit stalls behind the saturated cap-1 Postgres write pool #6351), which removes this producer.Still open, as before: the producer of the 10-05
HOTBn-…single-stream backlog, and the ~58 foldedCannot deliver the NotFound DeliveryFailurefingerprints that share this site key.rbuergi commentedon Oct 11, 2026
Closing: the back-pressure this issue tracks has not been recorded at Critical since 2026-10-06 02:00Z. Two things it collected along the way are named below so they are not lost.
The title's defect — one slow leg holding a slot while same-stream legs back up past the threshold — was a single 43-second burst on 2026-09-24. What followed under this issue:
c573a41138)Reading — the incident record (
namespace:Admin/_LogIncident name:*RoutingGrain* sort:lastModified-desc, memex.systemorph.com, 2026-10-11 ~11:20Z; the folder is live, other incidents were written at 11:03Z today): the newestRouting back-pressureincident is5f6421c4b3852c28, last modified 2026-10-06T02:00:56Z. Nothing since, over five days.Not established, and not blocking the close:
cache/L1PlUIhTo0WIzSJqWuyx1Q [HOTBn-…]). It fell in that day's fleet stress window and has not recurred.Logsactions with 429 (see ProvisionPlan log-ingestion run drops user identity on write-back (0 of 47 entries landed) and its Loki query phase also times out / hits 429 #5717, A LogStore pool: range reads against the log store run one at a time per process #6481).Two things that folded in here and are NOT this defect:
Cannot deliver the NotFound DeliveryFailure … to sender node/…— about 58 fingerprints, last on 2026-10-08 06:32Z on memex-cloud. The target wasAdmin/Settings/Registration. That path is not stale: it is the Hosting.Instance module's own settings node (MeshWeaver.Pluginssrc/MeshWeaver.Hosting.Instance/InstanceApp.cs). So the burst reads as point reads of a node that did not exist yet on that instance, from a view whose client hub was gone by the time the NotFound came back. A caller-side question for that module, not a routing one.Directed delivery to pod hub … failed — surfacing Failed DeliveryFailurewith aTimeoutException(254804a0aad8b746, last modified 2026-10-11T04:14Z). This one IS still occurring; the watcher last wrote it in the same pass as the incident-write stalls on Incident comment write stalls 30s then aborts when the incident hub cannot hydrate from persistence #4962 (both nodes at 04:14:53Z). It is tracked there.Fleet issue wave, Claude Opus 5.5 (agent session).
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-09-26)Still occurring: 59 more since the last update (140 total).
memex-portal-deployment-7596dd76bb-cbbj8,memex-portal-deployment-56fbdcd48f-sz5st,memex-portal-deployment-76cf776847-2x9rq,memex-portal-deployment-76cf776847-9p7z4,memex-portal-deployment-76cf776847-5x2kg,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-8498855cbf-vchtf,memex-portal-deployment-6f9d6848f8-d9jwp,memex-portal-deployment-f8dd949d7-b9kg8,memex-portal-deployment-589ff8f895-ghgjt,memex-portal-deployment-745745db7c-b2gjn,memex-portal-deployment-56fbdcd48f-7cd4h,memex-portal-deployment-56fbdcd48f-cbms9Recent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.