fix: the routing saturation alert reports what it measures, and a drained leg terminates by rbuergi · Pull Request #1358 · Systemorph/MeshWeaver

rbuergi · 2026-08-12T21:41:49Z

Closes #1172.

Issue disposition, deliberately asymmetric. #1172's subject IS the routing report — "dispatches
hang at 64 … a delivery leg is not completing" — and that premise is disproved below; the report was
the defect, together with the genuine slot leak it surfaced. Both are fixed here, so #1172 closes.

#1284 stays open. Its subject is the thread-pool delay, which is a true reading of CPU
exhaustion, and that survives this PR: the dominant burner was fixed in #1341, two more sites in
#1359, and ThreadExecution.PushToResponseMessage is still open (design on #1359). #1284 is the
right home for that residual — it is a sibling of #1172, not a consequence, and has been
retitled accordingly.

TL;DR

Neither issue is a stuck delivery, an Orleans limit, or a compile that never returns.
#1284 is not secondary to #1172 — they are two instruments reading one gauge
(ThreadPool scheduling latency), both downstream of CPU exhaustion. The dominant CPU
burner was the O(n²) activity-log write shape fixed hours ago in #1341; what remains is
enumerated at the bottom as scope calls, not silently changed here.

1. Where the 64 comes from

internal const int SaturationThreshold = 64 — src/MeshWeaver.Hosting.Orleans/RoutingGrain.cs:85.
A MeshWeaver constant, not configurable, not Orleans' NonReentrancyQueueSize. The
routing IIoPool is capped at 256 (IoPoolOptions.Routing) and the Compile pool is
separate (ProcessorCount). Nothing refuses, queues or throttles at 64: Dispatch hands
every route to the pool unconditionally and RouteMessage returns Forwarded immediately.

Why every prod line said exactly 64 — the report is edge-triggered and latching:

if (inFlight < SaturationThreshold) return;
if (Interlocked.Exchange(ref saturationReported, 1) == 1) return;   // latch

It fires on the single increment that crosses the line, so 64 is the only value it can
print. That artefact was read as evidence of a hard cap.

The same latch falsifies the wedge reading. ReportDrained only unlatches at ≤32. Five
criticals in 18 s on one pod (2026-08-10, z9t6r) therefore require the count to have
fallen below 32 four times in between — legs were draining en masse throughout.

2. The shared seam

Routing and compilation share no lock and no pool. Roslyn's Emit runs off the hub turn
(MeshNodeCompilationService.OnThreadPool / IoPoolNames.Compile). The seam is the
process CPU quota, reached through O(node-size) patch construction that runs
synchronously on the owning hub's action-block turn — every stream.Update re-serialises
the whole node to compute its patch, so writers that append to a growing collection are
O(n²). At ~7.8 cores demanded against a 6-core limit the pod sat at 51% of CFS periods
throttled, freezing the whole process in bursts.

How that reaches the routing counter specifically: a slot is claimed at DISPATCH — for
the stream branch at ENQUEUE, before the leg is subscribed at all — while the leg's own
bounds (ResolveTimeout 30 s, StreamPostTimeout 60 s) are operators inside the cold
observable and do not start until SubscribeThroughPool gets a ThreadPool thread and passes
the gate. That claim→subscribe window is bounded by nothing. So the counter is partly a
ThreadPool-starvation instrument — the same quantity LocalSiloHealthMonitor reports as a
"thread pool delay". Hence #1284 is a sibling, not a consequence.

The _Activity/compile and _Activity/import targets are not special: the report names the
latest dispatch target, i.e. whichever address happened to cross the threshold.

3. Slow, or never completing?

Neither — the compile is not the thing stalling. (The "compile activity with only 2 messages
⇒ Roslyn never ran" tell belongs to a different, already-fixed wedge, #583/#584.)

The changes

ReportSaturation reports what it measures. In-flight count, stream destinations
queued, deepest per-destination queue, routing-pool occupancy — and it states that the
named address is not a diagnosis. Deepest > 1 is the discriminator: legs waiting on a
leg (head-of-line) versus breadth. ReportDrained now prints the episode duration, which
separates a burst from a stall without profiling a live pod. New
OrderedRouteDispatcher.QueueSnapshot() takes both numbers under one lock, read only on
the report edge.
IoPool.SubscribeThroughPool no longer terminates silently. It swallowed the drain's
OperationCanceledException without terminating the observer, and the drain registration
only disposes the inner subscription — disposal emits nothing. So the observable
completed neither successfully nor with an error, and every .Finally(...) on it never
ran. That callback is exactly what releases the in-flight route slot and advances the
per-destination FIFO, so a leg the drain cancelled leaked its slot and stranded its
destination's queue. It now completes.

No band-aid: no bound raised, no timeout added, no retry, no reentrancy change.

Verification

Test	Result
`IoPoolTest.Drain_terminates_a_SubscribeThroughPool_leg_it_cancels`	RED before change 2 (fails in 5 s on the exact assertion), GREEN after
`IoPoolTest` (full class)	15/15
`RoutingBackpressureShapeTest`	builds the two shapes that yield an IDENTICAL in-flight count of 64 — 64 destinations × 1 leg vs 1 destination × 64 legs — and asserts the count cannot separate them while the snapshot can: `(64, deepest 0)` vs `(1, deepest 63)`
Orleans routing suite	20/20
What's New + doc link integrity	2/2
`MeshWeaver.Hosting.Monolith.Test` (teardown-SIGSEGV canary, since change 2 is a teardown path)	633 passed / 0 failed / 3 skipped, 7m35s

Deliberately NOT done — the remaining O(n²) writers

Same defect class as #1341, each needing the same progress-granularity scope call the owner
made there. Ranked by likely production cost:

ThreadExecution.PushToResponseMessage (src/MeshWeaver.AI/ThreadExecution.cs:1121,
driven by Sample(100 ms) at :2097) — ~10 writes/s for the whole streaming duration of
every agent round, each re-serialising a ThreadMessage whose Text, ToolCalls and
UpdatedNodes all grow monotonically. Hottest steady-state instance; needs a delta/append
field, not batching.
InstanceSyncService.AppendPending (src/MeshWeaver.InstanceSync/InstanceSyncService.cs:170)
— one write per changed node into a growing PendingChanges. RemoveDrained is already
batched; the symmetric AppendPendingRange plus buffering the change feed is the fix, and
the trade is durability latency for a pending entry.
ContentIndexingActivity.Append (…Indexing.Graph/ContentIndexingActivity.cs:185,
per-file at ContentIndexingObserver.cs:236/245) — one to two writes per indexed file; the
closest structural twin of the case fix: batch the import activity log per phase — the O(n²) CPU burner throttling memex-cloud #1341 fixed. Wants an AppendLogs-style batch.
GitSync ActivityRunner.Append (src/MeshWeaver.GitSync/ActivityRunner.cs:253) — one
write per unparseable file, and .Merge(8) makes them concurrent on one node.
Thread heartbeats (ThreadExecution.cs:1283, :2133) and ActivityLogLogger
(src/MeshWeaver.Kernel.Hub/ActivityLogLogger.cs:199) — already rate-capped, but each write
still re-serialises a list that has no upper bound.

ActivityLog.Messages is an unbounded ImmutableList in node content, so the general cure is
the satellite-node shape rather than more batching.

🤖 Generated with Claude Code

…ined leg terminates Issues #1172 / #1284. Neither is a stuck delivery, an Orleans limit, or a compile that never returns — and the diagnostic that said otherwise is itself a defect. ## The 64 is ours, and it is a report, not a bound `RoutingGrain.SaturationThreshold` (RoutingGrain.cs:85) is a MeshWeaver constant. It is NOT Orleans' `NonReentrancyQueueSize`; the routing IIoPool is capped at 256 (IoPoolOptions.Routing) and nothing refuses, queues or throttles at 64 — `Dispatch` hands every route to the pool unconditionally and `RouteMessage` still returns `Forwarded`. The reason all 22 prod occurrences printed EXACTLY 64 is the latch: the report fires on the single increment that crosses the line and then latches until the count falls to 32, so 64 is the only value it can print. That artefact was read as evidence of a hard cap. The same latch falsifies the "wedge" reading. Five criticals in 18 s on one pod (2026-08-10, z9t6r) require the count to have fallen below 32 four times in between — legs were draining en masse the whole time. ## What the count actually mixes A slot is claimed at DISPATCH — for the stream branch at ENQUEUE, before the leg is subscribed at all. And the leg's own bounds (`ResolveTimeout` 30 s, `StreamPostTimeout` 60 s) are operators INSIDE the cold observable, so they do not start until `SubscribeThroughPool` gets a ThreadPool thread and passes the gate. That claim→subscribe window is bounded by nothing. So the counter is partly an instrument for ThreadPool starvation — the same quantity Orleans' `LocalSiloHealthMonitor` reports as a "thread pool delay". #1284 is therefore NOT secondary to #1172: they are two instruments reading one gauge. Routing and compilation share no lock and no pool (Routing=256, Compile=ProcessorCount; Roslyn runs off-turn via `IoPoolNames.Compile`). The shared seam is the process CPU quota, reached through O(node-size) patch construction on hub action-block turns — the O(n²) class #1341 fixed for imports. The `_Activity/compile` addresses in the log are just the hottest addresses when the threshold was crossed; the report names the LATEST dispatch target, never a slow one. ## The two changes 1. `ReportSaturation` now reports what it can see — in-flight count, stream destinations queued, deepest per-destination queue, routing-pool occupancy — and states that the named address is not a diagnosis. `Deepest > 1` is the discriminator: legs waiting on a leg (head-of-line) vs breadth. `ReportDrained` now prints how long the episode lasted, which separates a burst from a stall without profiling a live pod. New `OrderedRouteDispatcher.QueueSnapshot()` takes both numbers under one lock; it is read only on the report edge. 2. `IoPool.SubscribeThroughPool` swallowed the drain's `OperationCanceledException` without terminating the observer. The drain registration only DISPOSES the inner subscription, and disposal emits nothing — so the observable terminated neither completed nor errored and every `.Finally(...)` on it never ran. That callback is what releases the in-flight route slot and advances the per-destination FIFO, so a leg the drain cancelled leaked its slot and stranded its destination's queue. It now completes. No band-aid: no bound raised, no timeout added, no retry, no reentrancy change. ## Verification - `IoPoolTest.Drain_terminates_a_SubscribeThroughPool_leg_it_cancels` — RED before change 2 (fails in 5 s on the exact assertion), GREEN after. Full IoPoolTest class 15/15. - `RoutingBackpressureShapeTest` — builds both shapes that produce an IDENTICAL in-flight count of 64 (64 destinations × 1 leg, vs 1 destination × 64 legs) and asserts the count cannot tell them apart while the queue snapshot can: (64 destinations, deepest 0) vs (1, deepest 63). - Orleans routing suite 20/20; Documentation What's New + link integrity 2/2; Hosting.Monolith.Test (the teardown-SIGSEGV canary, since change 2 is a teardown path) green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Improves diagnosability of Orleans routing back-pressure incidents by making the saturation alert report the quantities it actually measures (including per-destination queue shape and episode duration), and fixes IIoPool.SubscribeThroughPool so drain-cancelled legs still terminate (allowing .Finally(...) bookkeeping to run and preventing leaked routing slots / stranded FIFOs).

Changes:

Enhance RoutingGrain saturation/drain logging with queue-shape snapshot and “cleared after” duration.
Add OrderedRouteDispatcher.QueueSnapshot() to report destination spread vs head-of-line depth under one lock.
Ensure IoPool.SubscribeThroughPool completes observers on drain/unsubscribe cancellation, plus add regression tests + What’s New entry.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/MeshWeaver.Mesh.Contract/Threading/IoPool.cs	Completes observers on `OperationCanceledException` so drain-cancelled subscribe legs terminate and downstream `.Finally(...)` runs.
src/MeshWeaver.Hosting.Orleans/RoutingGrain.cs	Rewords saturation/drained alerts to report measured back-pressure signals, includes queue snapshot + episode duration.
src/MeshWeaver.Hosting.Orleans/OrderedRouteDispatcher.cs	Adds `QueueSnapshot()` to distinguish “many destinations” vs “one destination queued deep” shapes.
test/MeshWeaver.Hosting.Test/IoPoolTest.cs	Adds a deterministic regression test ensuring drain-cancelled `SubscribeThroughPool` legs still terminate.
test/MeshWeaver.Hosting.Orleans.Test/RoutingBackpressureShapeTest.cs	Adds a regression test demonstrating identical in-flight counts can represent opposite backlog shapes and asserting snapshot discrimination.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-12-routing-alerts-name-what-they-actually-saw.md	Documents the alert behavior change and the drain termination fix.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-08-12T21:56:42Z