fix(routing): a back-pressure GAUGE crossing is not an incident — level it by shape by rbuergi · Pull Request #5322 · Systemorph/MeshWeaver

rbuergi · 2026-09-22T19:37:32Z

Why you are being spammed

The routing back-pressure report is unconditionally LogCritical. The red-log ticketing path files an incident per Critical fingerprint, so every crossing opens a ticket — including the ones whose own text says nothing is wrong.

Measured 2026-09-21, over the crossings carried in this site's open incidents:

crossing shape	count
`deepest = 0` — nothing waiting on anything (load)	17
`deepest >= 1` — a leg blocked behind a leg (stall)	4

Roughly four in five Criticals reported that nothing was stuck. The line has always said so itself — "0 means nothing is waiting on anything, so read it as load" — while its level said the opposite.

Combined with the per-activation identity split (fixed separately in MeshWeaver.Plugins #2165/#2167/#2168/#2218) that produced 46 open issues from ONE log statement, 42 of them duplicates, all closed today against #5118.

The change

deepest >= 1 → a leg is waiting on a leg — head-of-line blocking on one channel. Actionable, stays Critical.
deepest == 0 → load, reported at Warning: still logged, with every field it had before, still paired with ReportDrained's duration — but it no longer files a ticket.

🚨 A permanent level decision with a cost/value argument, not a debugging tweak. Critical is what the ticketing path acts on, so it has to mean "act now". Nothing is hidden.

What this deliberately is NOT

Not a throttle, and not a raised bound. For the 80% nothing is stuck, so there is nothing to throttle; for the other 20% a blocked leg stays blocked whatever the load. Capping a bound to stop a gauge crossing is the band-aid the house rules forbid.
Not a fix for the routing condition. A cache destination head-of-line blocking ~62 legs is a real defect, still open, and still pages at Critical.

Why this lane and not the other one

The identity fix is already merged in MeshWeaver.Plugins but cannot reach production: the Plugins Portal image rebuild workflow has failed on every main push today at its GitHub App token step (actions: write not granted to the installation), so core's rebuild is never dispatched. This change is in core, so it ships on core's own CD, which is not blocked.

Verification

MeshWeaver.Hosting.Orleans.Test — Release -warnaserror: 0 Warning(s), 0 Error(s)
BackPressureLevelByShapeTest — Total: 5, Failed: 0

The decision is an internal static so it is pinned by test rather than by reading the call site. The boundary is asserted deliberately (deepest = 1 is already blocked), as is an impossible negative depth — so a guard cannot silently promote bad input to Critical and re-open the flood.

🚨 Run the tests through the MTP runner directly; dotnet test --filter is refused on this project under the .NET 10 SDK and exits non-zero, which reads exactly like a test failure and is not one.

Not verified

Not observed on a running portal — the effect on incident volume is predicted from the measured shape distribution, not yet seen in production.

Pairs-with: none — SaturationLevel is new and internal; no public surface changes.

🤖 Generated with Claude Code

…el it by shape The routing back-pressure report was unconditionally LogCritical. The red-log ticketing path files an incident per Critical fingerprint, so every crossing opened a ticket — including the ones whose own text said nothing was wrong. MEASURED 2026-09-21 over the crossings carried in this site's open incidents: | crossing shape | count | |--------------------------------------------------|-------| | deepest = 0 — nothing waiting on anything (LOAD) | 17 | | deepest >= 1 — a leg blocked behind a leg (STALL) | 4 | Roughly four in five Criticals reported that nothing was stuck. The line has always said so itself — "0 means nothing is waiting on anything, so read it as load" — while its LEVEL said the opposite. Combined with the per-activation identity split (fixed separately in MeshWeaver.Plugins #2165/#2167/#2168/#2218) that produced 46 open issues from ONE log statement, 42 of them duplicates, all closed today. THE RULE: deepest >= 1 means a leg is waiting on a LEG — head-of-line blocking on one channel — which is actionable and stays Critical. deepest == 0 is load and is reported at Warning: still logged, with every field it had before, still paired with ReportDrained's duration — but it no longer files a ticket. 🚨 This is a permanent level decision with a cost/value argument, NOT a debugging tweak. Critical is what the ticketing path ACTS on, so it has to mean "act now". Nothing is hidden: the load crossings keep their line, and the head-of-line shape — the one worth waking someone for — is untouched. This is deliberately NOT a throttle and NOT a raised bound. For the 80% nothing is stuck, so there is nothing to throttle; for the other 20% a blocked leg stays blocked whatever the load. The underlying routing condition (a cache destination head-of-line blocking ~62 legs) remains open and still pages. The decision is an internal static so it is pinned by test rather than by reading the call site: the boundary (deepest = 1 is already blocked) and an impossible negative depth are both asserted, the latter so a guard cannot silently promote bad input to Critical and re-open the flood. Verified: MeshWeaver.Hosting.Orleans.Test builds Release -warnaserror with 0 Warning(s) 0 Error(s); BackPressureLevelByShapeTest Total: 5, Failed: 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Copilot review overview

🟡 Changes recommended

The severity classification may miss a newly enqueued blocking leg and suppress the required Critical incident.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 · 1

Open (2)

Snapshot occurs before enqueue, suppressing Critical at threshold crossing · New
Update documentation to reflect the revised saturation severity policy · New

What changed in this PR

This PR changes routing back-pressure severity: pure load crossings become Warning, while queued head-of-line blocking remains Critical.

Changes:

Adds shape-based SaturationLevel classification.
Adds boundary and invalid-depth tests.

File	Summary
`test/MeshWeaver.Hosting.Orleans.Test/BackPressureLevelByShapeTest.cs`	Tests load, blocking, and invalid-depth classifications.
`src/MeshWeaver.Hosting.Orleans/RoutingGrain.cs`	Applies shape-based logging severity; a critical sampling issue remains, and documentation is outdated.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

rbuergi · 2026-09-22T19:54:07Z

        var episode = Interlocked.Increment(ref saturationEpisode);
        var (channels, destinations, deepest) = orderedDispatcher.QueueSnapshot();
-        logger.LogCritical(
+        logger.Log(SaturationLevel(deepest),


Confirmed, and this is a real hole in the change — thank you.

RouteMessage calls ReportSaturation(Interlocked.Increment(ref inFlightRoutes), addressPath) and only THEN orderedDispatcher.Enqueue(...), so QueueSnapshot() cannot see the leg that is crossing. For the exact shape you name — the crossing leg being the first extra frame on a channel that already has one executing — the sampled depth is 0, my new rule reads that as load, and it emits Warning. That suppresses the Critical precisely at the boundary the change exists to preserve.

It is also a fact about my test: SaturationLevel(int) is a pure function, so asserting 1 => Critical verified the mapping and never the ARGUMENT. A test that cannot see the input that is wrong is not covering the case it claims.

Fixing by sampling AFTER the enqueue, so the snapshot includes the candidate leg, plus a regression over that crossing shape (depth 1 arising from the enqueue itself). The pure-function test stays, because the boundary is still worth pinning, but it is no longer the only coverage.

rbuergi · 2026-09-22T19:54:12Z

+    internal static LogLevel SaturationLevel(int deepest) =>
+        deepest >= 1 ? LogLevel.Critical : LogLevel.Warning;


Agreed — shipping a level policy while the runbook still states the old one is worse than either behaviour on its own, because the page is what an operator reads at 3am.

Updating both in this change:

SaturationThreshold's own remarks (the "reported at Critical (once, until it recovers)" line), which now have to say that the LEVEL depends on the shape and the latch does not.

Doc/Architecture/ReadingARoutingSaturationReport.md, which is the runbook for this incident stream and currently tells the reader any crossing is Critical.

The point the docs need to carry is the discriminator, not just the new levels: deepest >= 1 means a leg is waiting on a LEG and still pages; deepest = 0 means nothing is waiting on anything, is logged at Warning with every field it had before, and no longer opens a ticket.

github-actions · 2026-09-22T19:41:59Z