fix(#2986): a read of a mid-recycle owner rides the ShuttingDown NACK out instead of giving up in 1 ms by rbuergi · Pull Request #3009 · Systemorph/MeshWeaver

rbuergi · 2026-09-01T19:43:41Z

The failure

ImportTypeBeforeInstanceTest timed out on CI run 33523142249 reading a node whose per-node hub was being recycled by the overlay self-heal. The import had already reported Imported / failed=0 — the defect is entirely in the read path.

outcome=Imported count=14 failed=0 blocked=[]
15:04:56.211 [Warning] Stream heCb5oZx…: resubscribe failed.
  DeliveryFailureException: Hub Tb666188a0/Inst is shutting down (RunLevel=Dead, activation #017DA86C) …
15:04:56.212 [Warning] Stream heCb5oZx…: resubscribe failed.   (activation #017DA86C)
15:04:56.212 [Warning] Stream heCb5oZx…: resubscribe failed.   (activation #017DA86C)
15:05:51.119 === TEST FAILED: The operation has timed out.

Three refusals, from one activation, inside one millisecond — then nothing at all for the remaining 55 s.

Verified root cause

JsonSynchronizationStream's recycle re-arm gates each re-ask on the rejecting instance's DisposalCompleted, so the attempt lands on a state that can answer (#1360). But DisposalCompleted is signalled AFTER RunLevel = Dead, so an activation that has already reached Dead satisfies that join instantly — and it can still be the instance routing hands the delivery to. The join is then a no-op, the re-ask returns to the same corpse at memory speed, and the whole MaxRecycleReArms budget — a counter only a successful resubscribe resets — is spent inside one teardown window. A pure recycle writes nothing, so the change-feed latch (the only other trigger) never fires and the stream is orphaned for good.

Three chances that all fall inside one millisecond are one chance.

The fix

The accounting is split along the two axes the failure confused. Neither bound was widened; nothing polls.

Axis	Question	Bound
Activation	"Is the address recycling in a loop?"	`MaxRecycleReArms` — unchanged at 3, now counting distinct activations
Time	"Is this one teardown still draining?"	`MaxSameActivationReAsks` = 16, paced by `SyncStreamOptions.RecycleReAskPace` = 500 ms

A rejection from the activation that already refused us is not a new recycle; it is the same teardown, still draining, which the join failed to wait through. Charging it against the activation budget was the bug. It is charged on the time axis instead, and the next attempt rests first — so each of the bounded chances is a genuinely separate opportunity. 16 × 500 ms is sized to reach MessageHub.DisposalWatchdogTimeout (8 s), past which a wedged teardown is force-torn-down and the address is gone. This is the same pacing GetMeshNodeOutcome already applies to a recycling address (#1701), at the same 500 ms.

This is not a watchdog or a retry loop: a timer exists only in response to a real rejection, exactly one re-ask is ever outstanding (the in-flight guard plus the Concat), the ride-out is bounded on both axes, and it stops the instant the owner answers or its activation changes.

Two more defects in the same lines:

The join was projected eagerly. Select projects on arrival and Concat only defers subscription, so every rejection in a burst snapshotted the owner's state at arrival time. Now Observable.Defer, with the verdict carried on a ReArmDecision rather than re-derived from counters that keep moving.
The give-up was silent — half of why this read as "slow". Both give-up paths now log one Warning naming the owner, the activation and which bound was hit.

Minting and parsing of the activation #XXXXXXXX token move to ShutdownNack (MeshWeaver.Messaging.Contract) — one marker, one formatter, one parser, in the assembly both riders reference. The minting sites had already drifted once (#2376).

Before/after evidence

RecycleReAskRidesOutTheDyingActivationTest (new, MeshWeaver.Data.Test) parks a real owner at RunLevel=Dead with routing held pointed at it, so every attempt gets the genuine transient NACK from the genuine MessageService, activation tag and all. No mocks, no crafted failures.

Against origin/main's source it reproduces the CI signature exactly and fails:

[Warning] Stream 7u8wtes2WEOQSY3iZOzQwg: resubscribe failed.   21:25:32.439
[Warning] Stream 7u8wtes2WEOQSY3iZOzQwg: resubscribe failed.   21:25:32.440
[Warning] Stream 7u8wtes2WEOQSY3iZOzQwg: resubscribe failed.   21:25:32.440
DIAG refusals by the dead activation: 4
=== TEST FAILED: ... The observable emitted nothing at all.   (30 s later)

With the fix it converges.

The load-independent discriminator

The test's primary assertion is not "did it finish in time" — it is did the latch make any further attempt at all once the address reactivated. Re-run at the very end against both trees, on a host at load average 85-122 (18 cores, ~6x oversubscribed):

	refusals by the dead activation	attempts AFTER reactivation	result
`origin/main` source	4	0, sampled 1801 times over 36 s	FAIL
with the fix	4	1	PASS

Last of 1801 emission(s) was: 0 — the counter never left zero, because on the unfixed code there is nothing left to try: the budget is spent and a pure recycle writes nothing, so the change-feed latch has no event to fire on either. That is a structural fact about the latch, not a timing fact. A loaded runner can only make a correct implementation slower to satisfy it; it can never make a broken one satisfy it.

⚠️ Measurement conditions. Every observation in this PR was taken on a host at load average 64-130 on 18 cores (up to 7x oversubscribed; seven concurrent agent sessions). That is precisely why the repro was built to be structural rather than timed, and why the corpse is parked deliberately (the owner is disposed to RunLevel=Dead and routing is held pointed at it by an explicit flag) rather than raced into existence.

Verification

dotnet build -c Release -warnaserror, one project per invocation, 13 projects (Messaging.Contract, Messaging.Hub, Data, Mesh.Contract, Documentation, Hosting, Hosting.Monolith, Connection.Orleans, Hosting.Orleans, plus 4 test projects): 0 Warning(s), 0 Error(s) everywhere.
Full MeshWeaver.Data.Test: 420/420.
Full MeshWeaver.Documentation.Test: 166/166 (includes DocumentationLinkIntegrityTest and every ratchet guard).
Full MeshWeaver.Messaging.Hub.Test: 341/341.
Full MeshWeaver.Graph.Test: 1084/1084 — including ImportTypeBeforeInstanceTest, the suite this issue was filed from.

Conserved work product

New doc page: Doc/Architecture/RidingOutAShuttingDownAddress — the ShuttingDown contract, the two riders, why a re-ask needs a join, why the join can be satisfied by a state that still cannot answer, the two axes, and the rules for a new rider. Linked from the Architecture topic map.
What's New entry (Category: Fix).
Corrected a stale claim in ShutdownFailureRideOutTest's prose, which named a ShutdownRoutingRejectClassificationTest that does not exist in this repo.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SD7aiLSo59xng2TzU32xEa

Copilot

🟡 Changes recommended

Two new “giving up” warning logs report the configured budget constants instead of the actual observed attempt counts at the moment of give-up, which makes the new diagnostics misleading.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR fixes a reliability bug in the mesh read/synchronization path where a ShuttingDown NACK from a single dying activation could burn the entire recycle re-arm budget in ~1ms, leaving the sync stream orphaned (appearing as a slow timeout instead of a recoverable mid-recycle read).

Changes:

Updates JsonSynchronizationStream’s recycle re-arm latch to distinguish distinct activations vs same activation still draining, adding paced re-asks for the latter and improved give-up logging.
Centralizes activation-tag formatting/parsing for ShuttingDown NACKs into ShutdownNack (contract), and reuses it from both MessageService and MeshNodeStreamExtensions.
Adds a deterministic regression test and documentation/What’s New entries describing the contract and the fix.

File summaries

File	Description
test/MeshWeaver.Data.Test/ShutdownFailureRideOutTest.cs	Updates test prose to reflect current producing/consuming coverage and references.
test/MeshWeaver.Data.Test/RecycleReAskRidesOutTheDyingActivationTest.cs	New regression test that reproduces the “same activation NACK burst” and asserts convergence after reactivation.
src/MeshWeaver.Messaging.Hub/MessageService.cs	Routes activation-tag rendering through the shared `ShutdownNack` formatter to prevent marker drift.
src/MeshWeaver.Messaging.Contract/ShutdownNack.cs	New shared formatter/parser for `activation #XXXXXXXX` tags embedded in `ShuttingDown` NACKs.
src/MeshWeaver.Mesh.Contract/MeshNodeStreamExtensions.cs	Delegates activation-tag extraction to `ShutdownNack` to keep both riders consistent.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-02-a-page-whose-node-is-restarting-now-waits-for-it.md	What’s New entry describing the user-visible effect of riding out a mid-recycle read.
src/MeshWeaver.Documentation/Data/Architecture/RidingOutAShuttingDownAddress.md	New architecture page documenting the `ShuttingDown` ride-out contract and the two-axis bounding rule.
src/MeshWeaver.Documentation/Data/Architecture.md	Adds the new architecture page to the Reliability & wedges index section.
src/MeshWeaver.Data/Serialization/SyncStreamOptions.cs	Introduces `RecycleReAskPace` option (default 500ms) for same-activation paced re-asks.
src/MeshWeaver.Data/Serialization/JsonSynchronizationStream.cs	Core behavior change: split accounting across activation/time axes; defer probe; paced re-ask for same activation; improved diagnostics.

Review details

Suppressed comments (1)

src/MeshWeaver.Data/Serialization/JsonSynchronizationStream.cs:892

The wedged-teardown "giving up" warning uses the budget constant (16) for {Attempts}, but the give-up occurs only after the increment check fails, so the actual observed count will be MaxSameActivationReAsks+1. Logging the real attempt count makes the diagnostic match the runtime behavior.

                        "Stream {StreamId}: giving up on owner {Owner} — the SAME activation "
                        + "#{Activation} refused {Attempts} re-subscribes as ShuttingDown without ever "
                        + "releasing the address. That is a WEDGED TEARDOWN at the owner, not a slow "
                        + "reader: readers of this stream will now time out with nothing to show for it.",
                        reduced.StreamId, owner, activation, MaxSameActivationReAsks);

Files reviewed: 10/10 changed files
Comments generated: 1
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-09-01T19:48:08Z