A deactivating silo asks Orleans for no new activation (#2252 class fix) by rbuergi · Pull Request #2270 · Systemorph/MeshWeaver

rbuergi · 2026-08-25T20:26:04Z

The invariant

"no deactivating silo should activate any grain whatsoever"

Not for a routed message, not for a stream delivery, not for a "goodbye" announcement.

The key question: Orleans already provides this — the mesh was bypassing it

Orleans enforces the rule itself. Decompiled Orleans.Runtime.Catalog (10.2.2, the pinned version):

// Catalog.GetOrCreateActivation
using (GetStripedLock(in grainId).EnterScope())
{
    if (TryGetGrainContext(grainId, out data)) { … return data; }
    if ((int)_siloStatusOracle.CurrentStatus == 3)      // == SiloStatus.Active, and ONLY Active
    {
        data = grainActivator.CreateInstance(address);
        activations.RecordNewTarget(data);
    }
}
if (data == null) return UnableToCreateActivation(this, grainId);   // logs "…WhenNotActive", returns null

PlacementService.GetCompatibleSilos likewise intersects candidates with the ACTIVE silo set. So there was no parallel gate to build, and this PR does not add one.

What Orleans cannot know is that we have already decided to go away. IHostApplicationLifetime.ApplicationStopping fires strictly before the Orleans silo hosted service stops — which is precisely why DeliverMessage can already use it as a readiness signal. For the same reason the membership oracle still reports Active for some seconds after we know we are leaving, and in that window Orleans faithfully creates any activation we ask for, on the silo that is leaving. Asking is ours to stop; refusing is Orleans'.

Where the mesh was bypassing it

OrleansRoutingService has three grain call sites and applied the stopping gate to one. The other two were the pod-hub claim/release pair, three lines below the gate:

// AttachPodHub's teardown — before
grainFactory.GetGrain<IPodHubGrain>(addressPath).Detach().ToObservable().Subscribe(…);

IPodHubGrain is [PreferLocalPlacement], so once the previous activation has gone this does not release anything — it creates a fresh activation on the dying silo purely to tell it nothing, and that activation then belongs to the silo that is leaving. It is the "goodbye announcement" in its purest form, and it runs for every registered hub as the process winds down.

The fix

One seam. grainFactory is now reachable only through GrainWhileRunning<T>(key), which returns null once the host is stopping. Every GetGrain in the class goes through it.
A ratchet closes the class. GrainActivationSiteRatchetGuard fails the build on any new GetGrain site under src/, memex/, tools/, so the next call site cannot reintroduce the bypass. It carries two deliberate, opposite allowances: the seam itself (1), and RoutingGrain's two calls (2).

Why `RoutingGrain` stays deliberately UNGATED

Those two are the drain. A message already accepted for routing must still land, and Orleans' own placement is what sends it to a healthy silo rather than this one. Gating there would drop live work instead of relocating it — the rule is "no new activations", never "no traffic" (#1971).

The refusal is TRANSIENT, not terminal

A late token flip throws into DeliverMessage's pre-existing .Catch, which re-reads IsHostStopping and NACKs the sender with ErrorType.ShuttingDown. ShutdownFlippingAfterTheGate_IsRefusedAsTransient_NotTerminal asserts exactly that, and opens the window deterministically (an outbound dispatch is held while the sender's own inbound attach is pending on OrleansStreamingReadiness; stop the host, then fire the signal) — no sleeps, no race. This agrees with, and does not contradict, #2199.

Measured, failing-first

With the gate disabled (one-line toggle):

test	gate off	gate on
`HostStopping_TheGoodbyeAnnouncement_NeverAsksOrleansForAnActivation`	FAIL — "found 1 item(s)"	pass
`HostStopping_ThePodHubClaim_NeverAsksOrleansForAnActivation`	FAIL — "found 1 item(s)"	pass
`ShutdownFlippingAfterTheGate_IsRefusedAsTransient_NotTerminal`	FAIL — 30 s, no NACK	pass
`HostRunning_TheGoodbyeAnnouncement_StillReachesOrleans`	pass	pass
`HostRunning_ThePodHubClaim_StillReachesOrleans`	pass	pass

Both healthy-host controls pass either way — that is the proof the gate is scoped, not blanket. A guard that refused everything would pass the three shutdown assertions and break the portal.

RecordingGrainFactory is a recorder, not a mock: the assertions are over what the router actually asked Orleans for.

Drain still completes / healthy silo still activates

Green on a real cluster: PodHubTransportTest (including AddressThatMovesSilos_ConvergesOnItsNewOwner — the case that proves a healthy Detach() must and does still happen), RoutingFailureTakesTheLocalRouteTest, OrleansRoutingShutdownClassificationTest, and OrleansGrainTeardownStragglerTest — the consequence test for this very invariant. 17/17.

Not made safe (deliberate)

OrleansMeshChangeFeed.BroadcastAsync publishes to an Orleans memory stream, which can place a MemoryStreamQueueGrain. It is not gated here: the queue holds already-accepted cross-silo invalidations, so refusing during shutdown would drop them rather than relocate them — the same "no traffic" mistake the drain carve-out exists to avoid. Orleans' own Catalog/placement gate closes it the moment the oracle flips.

Motivation

#2252 is the same defect one level up: an announcement that escaped through a healthy ancestor re-activated a grain that was mid-deactivation, so it never left the silo catalog and the straggler test timed out. Cross-refs: #1971, #2076.

🤖 Generated with Claude Code

Update: the ratchet fired on its own PR — verdict and resolution

The first CI run went red on exactly one test: GrainActivationSiteRatchetGuard.NoNewSiteAsksOrleansForAGrain. The guard caught a real new site, which is the best possible outcome for a new guard, and the answer was neither "add a line" nor "relax the predicate".

Verdict: the site is legitimate and is DRAIN. After this guard was seeded, main merged the #1742-residual-C fix, which changed RoutingGrain.PostFailure to answer a cross-silo sender through GetGrain<IPodHubGrain>(senderPath).Deliver(...) instead of publishing onto a subscriber-less stream. That is the category RoutingGrain's allowance already documents: it answers a delivery already accepted for routing, and refusing it would park a waiting sender on a silence — precisely the defect that PR fixed. It also bounds its own activation churn by asking only for stream-routed sender types, so it never mints an activation merely to be told PodHubNotHere. Raised 2 → 3.

Why this is not a widen-to-go-green. A bare count invites a reflexive bump, and a ratchet that gets bumped without thought is already dead. So the entry now costs more than a number:

the allow table carries a reason per entry, and all three RoutingGrain calls are named individually (forward leg to a pod hub, forward leg to a node hub, the NACK leg);
the MORE failure message now states the test explicitly — drain may be named and carried; a claim, a release, an announcement, or a probe must go behind the seam instead;
the guard is strictly stronger than before. TheOnlyUngatedDoorInTheRouterIsTheSeam pins that OrleansRoutingService's single GetGrain is still the gated one. Swapping the seam's call for an ungated call elsewhere in that file keeps the count at 1 and would have sailed straight through the count-based check. Verified non-vacuous: removing the gate fails it with the explanatory message.

On `PodHubTransportTest.CrossSiloNack_...` (seen once locally, not caused here)

One local run showed this main-owned test timing out at 32 s against its 30 s budget. Attributed, not chased:

it passes alone with the gate on, passes paired with each of my test classes, and the identical full set passes on re-run in 5 s;
decisively, in the failing run the test had already passed the step that proves the pod-hub claim is live across silos — the step that runs through the very Attach() this PR gates — and failed later, at RoutingGrain.PostFailure, which this diff does not touch;
structurally, the gate is IsHostStopping ? null : <original expression>, so while the host is running the path is bit-identical to main.

Load-induced timeout on a machine under memory pressure. Flagging it as a possibly load-sensitive new test rather than editing someone else's test.

Not a fix for #2301

This gate keys on the host stopping. OrleansGrainTeardownStragglerTest runs against a healthy cluster, where IsHostStopping is false and this change is a no-op — so it cannot be the fix for #2301. It passed in every local run here. Please do not close #2301 against this PR.

The invariant: "no deactivating silo should activate any grain whatsoever" — not for a routed message, not for a stream delivery, not for a "goodbye" announcement. Orleans ALREADY enforces this, and the mesh was bypassing it. Orleans 10.2.2's Catalog.GetOrCreateActivation creates an activation only while _siloStatusOracle.CurrentStatus == SiloStatus.Active (otherwise it takes the UnableToCreateActivation path and returns null), and PlacementService.GetCompatibleSilos intersects candidates with the ACTIVE silo set. So there is no parallel gate to build. What Orleans cannot know is that we have ALREADY decided to go away. IHostApplicationLifetime.ApplicationStopping fires strictly BEFORE the Orleans silo hosted service stops — which is exactly why DeliverMessage can use it as a readiness signal — so for some seconds the membership oracle still reports Active and Orleans faithfully creates any activation we ask for, ON THE SILO THAT IS LEAVING. Asking is ours to stop; refusing is Orleans'. OrleansRoutingService applied that knowledge at ONE of its three grain call sites. The other two were the pod-hub claim/release pair, three lines below the gate. The release leg is the goodbye announcement in its purest form: IPodHubGrain is [PreferLocalPlacement], so Detach() against an activation that has already gone does not release anything — it CREATES a fresh activation on the dying silo in order to tell it nothing, and that activation then belongs to the silo that is leaving. - grainFactory is now reachable only through GrainWhileRunning<T>, the seam that applies the gate. Every GetGrain in the class goes through it. - GrainActivationSiteRatchetGuard fails the build on a NEW GetGrain site in src/, so the next call site cannot reintroduce the bypass. Two allowances, deliberate and opposite: the seam itself (1), and RoutingGrain's two DRAIN calls, which stay UNGATED because a message already accepted for routing must still land and Orleans' own placement is what sends it to a HEALTHY silo (#1971 — the rule is "no new activations", never "no traffic"). The refusal is TRANSIENT, never terminal: a late token flip throws into DeliverMessage's existing Catch, which re-reads IsHostStopping and NACKs the sender as ErrorType.ShuttingDown, so SynchronizationStream's resubscribe latch rides it out instead of tearing down. Failing-first, measured: with the gate disabled the two shutdown assertions report "found 1 item(s)" (the claim and the release each asked Orleans for a PodHubGrain) and the transient-classification test times out; both healthy-host controls pass either way, so the gate is proven scoped rather than blanket. PodHubTransportTest (incl. AddressThatMovesSilos_ConvergesOnItsNewOwner) and OrleansGrainTeardownStragglerTest — the consequence test for this invariant — stay green on a real cluster, so the drain still completes. Motivating instance: #2252, where an announcement escaping through a healthy ancestor re-activated a grain that was mid-deactivation, so it never left the silo catalog. Cross-refs: #1971, #2076. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR hardens Orleans shutdown behavior to uphold the invariant “a silo that has begun stopping must not ask Orleans to create new grain activations,” by routing all grain access in OrleansRoutingService through a single stopping-aware seam and pinning the behavior with targeted tests plus a ratchet guard.

Changes:

Centralized IGrainFactory.GetGrain usage in OrleansRoutingService behind GrainWhileRunning<T>() and applied it to the pod-hub claim/release and late-flip routing path to avoid activation creation during ApplicationStopping.
Added a focused regression test suite that asserts both shutdown behavior and “healthy host still works” controls using a recording grain factory.
Added a repository-wide test ratchet that fails on any new GetGrain(...) call site under production roots (src/, memex/, tools/).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
`src/MeshWeaver.Connection.Orleans/OrleansRoutingService.cs`	Introduces a single “grain access while running” seam and uses it at previously ungated call sites to prevent new activations during shutdown.
`test/MeshWeaver.Hosting.Orleans.Test/SiloShutdownActivationGateTest.cs`	Adds shutdown-window and control tests validating “no activation requests while stopping” and correct transient NACK classification.
`test/MeshWeaver.Hosting.Orleans.Test/GrainActivationSiteRatchetGuard.cs`	Adds a governance test to prevent introduction of new `GetGrain` call sites outside the approved files/budgets.
`src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-25-a-restarting-server-stops-starting-new-work.md`	Documents the user-visible effect and rationale as a fix entry.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-08-25T20:50:26Z