Teardown drops the ANSWER, not just new work: PatchDataResponse/GetDataResponse refused at DisposeHostedHubs so the waiting caller times out

A response to an IN-FLIGHT request is refused during teardown, so the caller hangs

MeshWeaver.Graph.Test.NackReachesTheWaiterDuringTeardownTest.OwnerDisposingUnderMeshTeardown_StillAnswersTheWaitingCaller
fails intermittently with System.TimeoutException. It has now blocked two pull requests and
rejected a merge-queue group build, so core cannot merge reliably while it stands.

The mechanism, from the test's own trace

[write]   patch posted with marker teardown-nack-4de6af5289; owner merge is parked
[fence]   caller is armed on the late watch (ArmedCount=1)
[dispose] mesh disposal invoked — parent is past DisposeHostedHubs
[Warning] Message delivery failed for PatchDataResponse (ID: JKbVdl_ttECug_3A1Y2hjQ)
          in TestData/teardown-nack-node: Hub TestData/teardown-nack-node cannot route
          PatchDataResponse to cache/mFUWHEUx… — its parent hub mesh/mFUWHEUx… is shutting
          down (RunLevel=DisposeHostedHubs). The address may reactivate (recycle / restart);
          retry to get the authoritative answer.

Three deliveries are refused the same way in the same millisecond — DataChangedEvent ×2,
PatchDataResponse, and a GetDataResponse to client/8ae4ca32e32b.

The refusal does not distinguish new work from an answer owed to a caller already waiting. The
guard's own text — "retry to get the authoritative answer" — is advice a request sender can
act on. A response has no such option: the caller is parked on the reply, the request will never
be re-sent, and the reply is the thing being dropped. So the caller waits out its budget for an
answer the owner believes it sent. PatchDataResponse and GetDataResponse are answers, not work.

Why this is NOT a flake, and must not be catalogued as one

I nearly added it to .github/known-flakes.json to get a delivery-blocking fix through, and that
would have been wrong: cataloguing teaches the steward to re-queue this failure fleet-wide, and
the failure is real. The test is a detector doing its job — the name says exactly what it asserts
("still answers the waiting caller"), and the log shows the answer being dropped. What is
intermittent is the RACE (whether disposal reaches DisposeHostedHubs before the response is
routed), not the correctness of the complaint.

Almost certainly the same family as #3298

PodHubTransportTest.CrossSiloNack_ReachesASenderWhoseStreamSubscriptionIsGone fails the same way
one layer out: a NACK that must reach a sender during teardown does not arrive, and the requester
spends its whole budget. Different assembly, different transport, same sentence — an answer owed to
a waiting caller is dropped once the owner is shutting down. Worth investigating together; I have
not proven they share a root cause, and I falsified my first attempt at linking #3298 to a
disposal change, so
treat the connection as a hypothesis.

What I ruled out

Newly-introduced test. No — added 2026-09-02 by 51c158bd9 fix: the patch-ack completion arm stands aside for a shutting-down owner, 357 commits back.
Caused by Teardown lets accepted work finish and names what it had to stop — no forced teardown #3291 (fix/robust-disposal, merged 12:26Z). Not established. A tree containing
Teardown lets accepted work finish and names what it had to stop — no forced teardown #3291 passed the sibling shard, and a tree without it failed Flake: PodHubTransportTest.CrossSiloNack_ReachesASenderWhoseStreamSubscriptionIsGone — red on shard 0 with a control arm that isolates it #3298's test. Same-window
correlation is not evidence.

Suggested direction

DisposeHostedHubs should stand aside for replies to in-flight requests the way the commit that
added this test made the patch-ack arm stand aside — the shutdown guard is right to refuse new work
and wrong to refuse an answer. Evidence:

queue build https://github.com/Systemorph/MeshWeaver/actions/runs/33877631698 (job 101039142435, shard 4), which dequeued fix(ci): the bundle identity anchor is a platform property, not a module's reference graph #3293 with queue-rejected
the same test also blocks fix(i18n): the provider page must not claim an unpriced model is unbilled #3297

The hypothesis in the issue is CONFIRMED, and the fix already landed

Root cause, verified end-to-end. HierarchicalRouting.RouteAlongHostingHierarchy refuses every
delivery it would have to hand to a parent past DisposeHostedHubs and returns
delivery.Failed(…, ShuttingDown). That failure is reported by MessageService.ReportFailure,
which — like every other failure route here — answers delivery.Sender. For a REQUEST that is
exact. For a reply the sender is the RESPONDER: it is waiting for nothing, will never re-send,
and the party actually parked on the message is never told. Two of ReportFailure's own exits then
drop it in silence (the RunLevel >= DisposeHostedHubs gate, and MayAnswer() — which asks whether
the SENDER wants an answer, not the question a reply poses).

The refusal's own advice — "retry to get the authoritative answer" — is a claim about its
addressee, and on a reply that claim is false. That is why this read correctly in the source for
as long as it did.

Why #3196's claim-then-verify could not catch it. AckOnce latches its once-only gate before
posting, which disables RegisterOwnerDisposingNack's sink dispatch; PostPatchVerdict compensates
by falling through to the sink when the post is REFUSED. But it reads the delivery hub.Post
returned, and here the post is ACCEPTED — the owner's own run level is still open. The refusal
happens a turn later in routing, invisible to that seam. RoutePatchVerdict's own remarks name the
gap and locate the fix exactly where it landed:

The teardown hole is real and belongs one layer down: a correlated reply that
HierarchicalRouting cannot forward is DROPPED with nobody told, and the process still holds the
registry that could take it.

Why it lands only under load. Mesh.Dispose() synchronously cascades CloseCreation through
the whole subtree — every descendant's IsShuttingDown flips while the mesh is still at Started —
then posts Quiescing to its own loop. The parked merge turn wakes on that first signal, and its
ack is not one step either: hub.Post is evaluated on the owner's turn and ROUTED on a later one.
On an idle machine the merge wins; on a loaded shard the mesh reached DisposeHostedHubs in 3 ms.
I measured 15/15 local passes with the defect fully present.

What landed

684cc6ac1 (#3302) — IUndeliverableReplySink, offered by HierarchicalRouting on the two arms
where it gives up, implemented by LatePatchResponseRegistry as a factory over the same singleton.
Its branch is named for a quiescing gate that was dropped before merge; the change that shipped is
the hand-over alone. Core CD is running on it now, first delivery since 10:23Z.

What was still missing, and is in #3309

The fix shipped with no guard that can fail. Its only cover is
NackReachesTheWaiterDuringTeardownTest — the detector that found this — which reaches the state by
racing. Measured, with both HierarchicalRouting call sites disabled and the two tests run
together:

[xUnit.net 00:00:38.14]  RefusedReplyReachesTheWaiterTest…StillReachesTheArmedWatch [FAIL]
  Expected the observable to emit a value within 36s … it did not.
  [probe] owner → waiter route is live while nothing is shutting down
  [fence] parent=DisposeHostedHubs owner=Started — a reply posted now is accepted by the owner and
          refused by routing
  [Warning] Message delivery failed for PatchDataResponse (ID: _LUemby4UkWe5wYCX4OZdg) in
          refused-reply-owner/1: Hub refused-reply-owner/1 cannot route PatchDataResponse to
          refused-reply-waiter/1 — its parent hub refused-reply-parent/1 is shutting down
          (RunLevel=DisposeHostedHubs). …

Failed!  - Failed: 1, Passed: 1, Skipped: 0, Total: 2

Passed: 1 is the field detector, with the production fix removed. It printed the same refusal
it complains about in CI and passed anyway.

RefusedReplyReachesTheWaiterTest holds the state open by construction — the owner's action block
is parked on an accepted turn, so its own ShutdownRequest queues behind that turn and the owner
cannot leave Started, while the parent cannot leave DisposeHostedHubs until the owner finishes.
Both halves are fenced on. A LIVE waiter hub gives it two controls that do not depend on which layer
performs the hand-over: a probe proves the route carries this message before the teardown, and the
waiter's silence afterwards proves the reply never travelled.

Also there: UndeliverableReplyShapesTest (which shapes the registry takes — including the
DeliveryFailure arm of #2661, which nothing else covered), and
Doc/Architecture/RefusedRepliesDuringTeardown, which states the asymmetry as a rule so the next
refusal, intake gate or teardown drop asks who is waiting, not who sent it.

#3298 is a different defect — see the comment there

Same sentence, different mechanism, on three independent counts. Not fixed by this, and it stays
open.

test(#3303): key the watch assertions on THIS request id, not on the …

Closing — the defect is fixed in `main`, and I measured it rather than assuming it

684cc6ac1 (#3302) carries the hand-over. Verified on the merged tree, not on the PR:

$ git log --oneline origin/main -1
684cc6ac1 Merge pull request #3302 from Systemorph/fix/3261-quiescing-refuses-new-requests
$ grep -n "TryHandOverUndeliverableReply" src/MeshWeaver.Messaging.Hub/HierarchicalRouting.cs
37:    private bool TryHandOverUndeliverableReply(IMessageDelivery delivery)
173:                if (isDisposing && TryHandOverUndeliverableReply(delivery))
248:            if (TryHandOverUndeliverableReply(delivery))

And proved that it is what closes THIS defect, by removing it: with both call sites disabled the
deterministic reproduction fails on exactly the sentence in this issue, and restoring them makes it
pass in under a second. Core CD is running on 684cc6ac1 — the first delivery since 10:23Z.

The hypothesis in the issue body was right, including the part that mattered most: the guard's own
advice, "retry to get the authoritative answer", is a claim about its ADDRESSEE, and on a reply
that addressee is the responder rather than the caller. That is why it read correctly in source for
as long as it did.

#3309 adds what the fix shipped without: RefusedReplyReachesTheWaiterTest (the state held open
by construction, so it fails without the race), UndeliverableReplyShapesTest (the DeliveryFailure
arm of #2661, which nothing else covered), and Doc/Architecture/RefusedRepliesDuringTeardown. That
is hardening on top of a fix that is already in main, so it does not hold this issue open.

#3298 is not this — different funnel (RoutingGrain.PostFailure, never HierarchicalRouting),
different precondition (nothing is shutting down), different waiter (a RegisterStream callback on
another silo, not an armed registry entry). It stays open; details in the comment there.

Navigation Menu

Repository navigation

Teardown drops the ANSWER, not just new work: PatchDataResponse/GetDataResponse refused at DisposeHostedHubs so the waiting caller times out #3303

A response to an IN-FLIGHT request is refused during teardown, so the caller hangs

The mechanism, from the test's own trace

Why this is NOT a flake, and must not be catalogued as one

Almost certainly the same family as #3298

What I ruled out

Suggested direction

The hypothesis in the issue is CONFIRMED, and the fix already landed

What landed

What was still missing, and is in #3309

#3298 is a different defect — see the comment there

Closing — the defect is fixed in `main`, and I measured it rather than assuming it

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Teardown drops the ANSWER, not just new work: PatchDataResponse/GetDataResponse refused at DisposeHostedHubs so the waiting caller times out #3303

Description

A response to an IN-FLIGHT request is refused during teardown, so the caller hangs

The mechanism, from the test's own trace

Why this is NOT a flake, and must not be catalogued as one

Almost certainly the same family as #3298

What I ruled out

Suggested direction

Activity

rbuergi commented on Sep 4, 2026

The hypothesis in the issue is CONFIRMED, and the fix already landed

What landed

What was still missing, and is in #3309

#3298 is a different defect — see the comment there

rbuergi commented on Sep 4, 2026

Closing — the defect is fixed in main, and I measured it rather than assuming it

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Closing — the defect is fixed in `main`, and I measured it rather than assuming it