Flake: PodHubTransportTest.CrossSiloNack_ReachesASenderWhoseStreamSubscriptionIsGone — red on shard 0 with a control arm that isolates it #3298

Activity

1. rbuergi commented on Sep 4, 2026

   

   rbuergi

   on Sep 4, 2026

   ContributorAuthor

   More actions

   The assertion, verbatim

   System.TimeoutException : Timed out after 00:00:30: a request the router gave up on must come back
   to its cross-silo sender as a DeliveryFailure. Silence here is issue #1742 exactly: the NACK was
   published onto the sender's own subscriber-less stream, succeeded, and was discarded — so the caller
   waits out its whole budget for an answer the router believes it sent
     at PodHubTransportTest.WaitUntil(...) PodHubTransportTest.cs:372
     at PodHubTransportTest.CrossSiloNack_ReachesASenderWhoseStreamSubscriptionIsGone() :311
   

   Assembly summary: Total: 251, Errors: 0, Failed: 1, Skipped: 0, Time: 62.589s — so the host ran
   normally and exactly this one test timed out. The test's own two controls (a cross-silo PingRequest
   proving the pod-hub claim is live, and a unique stray- token proving the subscriber-less stream
   really cannot deliver) both PASSED before the failing wait, which is what makes this a clean signal:
   the setup was in the intended state and the directed NACK still did not arrive within 30 s.

   A lead worth checking first

   main landed teardown/quiesce work in the same window this failed
   (Doc/Architecture/TeardownLayers.md, ActivityTrackerQuiesceTest, HostedHubDisposalJoinTest,
   MessageHubTest). This test's whole subject is a NACK racing a sender whose subscription is already
   gone — teardown ordering is the same neighbourhood. Two possibilities worth separating before
   assuming an old race:

   1. the flake predates that work and is the shard-0 Orleans cluster resurfacing, or
   2. the teardown change altered when the sender's route is torn down relative to PostFailure.

   main's own Build and Test was green at b0d4a826f (12:46Z), so whatever this is, it is
   intermittent rather than a standing regression — which does not make it less real.

   Status of the PR that surfaced it

   #3293 was behind main by 8 commits. I merged main in rather than re-running the failed job, per
   AGENTS.md's "merge main only when the PR is DIRTY or CI fails on something your diff cannot reach" —
   that re-runs CI against a tree main has proven green instead of rolling the dice on the same tree.
   If it comes back red on this test again, that is a much stronger signal than the first occurrence and
   should be treated as reproducible.
2. rbuergi commented on Sep 4, 2026

   

   rbuergi

   on Sep 4, 2026

   ContributorAuthor

   More actions

   Correction: my own "check the teardown work first" lead is falsified

   Earlier I suggested #3291 (fix/robust-disposal, merged to main 12:26:03Z) as the first thing to
   check, on the grounds that it landed in the same window and this test's subject — a NACK racing a
   sender whose subscription is gone — is teardown-shaped. That correlation does not survive the
   ancestry check:

   PR head	contains f9d2a335a (#3291)?	Run tests (shard 0)
   616e39613	no	success
   4054f1ec9	no	FAILURE — this test
   061b020c1	yes	success (0 red, 0 pending)

   The tree that failed did not contain the disposal work, and the tree that did contain it passed.
   So #3291 is not implicated, and the run carrying it is evidence against the story rather than for
   it. I was pattern-matching on "teardown-shaped test + teardown-shaped change in the same hour", which
   is [[frequency is not causation]] with better vocabulary.

   What stands unchanged: the failure is intermittent and reproducible-by-luck rather than by tree —
   two heads whose only difference is a string-assertion test in a different assembly disagree on it.
   Both of the test's own controls passed before the failing wait, so the setup reached the intended
   state and the directed NACK still did not arrive within 30 s.

   What to start from now, absent the false lead: the pod-hub ShuttingDown verdict path and the
   shard-0 Orleans flake cluster (recorded as only partly resolved) remain the standing candidates, but
   neither is evidenced yet — treat them as untested guesses, not as leads.

   Related but almost certainly separate, filed here only so the next reader can rule them in or out
   rather than rediscover them: NackReachesTheWaiterDuringTeardownTest.OwnerDisposingUnderMeshTeardown_StillAnswersTheWaitingCaller
   (a TimeoutException, currently blocking core #3297) and
   MeshWeaver.PluginTester.Test.CascadeTest.IndependentNodesRunInParallel_BoundedByTheSlotCount
   (which failed a merge-queue group build and got #3296 queue-rejected). Three intermittent failures
   in one afternoon is worth someone asking whether they share a cause — but I have no evidence that
   they do, and saying otherwise would repeat the mistake above.
3. rbuergi commented on Sep 4, 2026

   

   rbuergi

   on Sep 4, 2026

   ContributorAuthor

   More actions

   Checked against #3303, and they are NOT the same defect

   #3303 asked whether this shares a root cause with
   NackReachesTheWaiterDuringTeardownTest — "an answer owed to a waiting caller is dropped once the
   owner is shutting down". That sentence fits both. The mechanism does not, on three independent
   counts, each read off the code rather than inferred:

   	#3303	#3298 (this)
   funnel	HierarchicalRouting.RouteAlongHostingHierarchy — the arm that refuses to hand a delivery to a parent past DisposeHostedHubs, reported by MessageService.ReportFailure	RoutingGrain.PostFailure — local route → directed pod-hub call → grain transport → stream fallback. It never touches HierarchicalRouting.
   precondition	a hub in the chain is past DisposeHostedHubs	nothing is shutting down. This test erases a stream subscription on purpose, on a healthy cluster, and both of its own controls (the cross-silo PingRequest arriving, the stray- token NOT arriving) pass first.
   the waiter	an armed LatePatchResponseRegistry entry in the same process	a RegisterStream callback in the routing service's local route table on another silo

   The third row is the decisive one. #3303's cure is an in-process hand-over to a registry the waiter
   armed before the request was posted. Here nothing arms that registry — the waiter is a stream
   registration for a PingRequest, and the message is a router-minted DeliveryFailure. So even if
   the hand-over ran on this path it would find nothing to hand the NACK to, and it does not run on
   this path at all.

   #3302 / 684cc6ac1 therefore does not fix this, and this issue stays open. I have not re-run it
   (AGENTS.md), and I have not catalogued it in .github/known-flakes.json.

   Where I would start, given the above

   Not with the teardown work — the same ancestry check that falsified the #3291 lead applies here, and
   this test's premise is a HEALTHY cluster. The subject is a directed pod-hub call that either lands
   or answers, so the question is which of PostFailure's four transports the NACK took and what that
   one returned. RoutingGrain.PostFailure writes a trace tag per branch
   (FAILURE_LOCAL_ROUTE, and the pod-hub / grain / stream arms below it) precisely so this is
   readable rather than inferable — a run that captures those tags for the failing delivery id would
   discriminate "the pod-hub activation answered PodHubNotHere" from "the grain resolve gate declined
   and it fell back to the subscriber-less stream" (which is #1742's residual, wearing its cure's
   clothes) without another guess.

   The pod-hub ShuttingDown verdict lead recorded above is still untested, and worth noting it
   requires a shutting-down pod hub that this test does not create — so it needs its own evidence
   before it is a lead rather than a neighbour.
4. rbuergi commented on Sep 4, 2026

   

   rbuergi

   on Sep 4, 2026

   ContributorAuthor

   More actions

   Fixed — #3313, merged as 42f0cbc4edd11492a0b285ce3af2fbe4f72fe976

   (Closing by hand: the merge queue does not fire Closes links, so the keyword in the PR body did
   not act.)

   Root cause: a missing precondition in the test, not a transport defect.

   RegisterStream establishes two things on different clocks — the local route synchronously, and the
   cluster-wide pod-hub claim asynchronously, on a capped backoff (100 ms → 2 s) with no give-up on a
   silo. Until the claim lands, [PreferLocalPlacement] puts a directed IPodHubGrain call on the
   caller's silo, which has no local route for the address, and it answers PodHubNotHere.
   PostFailure then falls back to the stream — which this test has deliberately emptied — and the NACK
   is discarded in silence. That is the reported failure exactly: a 30 s timeout with both controls green.

   Why the probe could not catch it, which is the part worth keeping. It proved reachability, and
   during that window reachability is satisfied by the stream — the transport the test erases three
   lines later. So it was green whether or not the claim had landed. It is also adversarial to the
   claim it appeared to prove: it posts from the other silo in a loop, and each failed directed call
   mints a throw-away activation there that the owner's next Attach must bounce off, restarting the
   backoff. Hence "reproducible by luck rather than by tree" — the control arm in this issue was reading
   a scheduling race, correctly.

   Measured rather than argued. With one bounce forced and the claim backoff pinned long, the old
   probe goes green in under a second while PodHubClaimSettled has demonstrably not completed —
   6 runs, 6 times. I did not commit that experiment: its setup depends on an activation-lifetime
   ordering I cannot enforce, so as a permanent test it would risk becoming the next flake.

   The fix waits on PodHubClaimSettled, the positive signal for "the claim stopped attempting" —
   which OrleansCrossSiloReplyTest already waits on before its own directed cross-silo delivery.
   SiloHostedHub_ReceivesDelivery_EvenWithItsStreamSubscriptionGone had the same latent gap (it waits
   for the stream subscription, a different condition that merely takes similar time) and got the same
   wait. No production changes.

   Your three standing candidates, all ruled out on evidence rather than left hanging:

   • Teardown lets accepted work finish and names what it had to stop — no forced teardown #3291 — already killed by your own ancestry check; I did not revisit it.
   • Teardown drops the ANSWER, not just new work: PatchDataResponse/GetDataResponse refused at DisposeHostedHubs so the waiting caller times out #3303 / A reply routing cannot deliver goes to the waiter, not the floor — and a quiescing hub stops taking work it can never answer #3302 — your discrimination held; 684cc6ac1 does not fix this, and the cause was
     elsewhere entirely.
   • the pod-hub ShuttingDown verdict — not implicated. It needs a shutting-down pod hub, and this
     test's premise is a healthy cluster. It stays a neighbour, not a lead.

   The rule is written up in Doc/Architecture/OrleansTestRoutingPattern → "Reachability is not a
   claim — wait for the transport you are about to depend on", so the next test that isolates one
   transport to assert another finds it before filing a flake.

   It was not catalogued in .github/known-flakes.json, and should not be — it was a real defect
   with a deterministic cause.

   🤖 Filed from Claude Code.
5. rbuergi

   closed this as completedon Sep 4, 2026