Repository navigation
Orleans ClientDirectory fails to push client routing table to silo — 30s timeout, stale routing view cluster-wide #2906
Description
What is failing
The Orleans ClientDirectory on the memex-portal pod fails to push its client routing table to one silo: the directed call to IRemoteClientDirectory.OnUpdateClientRoutes received no response within the 30s Orleans response timeout. Silos that miss the update hold a stale view of which clients are connected where, which can misroute or drop messages aimed at clients until the next successful publish.
Probable cause
Medium confidence, transient: the target silo (10.244.4.108:11111) was unresponsive — most plausibly draining or restarting during a rolling deploy / membership change. The grain/client directory is documented as exactly the component that goes unstable in that window (see Pod-Hub Delivery — the Transport Swap and its Roll Plan, "What the swap traded"), and the portal pod is an Orleans client, whose ClientDirectory publishes to every silo on each membership change. The stack is entirely inside Orleans.Runtime (framework code), with no MeshWeaver frame on it — so this is not our RoutingGrain/pod-hub delivery path failing. Alternative explanation if the silo was healthy: genuine network partition or an overloaded silo. With a single occurrence the evidence cannot distinguish these.
Impact
Minimal so far: one occurrence, one pod, one timestamp. The ClientDirectory publish loop retries, so a single missed publish self-heals. It would matter only if it recurs on a stable (non-deploying) cluster — that would point at a genuinely wedged silo rather than deploy churn.
Where to look
Orleans.Runtime.GrainDirectory.ClientDirectory.PublishUpdates(src/Orleans.Runtime/GrainDirectory/ClientDirectory.cs, line 499 per the stack) — the retry/timeout behaviour of the publish loop.- Check whether silo
10.244.4.108:11111was being drained or restarted at the sample timestamp, and correlate with deploy events for thememex-cloudnamespace — if a deploy was in flight, this is expected churn noise and the ticket can be closed as such. - If it recurs outside deploy windows, capture silo health (CPU/GC pause) on the target silo for the 30s window.
Evidence
| Fingerprint | 39005f6f08378861 |
| Category | Orleans.Runtime.GrainDirectory.ClientDirectory |
| Severity | Error |
| Exception | System.TimeoutException |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-9bf79b774-s9sr9 |
| Occurrences | 1 |
| First seen | 2026-08-31 15:42:03Z |
| Last seen | 2026-08-31 15:42:03Z |
Recent log lines
2026-08-31 15:42:03Z memex-portal-deployment-9bf79b774-s9sr9 fail: Orleans.Runtime.GrainDirectory.ClientDirectory[0]
Exception publishing client routing table to silo S10.244.4.108:11111:147126036
System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.4.53:11111:147159071 sys.svc.dir.client/10.244.4.53:11111@147159071]->[S10.244.4.108:11111:147126036 sys.svc.dir.client/10.244.4.108:11111@147126036] Orleans.Runtime.GrainDirectory.IRemoteClientDirectory.OnUpdateClientRoutes(System.Collections.Immutable.ImmutableDictionary`2[Orleans.Runtime.SiloAddress,System.ValueTuple`2[System.Collections.Immutable.ImmutableHashSet`1[Orleans.Runtime.GrainId],System.Int64]]) #45D413CF416B6CE2.
at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
at Orleans.Runtime.GrainDirectory.ClientDirectory.PublishUpdates() in /_/src/Orleans.Runtime/GrainDirectory/ClientDirectory.cs:line 499
Opened automatically from Admin/_LogIncident/39005f6f08378861. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 1, 2026
Tested this issue's own closing criterion — "close it if it lines up with churn" — and it does not line up. Recording the negative result so the next reader does not repeat the check.
What I measured
The occurrence is 2026-08-31 15:42:03Z. Commits merged to
mainin the surrounding window, with the first CD run for each:b88d705f93d34187bf7199a4The 15:38 merge is the tempting explanation — four minutes before the failure — but its CD run did not start until 15:53, eleven minutes after the ClientDirectory timeout. No deploy was rolling at 15:42. The deploy-window hypothesis is falsified, not merely unconfirmed.
What that leaves
The stack is entirely inside
Orleans.Runtime.GrainDirectory.ClientDirectory— vendored framework code, retry-protected — so a single occurrence is still not actionable on its own. But the cheap explanation is now off the table, which changes what a recurrence would mean: if this recurs outside a deploy window it points at a genuinely wedged silo rather than expected membership churn, and that is worth a real investigation — a dead silo in membership strands every partition's hub activations.Not checked — needs cluster access
A rolling deploy is not the only source of membership change. KEDA scaling events, node-pool operations and pod evictions could all produce one at 15:42, and none are visible from the repo. Settling that needs
az aks command invokeagainst the cluster (it is private, so that is the only route forkubectl), plus the silo membership table around that timestamp.Leaving open. Concretely, what would close it: either a cluster-side event at 15:42:03Z explaining the churn (close as expected behaviour), or a second occurrence with no churn at all (escalate).
Note: an earlier version of this comment was posted mangled — my shell evaluated the backticked spans. Corrected here; the measurement is unchanged.
rbuergi commentedon Sep 1, 2026
The missing input is now available, and it settles this — closing
The previous comment named exactly what was missing: "a cluster-side event at 15:42:03Z explaining the churn (close as expected behaviour), or a second occurrence with no churn at all (escalate)", and said it needed
az aks command invokeplus silo health. I have that access now. Both halves came back on the "close" side, and the mechanism is more specific than "expected churn".First: the instrument can see the event
Loki's
retention_periodis 31d, so 2026-08-31 is comfortably inside it. That matters more than the finding itself — it is what licenses reading an absence below as evidence rather than as a failed search.Correction to this issue's own evidence table
It records 1 occurrence, 1 pod. Loki has 4 lines across 2 pods:
…-qlks7vmss00001f…-qlks7vmss00001f…-s9sr9vmss00001g…-s9sr9vmss00001gAll four name the same target silo,
S10.244.4.108:11111:147126036. Two clients failed to publish to one silo — which already points away from "the reporter is sick" and at "the target is".The cluster-side event: the target silo was scheduler-stalled
Silo
10.244.4.108was pod…-jwrngon nodevmss00001g. Counting Orleans' long-turn warnings (took elapsed time) on that pod:A ~50× monotonic ramp, peaking in the bucket containing the 30-second timeout, then collapsing. A silo whose turns are taking seconds cannot answer
OnUpdateClientRoutesinside 30 s. TheClientDirectoryerror is the symptom; the stall is the event.Concurrently, from the node resource group's activity log (
MC_memex-aks-rg_memexaks-cluster_swedencentral— the cluster RG shows onlyrunCommandand reads as "no churn", which is a trap worth naming):So there was a membership-changing infrastructure operation spanning the failures — just a node-pool operation, not a deploy. The previous comment falsified the deploy hypothesis correctly; it did not falsify churn as such. Nodes
vmss00001fandvmss00001gare both gone today.And no recurrence
count_over_timeover the whole 44 h window 2026-08-31T00:00Z → 2026-09-01T19:45Z returns hits in exactly two 1-hour buckets, both on 08-31, totalling those 4 lines. Zero occurrences in the 28 h since — measured against a retained index, not inferred from a failed grep.Verdict — closing, and where the residue goes
This is a retry-protected framework fault that fired 4 times in 9 minutes against a silo that was independently degrading, and has not fired since. There is nothing left to fix here.
But it should not close as "expected churn", because the interesting half is the stall: a silo can ramp to 439 long-turn warnings per 10 minutes and stay in the Service endpoints answering
/alive. That is #2194 item 4 (progress-aware/alive) precisely, and I have added this as a second dated instance there — #2194 stays open and now measures worse than filed.Two of the three follow-ups this window produced are also recorded elsewhere: the same silo/directory degradation is visible in #2915's registration window, and the method is conserved as
Doc/Architecture/MeasuringALivePortalReadOnly.(All measurements read-only: Loki query API,
kubectl get,az monitor activity-log list. Nothing was mutated.)rbuergi commentedon Sep 1, 2026
Closing per the measurement above: the cluster-side event is identified (target silo scheduler-stalled, 9→439 long-turn warnings per 10 min peaking at the failure minute, concurrent with a VMSS node-pool operation 15:28:44Z→15:40:29Z), and there has been no recurrence in the 28 h since — measured against a Loki index whose retention (31 d) covers the window. The degraded-but-Ready half is carried forward on #2194.
systemorph commentedon Sep 23, 2026
Reopened — this fault is happening again.
Still occurring: 1 more since the last update (2 total).
memex-portal-deployment-777677694-p6hbn,memex-portal-deployment-9bf79b774-s9sr9Recent log lines
Re-addressed by the current identity function: this incident inherited
39005f6f08378861(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Sep 23, 2026
Reopen read: roll-window churn again; #5551 folded here. Disposition needs an owner decision
777677694-p6hbn→ silo10.244.9.136) and Orleans ClientDirectory.PublishUpdates logs unhandled TaskCanceledException when publishing client routing table to an unreachable silo #5551'sTaskCanceledExceptionat 14:21:48Z (6968cb99df-jrlb9) fall in one memex-cloud roll window. The pods rolling around it were867d79674d(installing 13:56–14:03Z),6968cb99df(stopping ~14:21–14:24Z) and7d7f4d84f9(booting 17:52Z). Orleans ClientDirectory.PublishUpdates logs unhandled TaskCanceledException when publishing client routing table to an unreachable silo #5551 is closed as a duplicate. Same site, the other terminal of the same call.Orleans.Runtime(10.3.1): the client-directory gossip publishing to a silo that is leaving membership. MeshWeaver has no frame to fix. The 2026-09-01 analysis stands. The one MeshWeaver-side follow-up (a stalled silo staying Ready) is on RCA 2026-08-25: sync/bake recompile waves fatten type-hosting silos to 20 GB — superseded builds stay rooted behind the Recycle banner, and a degraded pod never leaves rotation #2194.sev:L. The publish loop retries by itself, and no user path is affected.Owner decision (not code): Orleans logs this at Error for an expected membership transition. There are two choices:
Orleans.Runtime.GrainDirectory.ClientDirectoryError lines inside a membership-change window as roll churn.A category-wide log filter in the portal would also hide the genuine wedged-silo case the 09-01 comment describes, so I have not proposed one.
rbuergi commentedon Sep 24, 2026
Refs Systemorph/MeshWeaver.Plugins#2354 — option 1 of the 2026-09-23 owner decision is now implemented for the memory-stream reader site only (#4802 family), per the maintainer's 2026-09-24 ask: a burst is classified
RollChurnonly when a pod of the same workload loggedApplication is shutting down...within 5 min before it, and is recorded as a Suppressedroll-churn-{siteFold}incident; without a preceding exit it files as before. This issue's own site (Orleans.Runtime.GrainDirectory.ClientDirectory) is NOT covered — extendingRollChurnClassifier.IsMemoryStreamReadFailureto it is a one-predicate change, but that is the owner's call because the 2026-09-01 analysis found an occurrence here with no roll (a scheduler-stalled silo). Left open.systemorph commentedon Sep 26, 2026
Still occurring: 1 more since the last update (3 total).
memex-portal-deployment-777677694-p6hbn,memex-portal-deployment-9bf79b774-s9sr9,memex-portal-deployment-7596dd76bb-cbbj8Recent log lines
Re-addressed by the current identity function: this incident inherited
39005f6f08378861(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Oct 8, 2026
The 2026-09-24 06:29:27Z sample targets
S10.244.9.67:11111:149185698. In the same minute, that silo blocked a join (#5712, folded into #5300). So this is the stalled-but-Activesilo case, which the 09-01 analysis said should stay visible, not roll churn. No recurrence since. The owner decision (whetherClientDirectorytimeouts join roll-churn classification) is still open.sev:L sweep, Claude Opus 5.5 (agent session).
rbuergi commentedon Oct 10, 2026
I resolved the open owner decision by following the recommended option. It is implemented in Systemorph/MeshWeaver.Plugins#3287.
Design choice: churn expected inside a declared roll window is recorded with a count (a Suppressed
roll-churn-{siteFold}incident) and not ticketed. Anything outside a roll, or persisting past it, keeps filing. I picked this per the maintainer's standing instruction to follow recommendations.The
ClientDirectorypublish timeout is now the second siteRollChurnClassifiercovers, alongside the memory-stream reader. The rule is the same: it counts as roll churn only when a pod of the same workload loggedApplication is shutting down...in the 5 minutes before the burst. The stalled-but-Activecase this issue's 2026-09-01 and 2026-09-24 analyses said must stay visible therefore still files. It has no preceding exit, and the PR pins that with a negative-control test.The change is live once memex and memex-cloud run a portal and
mw-log-watcherimage that carries it. Leaving this open until a roll after that is read back.w8d, Claude Opus 5.5 (agent session).
rbuergi commentedon Oct 10, 2026
Post-roll check 2026-10-10 ~20:45Z: the classification is live, and two rolls have passed under it without either site firing. Nothing to read back yet. Leaving open (same for #5300).
e6986670c5) is an ancestor of Plugins1a98ffa, which image3.1.10472was built from (CD run 38079087271). memex-cloud runs that image on all 3 replicas. Not established: whichmw-log-watcherimage runs.Logson control, 19:15Z to 20:45Z, queryClientDirectory|MembershipAgent\[100661\]|roll-churn|RollChurn:Ops/Actions/logs-memex-cloud-20261010-2110-2906-rollchurn-sites): 0 lines, not cut.Ops/Actions/logs-memex-20261010-2110-2906-rollchurn-sites): 0 entries landed.ClientDirectorypublish timeout nor the join-time connectivity validation occurred during these rolls. A zero here says the sites were quiet, not that the classifier works.Closes on: a roll in which one of the two sites fires and is recorded as a Suppressed
roll-churn-…incident with a count and no new ticket comment. Re-check after the next few rolls.rbuergi commentedon Oct 11, 2026
Closing: the last occurrence (2026-09-24) predates the classification that is live since 2026-10-10 ~19:50Z, and the site has not fired in the 16 hours of rolls since.
Reading, 2026-10-11 11:10Z, governed
Logson the control instance, window 2026-10-10 19:10Z → 2026-10-11 11:10Z, queryClientDirectory|MembershipAgent\[100661\]|roll-churn|RollChurn, neither read cut:Exception publishing client routing table)MembershipAgent[100661])Ops/Actions/logs-memex-cloud-20261011-w21-membership-sitesOps/Actions/logs-memex-20261011-w21-membership-sitesThe lines that did match are other text containing the query's words:
Host is shutting downdelivery failures andgiving up on owner …roll-churn-…stream lines at each roll, and fivedumpasynclines printed by a dump-analysis Job. The window is not quiet:Failed to address message(the #5336 site) fired 62 times on memex-cloud in the same 16 hours (Ops/Actions/logs-memex-cloud-20261011-w21-5336-address), in nine bursts at nine different instants. So silos left repeatedly in this window and the client-directory publish did not time out once.What is in place (MeshWeaver.Plugins#3287, in image
3.1.10472): this site is recorded with a count inside a roll window and still files outside one. A publish that times out against a silo that is stalled butActive, with no preceding shutdown line, therefore reopens this issue by itself.Not established by this read: that the classifier suppresses a real occurrence in production. The site has not fired since it went live, so there was nothing to classify. The classifier's own tests (with their negative control) are the evidence for that half.
On the question this issue kept open ("a stalled silo that stays
Active"): the 2026-10 evictions on #6432 are one concrete way a silo gets there. Its root container lock is held from boot, it answers nothing, and membership needs minutes to vote it out. That is tracked on #6432.w21, Claude Opus 5.5 (agent session). Read-only.