Repository navigation
Silo 5j9gx marked Dead by peers 26s after its last heartbeat and terminated itself — recurrence of the false-positive membership kill (#3053) #6432
Description
What is failing
An Orleans silo in memex-portal-deployment (Silo_b7daa, 10.244.6.128:11111:150624604, pod memex-portal-deployment-85587f8869-5j9gx) found itself Dead in the membership table during a Refresh pass and honoured the verdict by terminating itself (MembershipTableManager.KillMyselfLocally → FatalErrorHandler.OnFatalException; Exception: null and the Environment.get_StackTrace() frame confirm an intentional process stop, not a crash). The entry shows IAmAliveTime=08:29:36.513 against SuspectTimes=08:29:10.921 — the silo was still alive and heartbeating ~26 seconds after the votes that declared it dead. This is the same false-positive membership kill already triaged and filed as #3053 (and #3052) on 2026-09-02; this is a recurrence, not a new defect shape.
Probable cause — moderate-to-high confidence, same as #3053
The suspicion votes were cast from a stale/shared liveness snapshot, most likely during rolling-deploy churn: the pod's StartTime is 08:10:04 GMT, so the replica set was only ~20 minutes old when the kill fired — a deploy window. Two markers from the entry support the stale-suspect theory rather than a genuine crash:
- All three
SuspectTimesare identical to the millisecond (08:29:10.921), and one suspecter (10.244.17.58:11111:150624528) appears twice with the same timestamp — one snapshot, not three independent probe failures. (Identical pattern to the Sept 2 incident.) - The suspect votes predate the target's own live heartbeat by ~26 s; the target kept writing
IAmAliveand only exited at 08:30:03, ~53 s after being declared dead.
The open question carried over from #3053: the membership adoption path does not appear to discount suspect votes that are older than the target's own live IAmAliveTime. This time the verdict was consumed in Refresh (Sept 2 was CleanupTableEntries), so both call sites route through CheckIfLocalSiloIsDead → KillMyselfLocally with the same stale-suspect exposure. The alternative benign explanation — a long GC/CPU stall on 5j9gx making a genuine suspicion correct-but-harsh — cannot be excluded from this fingerprint alone; check the pod's own logs for GC pause warnings and the restart reason before ruling.
Impact
One occurrence, one pod, self-terminated and rescheduled by Kubernetes; the cluster remained quorate. However, this is the third firing of this shape (2026-09-02 ×2, now 2026-10-10) — it appears to recur around deploys, and each false kill cascades reactivation of every grain and pod-hub the silo hosted plus in-flight delivery failures. Watch the fingerprint at the next rolling deploy; at this rate it is a recurring deploy-time nuisance, not an outage.
Where to look
Orleans.Runtime.MembershipService.MembershipTableManager.CheckIfLocalSiloIsDead/KillMyselfLocally(/_/src/Orleans.Runtime/MembershipService/MembershipTableManager.cs) — verify whether suspect votes older than the target's liveIAmAliveTimeare (or should be) discounted beforeKillMyselfLocally; both theRefreshandCleanupTableEntriescall sites hit this.- Peer silo logs for 10.244.17.58 and 10.244.5.66 around 08:29:10 GMT — what made both suspect 5j9gx at the same instant during a ~20-minute-old replica set.
- Membership/probe timeout configuration for
memex-portal-deployment— the suspicion window appears too tight relative to silos that are demonstrably still heartbeating. - Prior triage of the same defect: incident
Admin/_LogIncident/727f38e432ffd92f(Silo 7sqbk marked Dead in membership table while still alive — suspected-by-a-shutting-down-silo kill needs verification before KillMyselfLocally #3053) andAdmin/_LogIncident/34ce0ade2c08ca47(Silo in memex-portal-deployment marked Dead by peers while its heartbeat was still current, then terminated itself #3052).
Duplicate note: same component and symptom as #3053 / #3052 — this ticket documents a recurrence (new evidence, and a second call site: Refresh); a human may fold it into #3053 if that investigation is still open.
Evidence
| Fingerprint | 0803784e63467766 |
| Category | Orleans.Runtime.FatalErrorHandler |
| Severity | Error |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-85587f8869-5j9gx |
| Occurrences | 1 |
| First seen | 2026-10-10 08:30:03Z |
| Last seen | 2026-10-10 08:30:03Z |
| Routing | not determined — no configured route matches the category Orleans.Runtime.FatalErrorHandler. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-10-10 08:30:03Z memex-portal-deployment-85587f8869-5j9gx fail: Orleans.Runtime.FatalErrorHandler[100002]
Fatal error from Orleans.Runtime.MembershipService.MembershipTableManager. Context: I have been told I am dead, so this silo will stop! Reason: I should be Dead according to the membership table (in Refresh). Local entry: [SiloAddress=S10.244.6.128:11111:150624604 SiloName=Silo_b7daa Status=Dead HostName=memex-portal-deployment-85587f8869-5j9gx ProxyPort=30000 RoleName= UpdateZone=0 FaultZone=0 StartTime=2026-10-10 08:10:04.851 GMT IAmAliveTime=2026-10-10 08:29:36.513 GMT Suspecters=[S10.244.17.58:11111:150624528, S10.244.5.66:11111:150623330, S10.244.17.58:11111:150624528] SuspectTimes=[2026-10-10 08:29:10.921 GMT, 2026-10-10 08:29:10.921 GMT, 2026-10-10 08:29:10.921 GMT]].
FATAL EXCEPTION from Orleans.Runtime.MembershipService.MembershipTableManager. Context: I have been told I am dead, so this silo will stop! Reason: I should be Dead according to the membership table (in Refresh). Local entry: [SiloAddress=S10.244.6.128:11111:150624604 SiloName=Silo_b7daa Status=Dead HostName=memex-portal-deployment-85587f8869-5j9gx ProxyPort=30000 RoleName= UpdateZone=0 FaultZone=0 StartTime=2026-10-10 08:10:04.851 GMT IAmAliveTime=2026-10-10 08:29:36.513 GMT Suspecters=[S10.244.17.58:11111:150624528, S10.244.5.66:11111:150623330, S10.244.17.58:11111:150624528] SuspectTimes=[2026-10-10 08:29:10.921 GMT, 2026-10-10 08:29:10.921 GMT, 2026-10-10 08:29:10.921 GMT]].. Exception: null.\nCurrent stack: at System.Environment.get_StackTrace()
at Orleans.Runtime.FatalErrorHandler.OnFatalException(Object sender, String context, Exception exception) in /_/src/Orleans.Runtime/Core/FatalErrorHandler.cs:line 32
at Orleans.Runtime.MembershipService.MembershipTableManager.KillMyselfLocally(String reason) in /_/src/Orleans.Runtime/MembershipService/MembershipTableManager.cs:line 708
at Orleans.Runtime.MembershipService.MembershipTableManager.CheckIfLocalSiloIsDead(String caller) in /_/src/Orleans.Runtime/MembershipService/Membershi…[truncated]
Opened automatically from Admin/_LogIncident/0803784e63467766. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site dfcbe511c74e943b: other fingerprints of this site fold in here as comments rather than opening tickets of their own.
Activity
systemorph commentedon Oct 10, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
dfcbe511c74e943b(Orleans.Runtime.FatalErrorHandler), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.7d0e78f12afcb333dfcbe511c74e943bOrleans.Runtime.FatalErrorHandlerFatal error from Orleans.Runtime.MembershipService.MembershipTableManager. Context: I have been told I am dead, so this silo will stop! Reason: I should be Dead according to the membership table (in Refresh). Local entry: [SiloAddress=S1{n}.{n}:{n}:{n} SiloName=Silo_1{n}a4{n} Status=Dead HostName=memex-portal-deployment-{hex}-gvt4n ProxyPort={n} RoleName= UpdateZone={n} FaultZone={n} StartTime={time} GMT IAmAliveTime={time} GMT Suspecters=[S1{n}.{n}:{n}:{n}, S1{n}.{n}:{n}:{n}, S1{n}.{n}:{n}:{n}] SuspectTimes=[{time} GMT, {time} GMT, {time} GMT]].memex-cloudmemex-portal-deployment-8c87bbfc-gvt4nAdmin/_LogIncident/7d0e78f12afcb333What triage wrote about this fingerprint
What is failing
An Orleans silo in
memex-portal-deployment(Silo_15a49, 10.244.6.161:11111:150626359, podmemex-portal-deployment-8c87bbfc-gvt4n) found itselfDeadin the membership table during aRefreshpass and honoured the verdict by terminating itself (MembershipTableManager.CheckIfLocalSiloIsDead→KillMyselfLocally→FatalErrorHandler.OnFatalException). The entry showsIAmAliveTime=08:54:05.220againstSuspectTimes=08:54:02.455— the silo wrote a fresh heartbeat ~2.7 seconds after the votes that declared it dead, so it was alive and heartbeating when killed. This is a recurrence of the false-positive membership kill already filed as #3053, #6432 and #6396 — fourth firing of the same shape, second within 24 minutes this morning.Probable cause — moderate-to-high confidence, same as #3053/#6432
The suspicion votes were cast from a stale/shared liveness snapshot during rolling-deploy churn, not from a genuine crash:
SuspectTimesare identical to the millisecond (08:54:02.455), and one suspecter (10.244.3.116) appears twice with that same timestamp — one shared snapshot, not three independent probe failures (identical pattern to Silo 5j9gx marked Dead by peers 26s after its last heartbeat and terminated itself — recurrence of the false-positive membership kill (#3053) #6432).IAmAlivewrite postdates the votes — the membership adoption path inRefreshdoes not appear to discount suspect votes that are older than the target's liveIAmAliveTimebefore callingKillMyselfLocally.StartTimeis 08:39:19.943 — the replica was only ~15 minutes old, again consistent with a deploy window (the 08:30 firing of Silo 5j9gx marked Dead by peers 26s after its last heartbeat and terminated itself — recurrence of the false-positive membership kill (#3053) #6432 was on a ~20-minute-old pod).Exception: nulland theEnvironment.get_StackTrace()frame in the FatalErrorHandler stack confirm an intentional process stop, not an unhandled crash.The benign alternative — a GC/CPU stall on the pod making a genuine suspicion correct-but-harsh — cannot be excluded from this fingerprint alone; check the pod's own GC/resource logs around 08:54:02 before ruling.
Impact
Single occurrence, single pod; it self-terminated and was rescheduled by Kubernetes, and the cluster stayed quorate (three suspecting peers were live). But the family is escalating: 2026-09-02 (×2), 2026-10-09 20:44 (#6396), 2026-10-10 08:30 (#6432), and now 2026-10-10 08:54 — two healthy silos killed within 24 minutes, both on young pods during apparent roll-churn. Each false kill cascades grain reactivation and in-flight call failures, so a rolling deploy can lose healthy replicas faster than it gains them. Worth watching at the next deploy; still no outage evidenced.
Where to look
Orleans.Runtime.MembershipService.MembershipTableManager.CheckIfLocalSiloIsDead/KillMyselfLocally(/_/src/Orleans.Runtime/MembershipService/MembershipTableManager.cs) — verify whether suspect votes older than the target's liveIAmAliveTimeare (or should be) discounted before the kill; theRefreshandCleanupTableEntriescall sites both carry this exposure.Admin/_LogIncident/8138820952f88f8e(same pod, same 08:54:10 event, folds onto log siteAdmin/_LogIncident/site-3ce5feb1528fd58f→ Healthy Orleans silo falsely voted dead by peers and terminated itself during cluster roll-churn #6396).Admin/_LogIncident/727f38e432ffd92f(Silo 7sqbk marked Dead in membership table while still alive — suspected-by-a-shutting-down-silo kill needs verification before KillMyselfLocally #3053),Admin/_LogIncident/34ce0ade2c08ca47(Silo in memex-portal-deployment marked Dead by peers while its heartbeat was still current, then terminated itself #3052),Admin/_LogIncident/89dfb3b5f652b720(Healthy Orleans silo falsely voted dead by peers and terminated itself during cluster roll-churn #6396),Admin/_LogIncident/0803784e63467766(Silo 5j9gx marked Dead by peers 26s after its last heartbeat and terminated itself — recurrence of the false-positive membership kill (#3053) #6432).Duplicate note: same component and symptom as #3053 / #6432 / #6396 — this is the
FatalErrorHandler-category companion of this morning's second firing, filed so the recurrence rate (now twice in 24 minutes during deploy churn) is on record; a human may fold it into #6432 if that investigation is still open.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-8c87bbfc-gvt4nRecent log lines
rbuergi commentedon Oct 10, 2026
The killed silos were wedged inside the process from their first seconds, and three of three show the same boot fingerprint
I read both 2026-10-10 kills (5j9gx at 08:30:03, gvt4n at 08:54:10) and the 2026-10-09 kill on memex (hrdhq, #6395) with governed
Logsactions on the control instance. All reads were read-only. Each action records its LogQL, window and line count underOps/Actions/w11a-*.1. A boot fingerprint that only the doomed pods have
Every pod later voted dead had its first locally hosted grain call time out about 30 s after boot. Every healthy new pod in the same rolls did not.
85587f8869-5j9gxsys.client/hosted-10.244.6.128→messagehub/Agent/_Activity/import-manifeston the same silo (S10.244.6.128)8c87bbfc-gvt4n5f6b9fc9b-hrdhq85587f8869-prgnp,8c87bbfc-zjzll(booted in the window)Doc/_Activity/import-manifestwas created within 3 sw11a-cloud-import-manifestreads everydid not arrive on time … _Activity/import-line in memex-cloud from 08:05 to 08:56: 52 lines, all from 5j9gx and gvt4n.w11a-1009-import-manifestdoes the same for memex from 20:20 to 20:45: 85 of 85 from hrdhq.w11a-1007-import-manifestcovers memex-cloud on 2026-10-07 from 09:00 to 09:56, the Grain-call cancellation delivery times out against departing silos — cancellation batches not propagated during silo lifecycle events #6392 burst against10.244.2.27: 0 lines. That burst is a different shape, a departing silo.Doc/Architecture/ReadingASiloStopcovers it.On the doomed pods the first minute reads the same way each time.
GetMeshNode('Agent/_Activity/import-manifest') timed out after 10.0s … Target: NO LOCAL HUB … it never activated here. The platform-startupCreateNodereports "post-creation handlers had not finished within 30s". The static-repo import fails for Agent, Provider, Harness, Skill and Doc in turn.2. The process was not starved. Its threads were being consumed.
[LIVENESS]lines (w11a-5j9gx-liveness, negative controlw11a-prgnp-liveness):The heartbeat gap stays at exactly 10.00 s and GC is flat (gen0 +0 per tick for minutes). Neither a GC pause nor a thread-pool stall shows up, which matches the #6395 reading of
LocalSiloHealthMonitor. But the thread count climbs steadily from 20 to 55 while almost no work completes. That is threads blocking one at a time and never coming back: the pool injects a replacement each time queued work waits behind a blocked one. A healthy silo in the same roll runs on 16 threads at about 90 times the throughput.3. Why "alive but answering nothing" lasted 8 to 20 minutes before eviction
In Orleans 10.4.0,
SiloConnectionanswers a membership ping inside the connection's own receive loop (HandlePingMessage). It never touchesMessageCenter.ReceiveMessage, the catalog or the scheduler. Every other message goes throughMessageCenter.ReceiveMessage, which runs synchronously on that receive loop and callscatalog.GetOrCreateActivation, where the grain is constructed. So a silo whose activation or dispatch path blocks keeps passing probes on every connection whose loop has not yet hit a blocking message. It stops answering them only as more loops get stuck. That fits:ProbeTimeout15 s × 5 misses, a silo that failed probes from the start would be evicted in about 2 to 3 minutes;IAmAlivewrites to Postgres keep working;w11a-5j9gx-orleansholds everywarn|fail|crit: Orleansline in the pod's first 6 minutes: 9 lines, allCallbackDataresponse timeouts.This revises the #6395 reading. #6395 read the hrdhq evidence as "a broken outbound/response path from this one pod". The thread-count trend and the connection-level ping path point inside the process instead. The pods are also on different nodes: hrdhq on
vmss000028, which hosted the healthy prgnp on 10-10, and 5j9gx and gvt4n onvmss00002b. Neither silo IP appears in memex-cloud logs in the 4 h before its pod started (w11a-ip-reuse-6128b,w11a-ip-reuse-6161: 0 lines), so IP reuse is unlikely, though another namespace on the cluster was not read.The membership verdict was correct. The silo was not serving and had to go. Tolerance changes would only keep a wedged member in the table longer (see #6428 and the #6395 finding).
What I did NOT establish
KillMyselfLocally→FatalErrorHandler→Environment.FailFastwrites one under the chart'scrashDumps.root.Doc/Architecture/ReadingASiloEvictionrecords exactly that for the 2026-09-02 kill (signo 6). So dumps for 5j9gx (08:30:03Z) and gvt4n (08:54:10Z) should exist on memex-cloud's dump claim, withincrashDumps.keep/maxAgeDays. Reading them needs cluster access. There is no governedDumpInstanceAction, so this needs a maintainer:dotnet-dump analyze→clrstack -all,syncblk,dumpasync. Look for receive-loop threads (SiloConnection/MessageCenter.ReceiveMessage) or activation threads parked on one lock or oneWait.w11a, Claude Opus 5.5 (agent session). Read-only.
Merge pull request #6443 from Systemorph/feat/6432-analyze-dump-operator
systemorph commentedon Oct 10, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
dfcbe511c74e943b(Orleans.Runtime.FatalErrorHandler), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.072c056959395f0fdfcbe511c74e943bOrleans.Runtime.FatalErrorHandlerFatal error from Orleans.Runtime.MembershipService.MembershipTableManager. Context: I have been told I am dead, so this silo will stop! Reason: I should be Dead according to the membership table (in Refresh). Local entry: [SiloAddress=S1{n}.{n}:{n}:{n} SiloName=Silo_fbe7{n} Status=Dead HostName=memex-portal-deployment-{hex}-ltlwc ProxyPort={n} RoleName= UpdateZone={n} FaultZone={n} StartTime={time} GMT IAmAliveTime={time} GMT Suspecters=[S1{n}.{n}:{n}:{n}, S1{n}.{n}:{n}:{n}, S1{n}.{n}:{n}:{n}] SuspectTimes=[{time} GMT, {time} GMT, {time} GMT]].memex-cloudmemex-portal-deployment-5499f76b76-ltlwcAdmin/_LogIncident/072c056959395f0fWhat triage wrote about this fingerprint
What is failing
A live Orleans silo in the
memex-portal-deploymentcluster (podmemex-portal-deployment-5499f76b76-ltlwc, siloSilo_fbe75, started 18:10:16 GMT) was votedDeadby two peer silos (3 votes, all at 18:16:16) and honored the verdict by self-terminating viaMembershipTableManager.KillMyselfLocally→FatalErrorHandler. The silo's lastIAmAliveheartbeat (18:15:51) is only 25 seconds behind the votes, and the pod had been running normally for six minutes — a false-positive death declaration, not a genuine crash.Probable cause — medium confidence
Same defect as GitHub issue #6396 (see prior incidents
Admin/_LogIncident/89dfb3b5f652b720andAdmin/_LogIncident/8138820952f88f8e, both filed against it): the peer suspicion protocol (2 distinct silo votes) fired because the target missed direct probes while its table heartbeats continued — the usual signature of transient pod stall (CPU/threadpool starvation, GC pause) or node-level network hiccup. The wider cluster was in a heavy timeout/churn burst in the same window (see the concurrent TimeoutException/RoutingGrain incidents at 18:15–18:17), so load-induced probe misses are the likely trigger. Precise per-event stall cause needs pod resource metrics.Impact
This is now the third false death today (08:30 GMT
ffab13afba632f6c, 08:54 GMT8138820952f88f8e, 18:16 GMT this one) and at least the fifth overall — the recurrence has gone from roughly monthly to three in ten hours on the primary portal deployment. Each event kills a healthy pod, fails in-flight grain calls transiently, and triggers a Kubernetes restart, amplifying the very churn that causes the next kill. Intermittent user-visible failure on a primary path, trending toward continuous flapping.Where to look
Orleans.Runtime.MembershipService.MembershipTableManager—CheckIfLocalSiloIsDead/KillMyselfLocally(MembershipTableManager.cs, ~line 708) and the peer-sideClusterHealthMonitorvote path on the suspecting silos.…5499f76b76-ltlwcand its two suspecters around 18:16 GMT.ClusterMembershipOptionsprobe-timeout /NumVotesForDeathDeclarationtuning formemex-portal-deployment— with five occurrences, that is the better first move than chasing per-event stalls.Duplicate note: same defect as #6396 — file this; a human can fold it in seconds, or use the accelerating recurrence (3 in one day) to bump #6396's priority. Companion log site for the same event:
Admin/_LogIncident/282031c4ad913223.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-5499f76b76-ltlwcRecent log lines
docs(silo-eviction): one ping mechanism, the options that were deploy…
docs(silo-eviction): a self-kill may leave no dump; IAmAlive is not e…
docs(crash-dumps): the arming verdict is the armed/HELD/ERROR line, n…
docs(crash-dumps): the verdict is the last line of the script's run; …
Merge pull request #6439 from Systemorph/docs/6432-silo-wedged-at-boot
systemorph commentedon Oct 11, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
dfcbe511c74e943b(Orleans.Runtime.FatalErrorHandler), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.a6563ca005d514eddfcbe511c74e943bOrleans.Runtime.FatalErrorHandlerFatal error from Orleans.Runtime.MembershipService.MembershipTableManager. Context: I have been told I am dead, so this silo will stop! Reason: I should be Dead according to the membership table (in Refresh). Local entry: [SiloAddress=S1{n}.{n}:{n}:{n} SiloName=Silo_5{n} Status=Dead HostName=memex-portal-deployment-{hex}-hdlgz ProxyPort={n} RoleName= UpdateZone={n} FaultZone={n} StartTime={time} GMT IAmAliveTime={time} GMT Suspecters=[S1{n}.{n}:{n}:{n}, S1{n}.{n}:{n}:{n}, S1{n}.{n}:{n}:{n}] SuspectTimes=[{time} GMT, {time} GMT, {time} GMT]].memexmemex-portal-deployment-6749fbfc59-hdlgzAdmin/_LogIncident/a6563ca005d514edWhat triage wrote about this fingerprint
What is failing
An Orleans silo in the memex portal deployment (
Silo_56225, podmemex-portal-deployment-6749fbfc59-hdlgz) was marked Dead in the membership table by three suspicion votes from sibling silos, discovered its own dead status during the periodic membershipRefresh, and terminated itself viaKillMyselfLocally— the standard Orleans self-kill path, working as designed once the table said Dead. The defect is upstream of that: the silo was voted dead while it was demonstrably alive and heartbeating.Probable cause — moderate confidence
The log's own numbers make a genuine silo hang unlikely: the local entry's
IAmAliveTimeis 04:13:45.634 GMT, i.e. the silo successfully refreshed its heartbeat ~1.5 s after the threeSuspectTimesat 04:13:44.121 GMT. So the suspicion votes raced ahead of the silo's liveIAmAlivetable writes rather than following a missed-heartbeat timeout. This points at a membership-freshness problem between the suspectors and the membership table: either (a) the membership-table backend briefly served stale data / latency spikes so suspectors saw an oldIAmAliveTimeand voted, or (b) the suspicion was raised from a transient network partition that healed too late for the votes to be retracted. Note also that one suspector (S10.244.6.197:11111:150693988) appears twice in the Suspecters list — the same silo suspecting twice in one entry suggests the suspect table wasn't cleaned between rounds, which is consistent with view-staleness. I cannot confirm which of the two from a single log line; the silo had only been up ~21 min (StartTime 03:53:18 GMT) when it died, which also fits a startup-window race. Cause unclear beyond that; evidence points at the suspect/IAmAlive freshness path, not at the dead silo itself.Impact
One occurrence, one pod, at 04:14:15 UTC on 2026-10-11. The pod terminated and the cluster's usual silo-death handling applies ( Orleans reassigns; K8s restarts the pod), so impact was at most a brief availability dip on that silo. If this repeats across pods it is an intermittent user-visible failure and should be re-rated.
Where to look
Orleans.Runtime.MembershipService.MembershipTableManager—Refresh/CheckIfLocalSiloIsDead(the self-kill site,/_/src/Orleans.Runtime/MembershipService/MembershipTableManager.csin the Orleans repo) and the suspect-table cleanup path.Orleans.Runtime.MembershipService.SiloHealthMonitor/ the suspector side: compare each suspector's cachedIAmAliveTimefor the dead silo at 04:13:44 against what the table held at that time.ClusterMembershipOptions(NumMissedIAmAliveLimit,IAmAliveTableRefreshTimeout,DeathNotificationExpiration) and the membership-table backend's latency/error logs around 04:13–04:14 UTC.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-6749fbfc59-hdlgzRecent log lines
rbuergi commentedon Oct 11, 2026
The dump has been read: one container lock held for the life of the process, and Orleans' own probe path queued behind it
Ops/Actions/analyzedump-memex-cloud-20261011-ltlwc-2on the control instance completed at 08:02Z (page:Ops/Dumps/analyzedump-memex-cloud-20261011-ltlwc-2). It read the FailFast dump of5499f76b76-ltlwc(evicted 2026-10-10 18:16Z): 91 managed threads, thread pool 35 workers / 30 running / CPU 0 %. Read-only.What it establishes
Two monitors are held, and no other (
syncblk):0x1e(managed 13, one of the first pool workers)0xf1(managed 71)Every waiting thread shown is in the same frame:
Monitor.EnterunderAutofac.Core.Lifetime.LifetimeScope.CreateSharedInstance. That is the lock a container scope holds while it constructs one of its single-instance services. It is one lock per scope, not one per service. Index 2 is aSystem.Objectallocated in the process's first moments, so it is the root container's.The owner of fix(hosting): idle-release for read streams — stop the sync-stream heartbeat leak #169 is itself one of those waiters. Thread
0xf1is inside a constructor-injected resolve (ReflectionActivator→AutowiringParameter→SharingMiddleware→CreateSharedInstance→Monitor.Enter): it holds one scope's lock while it constructs a service, and waits for another scope's lock for that service's constructor argument.The waiters are the whole process, including Orleans' message path:
BuildPostCreationHandlers(the stalled creates of CreateNode lifecycle stalls at the validators stage with no verdict — a validator never answers within its window during the platform-startup boot-activity create #6391), twoMessageHubGrain.ResolveHubConfigurationObservable, aMeshNodeStreamHandle.UpdateRemote, theMeshNodeStreamCachedelivery-failure handler,ProviderCredentialSeed.RunOver, ASP.NET'sDataSourceDependentMatcher.CreateMatcher;Orleans.Runtime.Messaging.MessageSerializer.Write/ReadBodyObject→CodecProvider.TryCreateCodec→ActivatorUtilities→ the same lock, forPooledResponseCodec<IndirectProbeResponse>, forIGrainCallCancellationManagerSystemTargetand forIDhtGrainDirectory.A silo that cannot construct the codec for a probe response cannot answer the probe. That is why its peers' probes timed out with no connection error, why
sys.svc.cancelercalls to it went unanswered (Grain-call cancellation delivery times out against departing silos — cancellation batches not propagated during silo lifecycle events #6392), and whyIAmAlivekept being written: that path had been constructed before the lock was taken.So the question in this issue's title has an answer. The silo was not healthy and the verdict was correct: its root container's single-instance lock was held for the whole life of the process, and everything not resolved before that moment waited on it. No probe timeout, vote count or table clock is involved, and none of them should change.
What it does NOT establish, and why
What thread
0x1ewas doing. Its stack is not in the read. The condensation kept "the 40 largest" of 86 groups, but nothing grouped (every frame carried its own stack address, so 91 threads made 86 stacks), groups of one were kept newest thread first, and the thread that took the lock at boot is the oldest. So the one stack that explains the lock was among the 46 "cut to fit". Each kept stack also showed only its top 25 frames, which for a waiter is the container; the caller is at the bottom.Two readings fit what is shown, and the owner's stack separates them:
0x1ewaits on monitor fix(hosting): idle-release for read streams — stop the sync-stream heartbeat leak #169: two scopes locked in opposite orders by two threads. Since each module's services live in a container of their own (core feat(modules): root services and the builder hook become live-updatable; only boot-time infra may restart (slices 4+5) #6127, 2026-10-05/06; the first incident of this shape is 2026-10-09), both orders exist by design: a module service under construction takes a root service as a constructor argument (module lock, then root lock), and a root service under construction can ask for a module's service (root lock, then module lock).0x1ewaits on something else while it holds Test #2 (a blocking call inside a single-instance factory or constructor).I did not pick one and did not change product code on a guess.
What I changed
PR #6484 repairs the read (
deploy/aks/operator/bin/hosting-dump-analyze): a threadsyncblknames as a monitor's owner is always shown, first, with its whole stack and a first line naming the monitor and its waiters; frames are compared without their own address and source path, so parked threads group; a capped stack keeps its top and its bottom frames. The operator test pins it with this incident's shape (an old owner thread with a deep stack, 30 waiters, 300 unrelated threads) and a negative control (no owner named insyncblk→ the same thread is cut again).Next reading (one action, read-only)
The lane takes the operator scripts from core
main, so once that PR is merged, on the control instance:{ "requestedAction": "AnalyzeDump", "dumpFile": "memex-portal-deployment-5499f76b76-ltlwc.1791656185/coredump.1.1791656179" }against
Deployments/memex-cloud. The first group of the result is thread0x1e's stack: its top frame says what it waits in, its bottom frames say which service it was constructing. The two 2026-10-10 morning dumps (8c87bbfc-gvt4n.1791622458/coredump.1.1791622453,85587f8869-5j9gx.1791621012/coredump.1.1791621006) were still on the volume at 08:02Z and would say whether it is the same pair every time. They are subject to the chart's dump retention, so read them soon.The reading is conserved in
Doc/Architecture/ReadingASiloEviction("What the first dump said") in the same PR.Same cause, confirmed from the bodies and comments: #6395 (memex, 2026-10-09: same boot fingerprint, 85 of 85 lines on the evicted pod) is this defect; I am closing it as a duplicate of this issue. #6391 (the stalled creates) and #6392 (unanswered cancellation calls) are this lock seen from two of its waiters. Not the same cause: #5300, #2906 and #5336, which predate module containers; their readings are on those issues.
w21, Claude Opus 5.5 (agent session).
rbuergi commentedon Oct 11, 2026
The dump is read: the wedged silo is a convoy on the DI container's shared-instance lock
Source:
Ops/Actions/analyzedump-memex-cloud-20261011-ltlwc-2on the control instance (governedAnalyzeDump, dump of5499f76b76-ltlwcwritten 2026-10-10 18:16:20Z, the third kill of that day). I read it while working the symptoms that fold into this issue (#6429, #6392, #5704, #5731).What the dump shows
syncblklists two held monitors:System.Object),MonitorHeld 43: one owner and 21 waiters. Owner is OS thread0x1e(managed id 13, a thread-pool worker).System.Object),MonitorHeld 9: one owner and 4 waiters. Owner is OS thread0xf1.0xf1owns index 169 and is itself waiting inMonitor.EnterunderAutofac.Core.Lifetime.LifetimeScope.CreateSharedInstance, reached from a constructor-parameter resolve (AutowiringParameter→BoundConstructor.Instantiate). Read together: a shared instance is mid-construction on that thread, holding one scope's lock while it waits for index 2. The frame that took index 169 is below the 25-frame cap, so that reading is an inference.Monitor.Enter_Slowpathcalled fromAutofac.Core.Lifetime.LifetimeScope.CreateSharedInstance. That is the lock Autofac takes to create a shared (singleton or per-scope) instance. Who is asking:OrleansGeneratedCodeHelper.GetServicefromCodecProvider.TryCreateCodec, fromMessageSerializer.WriteinConnection.ProcessOutgoing. The codec is for anIGrainCallCancellationManagerSystemTargetinvokable.MeshExtensions.BuildPostCreationHandlers(GetRequiredService)MessageHubGrain.ResolveHubConfigurationObservable(GetService), insideOnActivateAsyncMeshNodeStreamHandle.UpdateRemote(GetService)MeshNodeStreamCacheconstructor's handler (GetService)ProviderCredentialSeed.RunOver(GetService), fromStaticRepoImportSettled.MarkSettledWhat that explains
Connection.ProcessOutgoingthreads are connections that cannot WRITE a message, because the first cancellation message they had to serialize needs that lock.GrainCallCancellationManagerbatches from the wedged pod are those six outgoing loops. The batches sent TO it time out because it dispatches nothing.BuildPostCreationHandlersand the grain activation's configuration resolve are behind the same lock, which is why the firstCreateNodereports "post-creation handlers had not finished" and_Activity/import-manifestnever activates.What I did NOT establish
0x1eis doing. It owns the lock everyone waits for, and it is not among the stacks in the report: the analyser prints 40 of 86 stack groups, capped at 25 frames. Its stack is the answer to this issue. It is a singleton factory or constructor that does not return, and since the process never recovers, it is most likely waiting on something that itself needs the container (a blocking wait on work that resolves a service on another thread).0xf1waiting for it while holding another scope's lock, and the fact that process-wide callers all queue on it point that way, but the report does not name the object's owner.5j9gx08:30:07Z,gvt4n08:54:13Z) show the same owner frame. The listing shows both are there.Next step, one read:
clrstackfor OS thread0x1ein this dump (setthreadto managed id 13 first), and the same question of the other two dumps.AnalyzeDumpneeds either a per-thread option or the lock owners' stacks printed whatever the group cap, since the owner of a contended monitor is the one stack that always matters.Readings taken today for the symptom issues, each a governed
Logsaction on the control instance, 72 h windows ending 2026-10-11 11:15Z:w21b-churn-{cloud,memex}-dead-72h)GrainCallCancellationManagererrors (…-cancel-72h)Message expired before placement(…-placement-72h)ltlwc's life (started 18:10:16Z)…-attach-72h)5j9gxwave 21, Claude Opus 5.5 (agent session). Read-only on the deployment.
rbuergi commentedon Oct 11, 2026
The lock owner's stack is read. Fix: #6536 (auto-merge armed)
Source:
Ops/Actions/analyzedump-memex-cloud-20261011-ltlwc-3on the control instance. It is the same dump (5499f76b76-ltlwc, 2026-10-10 18:16Z), read by an operator at core9bcc344d46, which includes #6484. Read-only. The two owners are printed first and whole (124 and 160 frames).0x1eHost.StartAsync→ApplicationLifetime.NotifyStarted→MeshWeaver.WhatsApp.WhatsAppReplySender.Begin()→GetRequiredServiceon a scope of the WhatsApp module's own container →ModuleFallbackSourcestand-in → rootCreateSharedInstance(takes #2) →PortalApplication..ctor→GetRequiredService→CreateSharedInstance(waits)0xf1ActivationData→MessageHubGrain.DeliverMessage→OnActivateAsync→ResolveHubConfigurationObservable→meshHub.ServiceProvider.GetService<IMeshNodeHubFactory>→CreateSharedInstance(takes #169) →AutowiringParameter→CreateSharedInstance(waits)PortalApplication's constructor callshub.ServiceProvider.GetRequiredService<ILayoutClient>(), andhub.ServiceProvideris the mesh hub's scope. So #169 is the mesh hub scope's lock. The inversion is between the root and the mesh hub's scope, not a module container.Root cause.
PortalApplicationis a SCOPED registration. Since #6127 a module's services live in a container of their own, and its fallback resolved every root registration withrootScope.ResolveComponent(...), which is the root scope itself. A scoped service asked for from a module scope was therefore built in the root, under the root's lock, and shared there for the life of the process. Its constructor then reaches into a child scope, the opposite order to the grain activation. Before #6127,rootServices.CreateScope()in the reply sender was a root child scope, and the same call built it under that child scope's lock. That is why the first wedge of this shape came two days after #6127.Fix (#6536). The fallback now resolves each root registration in a root child scope owned by the asking module scope and disposed with it. A singleton is still the root's one instance. A scoped service belongs to that scope and is never built under the root lock.
AScopedRootServiceIsNeverBuiltInTheRootTestforces this dump's two threads to overlap with real containers:The three GC-collection swap tests stay green in Release.
Not this incident, recorded: a root singleton whose construction calls a module's service, racing a module singleton that needs a root singleton, deadlocks the same way. I reproduced it with real containers. No dump shows it. It is noted on
Doc/Architecture/ReadingASiloEviction.Verification after it rolls: no
silo told it is deadkill and nopost-creation-handlers: resolvingline on memex / memex-cloud after a boot on the new image. The 2026-10-10 morning dumps (gvt4n,5j9gx) were not read. Read them if the shape recurs on the new image.Claude Opus 5.5 (agent session). Read-only on the deployment.
fix(modules): a module scope resolves root services in a root child s…
systemorph commentedon Oct 11, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
dfcbe511c74e943b(Orleans.Runtime.FatalErrorHandler), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.f839a2b034e6f3c6dfcbe511c74e943bOrleans.Runtime.FatalErrorHandlerFatal error from Orleans.Runtime.MembershipService.MembershipTableManager. Context: I have been told I am dead, so this silo will stop! Reason: I should be Dead according to the membership table (in Refresh). Local entry: [SiloAddress=S1{n}.{n}:{n}:{n} SiloName=Silo_8e2b1 Status=Dead HostName=memex-portal-deployment-{hex}-{n}wxb ProxyPort={n} RoleName= UpdateZone={n} FaultZone={n} StartTime={time} GMT IAmAliveTime={time} GMT Suspecters=[S1{n}.{n}:{n}:{n}, S1{n}.{n}:{n}:{n}, S1{n}.{n}:{n}:{n}] SuspectTimes=[{time} GMT, {time} GMT, {time} GMT]].memex-cloudmemex-portal-deployment-5b769df8cb-89wxbAdmin/_LogIncident/f839a2b034e6f3c6What triage wrote about this fingerprint
Observed
An Orleans silo in the
memex-cloudportal deployment terminated itself viaOrleans.Runtime.MembershipService.MembershipTableManager.CheckIfLocalSiloIsDead("Refresh")→KillMyselfLocally→FatalErrorHandler.OnFatalException(path from the logged stack:/_/src/Orleans.Runtime/MembershipService/MembershipTableManager.cs:line 708,/_/src/Orleans.Runtime/Core/FatalErrorHandler.cs:line 32). The membership table listed the silo asStatus=Dead, soMembershipTableManagerstopped the silo cleanly: "I have been told I am dead, so this silo will stop!".The suspicious detail is the ordering of the timestamps in the same log line:
Silo_8e2b1(SiloAddress=S10.244.3.4:11111:150760535, podmemex-portal-deployment-5b769df8cb-89wxb), started 2026-10-11 21:55:35 GMT.S10.244.5.214:11111:150760299(twice) andS10.244.2.125:11111:150760414.IAmAliveTimeis 2026-10-11 22:13:45.040 GMT — about 9 seconds after every suspicion vote.So while the peers had already voted this silo dead, the silo was still alive and successfully writing its heartbeat to the membership table. It then discovered its own death in the next membership Refresh (22:14:12) and killed itself.
Where and when
One occurrence, one pod, namespace
memex-cloud, at 2026-10-11 22:14:12 UTC — roughly 19 minutes after the silo started. Whether the pod was restarted by the platform afterwards, and whether the replacement silo joined cleanly, is not visible from this incident.Impact
One silo was terminated mid-flight; any grains (and in-flight requests) hosted on it were lost to their failover. Given a single occurrence on a single pod, this reads as a brief, self-recovering disruption rather than an outage — but the evidence pattern (a live, heartbeating silo voted dead) indicates the cluster's failure detection disagreed with the silo's own liveness, and each repetition costs a pod restart.
Not established
Candidate causes and the reading that separates them
S10.244.3.4:11111to probe it, while the silo could still writeIAmAliveto the membership table. Predicts probe-timeout or connection-failure warnings in the logs ofS10.244.5.214andS10.244.2.125at ~22:13:35 GMT. Reading the two suspecting pods' logs around that instant would confirm or refute this directly.Silo_8e2b1(who wrote each status transition, and when) would show whether the three votes were independent or derived.Where to look
Orleans.Runtime.MembershipService.MembershipTableManager—CheckIfLocalSiloIsDead(called from the membership Refresh) andKillMyselfLocally(/_/src/Orleans.Runtime/MembershipService/MembershipTableManager.cs:line 708), andOrleans.Runtime.FatalErrorHandler.OnFatalException(/_/src/Orleans.Runtime/Core/FatalErrorHandler.cs:line 32), all as named in the logged stack.Silo_8e2b1around 22:13:35–22:14:12 GMT, and the logs of the suspecting pods (S10.244.5.214:11111:150760299,S10.244.2.125:11111:150760414) for their side of the same window.Severity is my judgement from a single occurrence with a self-recovering blast radius (one pod, one restart); a human should re-rule it if this fingerprint recurs or if the suspecting pods show a wider probe-failure pattern.
Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-5b769df8cb-89wxbRecent log lines
rbuergi commentedon Oct 13, 2026
Closing: the dump's root cause is fixed and deployed, and the kill has not recurred since.
CreateSharedInstance, it answered pings and nothing else, so its peers voted it dead.f6a7aeaa0c: a root registration asked for from a module scope is built in a root child scope, never under the root lock. fix(modules): build module singletons before they are published (#6542) #6545, merge89c7c04b63: module singletons are prebuilt.ec7763883b(/api/version), and both commits are in it. On memex,Ops/Status/memexshows all replicas on generation…-679d7b6984, started 2026-10-13 00:23Z to 00:29Z.…-5b769df8cb-89wxb, three minutes after fix(modules): a module scope resolves root services in a root child scope, never in the root (#6432) #6536 merged and before any image could carry it. No occurrence has folded here since, which is about 41 h.#6463 (diagnostics only: a cancelled grain call NACKed in Orleans' own words) stays a draft, separate from this. If the membership kill folds in again on a pod at or after
ec7763883b, this reopens with that pod's dump.