Repository navigation
Pod-hub directed delivery fails at placement with no classified fallback when no active silo carries IPodHubGrain #3984
Description
What is failing
The directed pod-hub delivery leg (IPodHubGrain, the transport that replaced the stream publish in the pod-hub roll plan) fails at placement: Orleans' PlacementService reports no active node in the cluster carries the IPodHubGrain grain type at all (Known nodes with grain type: none). The bare Orleans.Runtime.OrleansException then escapes through the Polly retry policy (attempt 3, handled) instead of being classified.
Probable cause — moderate confidence
This is not the known directory-instability shape (#2357: "…not stable to perform the lookup… Retry later"). Here placement says the grain type is unknown on every silo, which means the call was made during a window when no active silo advertised IPodHubGrain — i.e. cluster membership churn while pods were starting/restarting (both pods in this deployment restarted within the 65-second burst window, consistent with a rolling deploy overlap or a readiness-probe-gated restart, cf. Pod-Hub Delivery Roll Plan). Two candidate defects, both consistent with the evidence:
- The router attempts a directed
IPodHubGraincall before any silo in the current membership has attached/registered the grain type, andRoutingGrain's classification does not treat this placement rejection as transient — so the retry-with-fresh-resolve that already exists is never reached (same inert-classifier shape as The routing layer discarded every classification below it (#2346) #2451 / the Grain calls are dropped instead of retried when the Orleans directory is unstable during a rollout #2357 cure). - Alternatively a split-version window: release-N pods (which never attach a pod-hub grain) are the only active members while a caller routes via the directed leg.
Impact
Small burst: 5 occurrences over ~65 seconds on 2 pods of one deployment, Polly-handled. Per the roll plan this window is expected to fall back to the stream publish — if deliveries were actually lost rather than fallen back, that is the #1742 silence class again; if they fell back, the remaining defect is that the failure is logged as fail and retried into a dead end rather than classified. Nuisance now, but it is the exact overlap window a surge-first rolling deploy produces fleet-wide.
Where to look
- RoutingGrain / OrleansRoutingService — the directed-call branch,
IsTransientFailure/ClassifyDeliveryException: extend the classification to treatOrleansExceptionfromPlacementService("No active nodes are compatible…") as transient and fall back to the stream publish, as Grain calls are dropped instead of retried when the Orleans directory is unstable during a rollout #2357's cure did for the directory-instability arm. PlacementService.GetCompatibleSilos(Orleans.Runtime,/_/src/Orleans.Runtime/Placement/PlacementService.cs:221) viaPreferLocalPlacementDirector.OnAddActivation— the throwing frame.
Possibly related to #2357 / #1742 (same component, same window, different exception shape). Not an exact duplicate — the fingerprint text differs from the directory-instability message that #2357 collected.
Evidence
| Fingerprint | 06ca1493d35ec65a |
| Category | Polly |
| Severity | Error |
| Exception | Orleans.Runtime.OrleansException |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-5867566569-k9nmf, memex-portal-deployment-5867566569-frfqx |
| Occurrences | 5 |
| First seen | 2026-09-10 21:01:27Z |
| Last seen | 2026-09-10 21:02:32Z |
Recent log lines
2026-09-10 21:01:27Z memex-portal-deployment-5867566569-k9nmf fail: Polly[3]
Execution attempt. Source: 'Orleans.Placement/(null)/Retry', Operation Key: '', Result: 'No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none', Handled: 'True', Attempt: '3', Execution Time: 0.1381ms
Orleans.Runtime.OrleansException: No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 221
at Orleans.Runtime.Placement.PreferLocalPlacementDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/PreferLocalPlacementDirector.cs:line 21
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 612
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
2026-09-10 21:01:27Z memex-portal-deployment-5867566569-k9nmf fail: Polly[3]
Execution attempt. Source: 'Orleans.Placement/(null)/Retry', Operation Key: '', Result: 'No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none', Handled: 'True', Attempt: '3', Execution Time: 0.146ms
Orleans.Runtime.OrleansException: No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 221
at Orleans.Runtime.Placement.PreferLocalPlacementDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/PreferLocalPlacementDirector.cs:line 21
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 612
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
2026-09-10 21:01:28Z memex-portal-deployment-5867566569-k9nmf fail: Polly[3]
Execution attempt. Source: 'Orleans.Placement/(null)/Retry', Operation Key: '', Result: 'No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none', Handled: 'True', Attempt: '3', Execution Time: 0.1363ms
Orleans.Runtime.OrleansException: No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 221
at Orleans.Runtime.Placement.PreferLocalPlacementDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/PreferLocalPlacementDirector.cs:line 21
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 612
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
2026-09-10 21:02:32Z memex-portal-deployment-5867566569-frfqx fail: Polly[3]
Execution attempt. Source: 'Orleans.Placement/(null)/Retry', Operation Key: '', Result: 'No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none', Handled: 'True', Attempt: '3', Execution Time: 0.1622ms
Orleans.Runtime.OrleansException: No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 221
at Orleans.Runtime.Placement.PreferLocalPlacementDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/PreferLocalPlacementDirector.cs:line 21
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 612
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
2026-09-10 21:02:32Z memex-portal-deployment-5867566569-frfqx fail: Polly[3]
Execution attempt. Source: 'Orleans.Placement/(null)/Retry', Operation Key: '', Result: 'No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none', Handled: 'True', Attempt: '3', Execution Time: 0.1422ms
Orleans.Runtime.OrleansException: No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 221
at Orleans.Runtime.Placement.PreferLocalPlacementDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/PreferLocalPlacementDirector.cs:line 21
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 612
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
Opened automatically from Admin/_LogIncident/06ca1493d35ec65a. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 11, 2026
Implemented in PR #3988 and shipped to production as 3.0.0-ci.8323. Production verification on memex-cloud: the Kubernetes rollout completed successfully; a live audit recorded the running image as meshweaver.azurecr.io/memex-portal-ai:3.0.0-ci.8323; the public health endpoint returned HTTP 200; and a 60-minute post-rollout log query for fingerprint 06ca1493d35ec65a plus the exact pod-hub placement failure returned 0 lines. Closing as completed.
systemorph commentedon Sep 17, 2026
Reopened — this fault is happening again.
Still occurring: 26 more since the last update (31 total).
memex-portal-deployment-5867566569-k9nmf,memex-portal-deployment-5867566569-frfqx,memex-portal-deployment-5867566569-c9xbt,memex-portal-deployment-5867566569-z4zzk,memex-portal-deployment-7f74766b8d-rltnt,memex-portal-deployment-7f74766b8d-rmjfc,memex-portal-deployment-67f7fc74f6-hxmct,memex-portal-deployment-67f7fc74f6-cqwr5,memex-portal-deployment-67f7fc74f6-cvvnq,memex-portal-deployment-574d54c98d-cdd47,memex-portal-deployment-574d54c98d-s8g9b,memex-portal-deployment-574d54c98d-vvcwqRecent log lines
rbuergi commentedon Sep 17, 2026
Re-closed — the reopen above was on stale evidence (#4577).
The recurrence bot reopened this at
2026-09-17T07:53:12Zunder the headline "this fault is happening again", but the occurrence its own comment cites is OLDER than the close it reverted:Last seen:2026-09-11 03:48:35Z2026-09-11T05:24:11ZSo the counter moved, not the clock — the fault's newest occurrence predates the close, which is the one thing "still happening" has to mean. Root cause and fix:
LogIncidentFiler.Reopenreopened on the sole condition that the issue was closed, while the comment-debt clock pays a final count for incidents whose fault has stopped (the fix, Systemorph/MeshWeaver.Plugins#2014, is written but NOT yet merged — it makes a reopen requireincident.LastSeen > issue.ClosedAt).This restores the close exactly as it stood; nothing is asserted about the defect itself. If the fault happens again, the next occurrence postdates this close and reopens the issue on evidence that means what it says.
One of 53 issues whose reopen was measured stale (129 bot reopens on 2026-09-17, 125 of them inside the 07:51-07:53Z sweep; 76 postdate their close and were left open). Measured and re-closed 2026-09-17 by Claude Opus 5.