Repository navigation
Sustained ROUTE storm: ~36 failed deliveries/s to three never-activated node hubs, with the DeliveryFailure itself undeliverable #2546
Description
Observation, not yet a diagnosis
Measured on memex.meshweaver.cloud (namespace memex-cloud, memex-portal-ai:3.0.0-rc8.ci.6145, 3 replicas) on 2026-08-27 ~20:33 UTC.
One replica is logging 64 627 fail: MeshWeaver.Hosting.Orleans.RoutingGrain in 30 minutes — roughly 36 per second, sustained, with no user action driving it.
[ROUTE] Stream-routed delivery to 'node/eapdieia6urs9qd0hbbej' has no live subscriber:
no silo in this cluster is currently serving that hub, so the message could not be
delivered. Message RawJson (…) was NOT posted — a publish to a subscriber-less stream
succeeds and discards, which is why this had to be checked rather than observed.
Surfacing DeliveryFailure to the sender.
Over a ten-minute window it is three specific addresses, retried without end:
| address | occurrences / 10 min |
|---|---|
node/eapdieia6urs9qd0hbbej |
1 672 |
node/ea2a5ed4cb664db7884f609f5bdbe348 |
1 672 |
node/d373128c6ac34177a86c1e1137e82047 |
1 624 |
So: something keeps publishing to per-node hubs that no silo activates, the delivery fails, and the publisher retries at ~5.5/s per address indefinitely. Note the first id is not a GUID shape like the other two, which may be a clue about the producer.
The sibling portal (namespace memex, image ci.6105) shows the same defect class with a different victim — there it is two disposed Blazor circuits (portal/uNF5th…, portal/Q2ERYE…), 7 128 + 7 128 failures in 13 minutes, from one user, each accompanied by:
[ROUTE] Cannot deliver the NotFound DeliveryFailure for … to sender …: no pod-hub activation
claims that address AND its stream has no live subscriber, and a NACK has no NACK of
its own. The sender will wait out its own request budget …
That second line is the interesting half: the failure notification cannot be delivered either, so the sender never learns and never stops. This is the classic no-back-pressure storm shape from the /storm skill — a retry whose failure signal is itself undeliverable.
Why it matters
It is invisible from the UI, so nothing surfaces it, but a replica spending 36 failed deliveries a second is burning CPU and log ingest that real work competes for. It was found while sizing perceived latency for the SAV talk demo flow (Systemorph/MeshWeaver.SocialMedia#81).
What I did NOT establish
- Who the publisher is, and whether it is one producer or several.
- Why those three node hubs never activate (deleted nodes? a stale subscription to a recycled address? a satellite path with no owner?).
- Whether the retry has any bound at all, or whether the producer simply re-subscribes on each failure.
Filed so it is not lost. Reproduction should start from the producer side: find what posts RawJson to node/<id> and what it does with a DeliveryFailure it cannot receive.
Activity
Merge pull request #2620 from Systemorph/fix/2426-dead-target-storms
rbuergi commentedon Aug 28, 2026
Fixed by #2620 — same root as #2426, different victim.
Answering the three things this issue left open
DataChangedEvents out to subscribers. The "three never-activated node hubs" are not per-node hubs at all:node/…is the gRPC Bun/Node participant address type (GrpcHostingExtensions.NodeAddressType, registered as stream-routed byAddGrpcHub), which is why the line reads "Stream-routed delivery to 'node/…'".node/eapdieia6urs9qd0hbbej(nanoid-shaped) and the two GUID-shaped ids are three worker/client connections that had subscribed to nodes and then disconnected — theGrpcConnectionRegistrydisposes the participant hub and its route on disconnect, but nothing told the owners._Activity/log-heavy nodes). The owner never learned because its NACK — the "cannot deliver the DeliveryFailure" line you spotted as the interesting half — was published to a stream a per-node grain hub never subscribes.What changed
The refusal now carries
DeliveryFailure.TargetUnserved, the NACK is delivered to a node-backed sender over the grain transport, and the owner evicts that subscriber's server-side streams (Workspace.EvictClientSubscriptions). One refusal per (owner, dead participant), then silence — no watchdog, no negative cache on the delivery path. The routing log also windows a known-dead address to one fullErrorline per minute (repeats atDebug, counted into the next full line).Evidence and the deterministic tests are on #2426 and the PR. Post-deploy check on memex-cloud: after a gRPC participant disconnects, expect one
ClientSubscriptionEvictionwarning per owner that was serving it, and no sustainedRoutingGrainrefusal stream.fix(routing): pod-hub claim gets a derived lifetime; PodHubNotHere is…