Repository navigation
A message vanished from a live chat at end of streaming: a SubscribeRequest to a NotFound path never answers, so the view dies silently #1469
Description
A user's assistant message disappeared from the live chat view at the end of streaming and came back on reload. Reported by the thread owner; reproduced from logs.
Durable state was correct the whole time. What failed is the live subscription.
The evidence (memex-cloud, 2026-08-14, thread felice.buergi/_Thread/schreib-mir-eine-geschichte-es-war-einma-1345)
07:05:39.775 Node created …/e4e1a36f by felice.buergi ← her message
07:05:40.074 Node created …/927df3b1 by felice.buergi ← response cell
07:05:40.152 [ROUTE] NotFound: No node found at 'felice.buergi/felice.buergi'.
Closest ancestor is 'felice.buergi' (remainder='felice.buergi').
07:05:45.148 [ROUTE] NotFound: … same …
07:06:07.036 Node created …/_Notification/KKtZPD5W… by system-security ← round COMPLETED
07:06:13.155 [STALE-CALLBACK] cache/WMkjfyG-vkahlapz7sfKYw: 1 callback(s) pending > 30000ms:
hWtayyfN3Uq0u5B838dbcA=SubscribeRequest@felice.buergi/felice.buergi(33008ms)
07:06:18.155 … (38008ms)
07:06:23.154 … (43007ms)
The _Notification at 07:06:07 is EmitCompletionNotification, which only fires on a terminal state — so the answer was generated and durably written. Reload reads durable and shows it. The live view had no working feed.
Defect 1 (the generic one, and the reason the message vanished): a SubscribeRequest to a non-existent path NEVER ANSWERS
Routing knew at 07:05:40 — it logged NotFound and even resolved the closest ancestor. The requester was never told. The callback sat pending for 43 s and counting, re-warned every 5 s by [STALE-CALLBACK].
This is the never-answers class this repo has now fixed four times — #1362/#1364 (a handled GetDataRequest with no completion arm), #1360/#1409 (a re-ask routed back to the same dying hub), #1446/#1454 (a starved permission fold with no terminal), #1440/#1450 (FollowGo awaiting an undeliverable event). The (33008ms) shape is literally #1446's signature (Executing(CreateNodeRequest, 33014ms)).
Any live view whose subscribe targets a missing path is therefore silently dead until reload — this is not specific to chat, and the user-visible symptom (content vanishing mid-stream) gives no clue as to the cause.
🚨 Do not fix by adding a timeout. Routing already has the answer at the moment it logs NotFound; the fix is to deliver it. A timeout would convert a hang into a slow failure and still lose the view.
Defect 2 (the trigger): a doubled path felice.buergi/felice.buergi
Closest ancestor is 'felice.buergi' means the partition root exists and something asked for a child named after the user, inside that user's own partition — i.e. {partition}/{userId} where the two are the same string.
Scope: 3 occurrences in 20 h, only this user. Other NotFounds in the same window are the unrelated Admin/Menu/* set. So it is not universal — establish what is different about this account (a newer user? a home node never created? an id containing a dot?) before assuming a general rule.
This is the same shape as the path-doubling that #1459 removed from node files (MeshNode.Path => $"{Namespace}/{Id}"), but that was declared data; this is a runtime construction, so it is a distinct site.
Why this is worth prioritising
The two defects compose badly. Defect 2 is rare; defect 1 turns any occurrence of it into silent data loss from the user's point of view — the answer they watched being written vanishes, with no error, no toast, and no log the user can see. The only recovery they can discover is reloading.
Fix defect 1 first: it bounds the blast radius of every future missing-path subscribe, whatever the cause.
Activity
rbuergi commentedon Aug 14, 2026
Defect 2 (the trigger) — found, fixed, PR #1482
MeshOperations.ResolveContextPath(src/MeshWeaver.AI/MeshOperations.cs) decidesrelative-vs-absolute by looking for a
/: a token without one is treated as relative to thechat context. A single-segment mesh path is by definition a partition root, so when the chat is
anchored to one — a user's own home page — and that same path comes back through the resolver, it is
placed underneath itself.
The chain, each hop verified:
ThreadChatViewwrites the context chip intoattachmentsand shipsit unfiltered →
ThreadExecutionsetsAgentContext.ContextandSetAttachments→AgentChatClientcallsmeshPlugin.Get("@felice.buergi")→MeshPlugin.Get→ResolveContextPathreturns@felice.buergi/felice.buergi→MeshOperations.FetchNode→GetMeshNodeStream("felice.buergi/felice.buergi")→SubscribeRequest→RoutingGrainproducesthe exact production string.
Scope matches: a user anchored on a sub-node (
ACME/Doc/…) takes the existing "multi-segment ⇒already absolute" exit and never doubles, so only home-page chatters hit it — once per submitted
round, not per render. The dot in the id is not load-bearing;
rbuergidoubles identically. Itis silent because
AgentChatClientskips content starting with"Not found".Fix: the self-prefix guard
PathUtils.ResolveRelativePathalready carries, matched at a segmentboundary. Fail-before demonstrated.
Defect 1 — the generic claim did NOT reproduce
Measured against
origin/main, subscribing to a missing path and taking the first notification:GetMeshNodeStream)OnError30 msOnError21 msOnError63 msOnErrorOnError126 msOnError170 ms then cachedEvery one:
DeliveryFailureException: No node found at '…'. Closest ancestor is '…' (remainder='…'). So a missing-path subscribe is not silently dead as a rule, and there is nofailing test to write for the claim as stated. It also means the consumer side already distinguishes
the three states the issue asks about:
AreaErrorClassifier/MeshNodeStreamCache.IsMissingNodeFailuretreat "No node found" as a terminal, negative-cached, expected-user-action failure and render it,
while a transient reject stays out of the negative cache.
The two real gaps in that path, found by reading it
Both are in
RoutingGrainand neither is reachable by a clean-cluster test, which is why theysurvived:
RoutingGrain.PostFailureis the only inbound path to a co-hosted hub that requires theOrleans memory-stream subscription. Every other same-process delivery to
cache/…short-circuitson
OrleansRoutingService.DeliverMessage's local route table (OrleansRoutingService.cs:162-166)and never touches a stream;
PostFailure(RoutingGrain.cs:424) writes straight tostreamProvider.GetStream(delivery.Sender.ToString()). A publish to a stream with no livesubscriber succeeds, so "delivered" and "discarded" are indistinguishable — and unlike the
forward leg (
PostToStream, bounded at 60 s and NACK-on-failure) this post has no timeout atall: if
OnNextAsyncnever completes, theContinueWithnever runs and nothing is loggedanywhere.
SubscribeWhenStreamingReadyAsynccan also give up (returnsnullafter aLogCritical,:614-626) while the local route stays live — the code's own comment says"in-process routing remains active", which is true for everything except the NACK.
Compare
RoutingServiceBase.PostNotFoundandOrleansRoutingService.SendDeliveryFailure: bothpost through the mesh hub, and the latter returns
booland warns when it could not.The answer-once guards on the routed path are dead code. The mesh hub
Package()s thedelivery before handing it to
IRoutingService(MeshBuilder.cs:229-237), so by the time itreaches
RoutingGrain.PostFailure(:415-417) andOrleansRoutingService.MayAnswer(:380-382)delivery.Messageis alwaysRawJson—is DeliveryFailureandHasAttribute<CanBeIgnoredAttribute>()can never match. Confirmed empirically: the monolith logsRouteMessage: NotFound for RawJson → TestData/TestData. So the router NACKs heartbeats — exactlythe storm
JsonSynchronizationStream's comment says routing prevents ("an undeliverable heartbeatis [CanBeIgnored], so routing DROPS it without a NACK") — and can NACK a NACK.
Packagealready stampsIDiagnosticKeyed.DeliveryPropertyonto the envelope for precisely thisreason (carrying identity across the type erasure); the payload's type name needs the same
treatment.
Neither is shipped here: (1) cannot be proven to be what happened without evidence from the pod, and
guessing at a transport rewrite is the wrong trade; (2) is a real, separately-testable defect but a
different root from this issue.
What would settle the incident in one look
The
[STALE-CALLBACK]line is truncated in the report at(33008ms). It is followed by ahandler-side fate block (
MessageHub.ScanStaleCallbacks→FormatPendingCallbackFates):RESPONSE_ARRIVED_NO_SUBJECT type=DeliveryFailure⇒ the NACK arrived but landed on thewrong hub (correlation problem);
POSTED target=felice.buergi/felice.buergiwith nothing after ⇒ the NACK neverreached any hub in the process ⇒ gap (1).
Then, on the same pod:
stream subscription could not be attached for cache/·Failed to deliver NotFound failure to sender·reported a delivery error — frames may have been lost·Route dispatch faulted.One correction to the timeline reading
The two
NotFounds are not a repeat.SyncStreamOptions.FirstHeartbeatis 5 s, and the secondline lands at
07:05:45.148— 5.001 s after theSubscribeRequestwas posted (07:05:40.147,back-computed from the 33 008 ms age). So it is the sync stream's first heartbeat, and the
heartbeat firing at all proves the subscribe's
OnErrorhad not fired by then (keepAliveisdisposed on a terminal NotFound). After that the cadence is 45 s, which is why there is no third line
inside the window.
rbuergi commentedon Aug 14, 2026
The two gaps found by reading the routing failure path are now filed with their own reproductions and acceptance criteria, so they do not get closed along with this issue:
Package()erases the payload type before they run), so the router NACKs heartbeats and can NACK a NACK. Verified by readingMeshBuilder.cs:238,MessageDelivery.cs:182-208,RoutingGrain.cs:415,OrleansRoutingService.cs:380. The fix is precedented insidePackageitself, which already stampsDiagnosticKeyto carry identity across exactly this type erasure (Import activity triggers unbounded message storm to cache hub via RawJson/DataChangedEvent loop #1200).RoutingGrain.PostFailureis the only inbound path to a co-hosted hub that requires the Orleans stream subscription, and unlike the forward leg it is unbounded; a publish to a subscriber-less stream succeeds silently.Also recording the correction to this issue's own title, since it shaped the investigation: a missing-path subscribe is answered, terminally and fast — 21–170 ms across Monolith, 1-silo and 2-silo Orleans, from client / silo-side cache / client-side cache senders, in both the no-ancestor and ancestor-exists shapes. Consumers already distinguish the three states (
MeshNodeStreamCache.IsMissingNodeFailure,AreaErrorClassifier). The title's claim is false; the real trigger was the context-path doubling fixed in #1482.Merge pull request #1482 from Systemorph/fix/1469-context-path-doubling
rbuergi commentedon Aug 14, 2026
Fixed by #1482 (merged
c694ca358).What actually happened, since this issue's title describes a mechanism that does not exist:
MeshOperations.ResolveContextPathdecided relative-vs-absolute by looking for a/. A single-segment mesh path is a partition root, so a user chatting from their own home page had context == attachment =={userId}, and the prepend produced{userId}/{userId}— a node that cannot exist. The failed read was then discarded silently (AgentChatClientskips attachment content starting with"Not found"), so the round continued without the content it had asked for.The distinguishing factor was the single-segment context, not the dot in the username. Anyone anchored on a sub-node takes the multi-segment ⇒ already-absolute exit and never doubled. It fires once per submitted round, which is why it was 3 occurrences in 20 h on one account rather than constant — it reads like a rare race and is not one.
Correction to the title, recorded so nobody re-investigates it: a
SubscribeRequestto a NotFound path DOES answer — terminally, in 21–170 ms, measured across Monolith, Orleans 1-silo and 2-silo, from client / silo-side cache / client-side cache senders, in both the no-ancestor and ancestor-exists-with-remainder shapes. AlwaysDeliveryFailureException … ErrorType.NotFound. Consumers already distinguish the three states (MeshNodeStreamCache.IsMissingNodeFailure,AreaErrorClassifier). Related trap: a secondNotFound~5 s after a subscribe isSyncStreamOptions.FirstHeartbeat, and its firing proves the subscribe'sOnErrorhad not fired, since a terminal NotFound disposeskeepAlive.Durable state was correct throughout — the
_Notificationsatellite shows the round completed, which is why a reload brought the message back.The two genuine routing defects found while investigating this are filed separately and are not closed by this: #1485 (both routers' answer-once guards are dead code after
Package()erases the payload type) and #1486 (PostFailureis unbounded and uniquely dependent on the Orleans stream subscription).