Repository navigation
Activity tracking times out: _UserActivity per-node hub never delivers initial state to the mirror #5115
Description
What is failing
The MeshWeaver.Graph.ActivityTracking service fails to record user-activity updates: it creates/updates a _UserActivity node, then the awaiting update times out after 30s because no initial state ever arrives at the mirror — the per-node hub either never activated, or activated but its MeshDataSource produced no snapshot.
Probable cause
Confidence: medium. The exception explicitly names a wedged per-node hub (already tracked as MeshWeaver#2896) and RLS silently rejecting the prior CreateNode as the leading candidates, and states it cannot distinguish between routing failure, a wedged owner, or a lost reply. This fingerprint — BaseStateTimeoutException with "no change item at all reached the mirror" — matches the same symptom reported across many users and paths over the past month, so this is very likely the same underlying defect rather than a new one: a _UserActivity hub (or its MeshDataSource initial read) failing to hydrate under load. The concrete case is user rbuergi, path ClientA/Esl/GitHubActionsIdentityAndBuildQueueMail, on the memex namespace.
Impact
Low for this instance — one occurrence on one pod, an activity-history entry silently not recorded. However, the folded parent fingerprint recorded 425 occurrences across 59 pods over ~6 weeks for the same log site, so the systemic issue is ongoing and user-facing activity tracking is intermittently unreliable; if you can connect the two, the aggregate impact is material.
Where to look
MeshWeaver.Graph.ActivityTracking— the code path that creates/updatesuser/_UserActivity/<path>nodes and awaits the hub's initial state (30s bound viaMeshNodeStreamHandle.BaseStateTimeoutException).- Investigate why a subscribed hub acks but produces no frame: cold
MeshDataSource, per-node hub still activating, or RLS rejecting theCreateNodeunder the service identity. - Known related issues: MeshWeaver#1174 (prior ticket for this exact log site, filed 2026-09-18, likely a duplicate — close as such after confirming) and MeshWeaver#2896 (wedged per-node hub).
Evidence
| Fingerprint | 38ea9b265feaa23e |
| Category | MeshWeaver.Graph.ActivityTracking |
| Severity | Error |
| Exception | System.TimeoutException |
| Namespace | memex |
| Pods | memex-portal-deployment-75b545bbd-gwrc6 |
| Occurrences | 1 |
| First seen | 2026-09-21 08:42:23Z |
| Last seen | 2026-09-21 08:42:23Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Graph.ActivityTracking. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-21 08:42:23Z memex-portal-deployment-75b545bbd-gwrc6 fail: MeshWeaver.Graph.ActivityTracking[0]
Failed to track activity for user=rbuergi path=ClientA/Esl/GitHubActionsIdentityAndBuildQueueMail
System.TimeoutException: Update aborted: no initial state arrived for 'rbuergi/_UserActivity/ClientA_Esl_GitHubActionsIdentityAndBuildQueueMail' within 30s. Likely causes — (1) RLS silently rejected the prior CreateNode, (2) the path is misspelled / points at a namespace no NodeType claims, (3) the node was deleted between create and update, or (4) the per-node hub activated but its MeshDataSource didn't load the node from persistence. NO change item at all reached the mirror inside the bound — no snapshot ever hydrated. This does NOT establish that the owner never answered: a fresh subscribe is ACKNOWLEDGED BEFORE hydration, so an owner that acked and then produced no frame (a cold data source, a per-node hub still activating, an initial read that yielded nothing) reads identically here. Both the owner's activation/routing AND its initial read are still in scope.
---> MeshWeaver.Mesh.MeshNodeStreamHandle+BaseStateTimeoutException: NO change item at all reached the mirror inside the bound — no snapshot ever hydrated. This does NOT establish that the owner never answered: a fresh subscribe is ACKNOWLEDGED BEFORE hydration, so an owner that acked and then produced no frame (a cold data source, a per-node hub still activating, an initial read that yielded nothing) reads identically here. Both the owner's activation/routing AND its initial read are still in scope.
--- End of inner exception stack trace ---
Re-addressed by the current identity function: this incident inherited 7473a637daf1e169 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/38ea9b265feaa23e. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 22, 2026
Same log site as #1174 — proved by fingerprint lineage — and the 2026-09-21 decision to keep them apart is right
Grouped as part of the
area:meshtimeout cluster.This ticket's body says "MeshWeaver#1174 (prior ticket for this exact log site, filed 2026-09-18, likely a duplicate — close as such after confirming)". Confirmed that they are the same site, and confirming is not a reason to close.
The evidence that it is the same site
This ticket's footer: "Re-addressed by the current identity function: this incident inherited
7473a637daf1e169".7473a637daf1e169is #1174's fingerprint (its Evidence table). So the identity function re-keyed one log site from7473a637daf1e169to38ea9b265feaa23eand filed a second ticket for it. SameMeshWeaver.Graph.ActivityTracking[0]category, sameFailed to track activity for user=… path=…, sameTimeoutException→BaseStateTimeoutExceptionterminal. That is a split, not two faults.Why the split stays split
The 2026-09-21T19:52Z consolidation comment on this ticket already decided it and gave the reason: "folding an undiagnosed family into a diagnosed root buries whichever is wrong." That is the correct rule and it holds here:
TrackActivitydecides create-vs-update from an eventually-consistent query POSITIVE atMeshNodeExtensions.cs:320-323→:379→:392) and a named blocker (CreateOrUpdateNodeRequest cannot express a counter increment, so the one verb with owner-side existence and the ghost-hydration repair is unavailable to the callers that need it (blocks #1174) #4928 — the upsert cannot express a fold).AGENTS.mdnow names A CreateNodeRequest addressed to the sending hub itself is never answered (nodeops, empty queue, RunLevel=Started) — activity tracking and every downstream read then time out #1174 by number as the fallback its CQRS rule forbids.If the family were closed on #1174's fix and the stall were a different cause, the remaining occurrences would have no ticket. If #1174 were closed into the family, its diagnosis and its #4928 gate would be lost. Keep both, cross-referenced by terminal. The general form is committed as
Doc/Architecture/OneLogSiteManyTerminals(PR #5170).One eliminated hypothesis, recorded on #1174
The
Workspace.EvictForPathmechanism proposed on #1174 (every hot write re-hydrates a brand-new mirror inside the 30 s bound) predicted census branch one and was to be falsified once a replica carrying48a3beerolled. Branch one has since been read on images carrying #4428, the change that stopsEvictForPathevicting the writer's own mirror — including this ticket's own 2026-09-21T08:42:23Z sample, which prints "NO change item at all reached the mirror inside the bound — no snapshot ever hydrated". So that competing hypothesis is out, and the family's cause is still open. Full reasoning and its caveat on #1174.Disposition
Leave open as the symptom family,
sev:M. The two discriminators the consolidation comment names are still the right next steps and I ran neither:ActivityTrackingentirely;UPSERT_READ absentvsUPSERT_READ existingstage, which is durable truth and which the query cannot answer at all.What I did NOT establish
rbuergi commentedon Sep 23, 2026
Duplicate of #5256. Same log site (
MeshWeaver.Graph.ActivityTracking), same_UserActivitywrite, same terminal: "NO change item at all reached the mirror inside the bound — no snapshot ever hydrated" (branch one of the base-state census).The only difference is the wrapper, and the code explains it:
TimeoutExceptionfromstream.Update.ad17f6e5c9, in both running builds), the same stall surfaces through the upsert's update leg asInner UpdateNode faulted: …, which is Activity tracking fails to hydrate user activity node — inner UpdateNode aborts with no initial state within 30s #5256.#5256 carries the current-code analysis: every sample falls in a roll window, and there is a named mechanism (the owner acks a fresh subscribe, then tears down before its first frame, and the teardown announcement is suppressed). It also has the post-roll discriminators. The consolidation note here kept this ticket apart from #1174 (a diagnosed root). #1174 is now closed as fixed, so the symptom family belongs on one ticket. Closing as duplicate.
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by @rbuergi on 2026-09-21)
Consolidation 2026-09-21 — 2 tickets folded here
Folded as duplicates: #5109 #5124. Three users (
rsalzmann/SecureDevelopmentFramework,rbuergi/ClientA/Esl/…,mkleiner/LearningRoadmap), one shape:ActivityTrackingwrites a_UserActivitynode, the subscribe is acked, no initial state ever reaches the mirror, and the update aborts withMeshNodeStreamHandle+BaseStateTimeoutExceptionat 30 s.Kept as a symptom family on purpose. #5109 named #1174 as its likely root and the exception text names #2896; both are hypotheses. #1174 (a
CreateNodeRequestaddressed to the sending hub itself is never answered) and #1122 (per-node hubDataContextinit times out at 120 s) stay open — folding an undiagnosed family into a diagnosed root buries whichever is wrong.The two discriminators worth running
CreateNode, a namespace no NodeType claims, a node deleted between create and update — and toward a systemic hydration stall.RoutingGrainback-pressure and Orleans queue read-timeout incidents in the same window. If these timeouts track routing saturation (RoutingGrain dispatch slots saturate with legs head-of-line blocked behind one shared-cache destination while the second pod starves on load #5009/RoutingGrain dispatch pool saturates repeatedly on portal silos; slots held through unbounded ThreadPool wait and episode-stamp splitting floods incidents #5018), the cause is upstream ofActivityTrackingentirely and this ticket is a symptom of that.Aggregate denominator, from #5115's own folded parent fingerprint: 425 occurrences across 59 pods over ~6 weeks for this log site — so the family is continuous, not a one-day event. Per-instance impact is one unrecorded activity entry (a stale "recently viewed"); no data loss beyond that, no outage — hence
sev:Mand notsev:H, since nothing measured shows a broken user-facing feature.Also on the record from #5109: the newer exception text states explicitly that a subscribe is acked before hydration, so an acked subscribe is not evidence of a live mirror.