Repository navigation
Fix MCP-reconnect wedge: originate activity tracking from a dedicated activity hub - #625
Merged
Merged
Conversation
… activity hub
Production wedge (memex-cloud 2026-07): MCP client reconnect was rejected. Token
validation succeeded; the connection wedged on activity tracking. TrackActivityRequest
was handled on whichever hub posted it — the top-level portal hub or a transient
per-connection / MCP back-connection hub. Its GetMeshNodeStream(activityPath).Update(...)
therefore opened its IMeshNodeStreamCache sync subscription on that hub's cache
(cache/{connectionId}), whose initial-state / PatchDataResponse routed back through the
hub's transient, UNREGISTERED mesh root (mesh/{connectionId}) -> [ROUTE] NotFound -> the
write stalled 30s ("no initial state arrived within 30s") -> the reconnecting client was
rejected.
Fix: all activity tracking now originates from a dedicated, stable, registered
ActivityTrackingHub, hosted off the mesh ROOT and resolving the mesh root's SHARED,
registered IMeshNodeStreamCache (cache/{meshRootId}) by DI fallback — never a
per-connection cache. Its sync-subscription responses therefore route to a registered
hub, never a transient mesh/{connectionId}.
- ActivityTrackingHub: lazy, idempotent hosted hub at activity/_tracking off the mesh
root; AddData (own workspace) + WithGraphTypes (typed content) + RegisterStream with
routing. Registers no persistence, so IMeshNodeStreamCache resolves to the root's
shared singleton.
- HandleTrackActivity: relocated onto the tracking hub's workspace; caller AccessContext
captured and re-established across each cold write via Observable.Using(Impersonate) on
the tracking + root AccessServices (AsyncLocal does not survive Rx scheduler hops).
Fully reactive, fully observed — faults surface at Error, nothing fire-and-forget.
- UserContextMiddleware.TrackLogin: removed the swallowing catch {} (log at Warning
instead); auth still never depends on tracking.
- Repro/regression: ActivityTrackingHubTest pins the fix's contract — the tracking hub is
stable, hosted off the mesh root, and shares the mesh-root cache (the invariant the old
code violated), and a track initiated from a distinct connection-style hub still lands
the node. (The exact 30s stall needs an unregistered per-connection mesh root, a
distributed condition a monolith cannot express.)
Tests green: ActivityTrackingHubTest (2), UserActivityTrackingTests (3),
UserActivityDashboardQueryTests + RecentlyAccessedSearchTest (9),
UserOnboardingServiceTests (5), ApiToken/OAuthConnect (76).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012wP3c7DLHutUczcBWgs4h8
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses an MCP reconnect wedge caused by user-activity tracking (TrackActivityRequest) originating from transient per-connection hubs, which led to unroutable sync-subscription responses and 30s stalls. The fix introduces a dedicated, stable activity-tracking hosted hub off the mesh root and updates activity tracking to originate writes from that hub (with explicit AccessContext re-stamping and surfaced faults), plus a regression test to pin the contract.
Changes:
- Added a dedicated
ActivityTrackingHub(activity/_tracking) hosted off the mesh root and registered with routing. - Updated
HandleTrackActivityto originate tracking writes from the tracking hub’s workspace and re-establish AccessContext for cold writes. - Added an integration-style regression test to validate hub stability/root-cache sharing and that tracking initiated from a connection-style hub still lands.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| test/MeshWeaver.Query.Test/ActivityTrackingHubTest.cs | New regression test pinning the “tracking hub is stable/off-root/shares root cache” contract and ensuring a connection-style hub post lands a UserActivity node. |
| src/MeshWeaver.Graph/MeshNodeExtensions.cs | Routes activity tracking writes through the dedicated activity hub and re-stamps AccessContext across reactive boundaries. |
| src/MeshWeaver.Graph/Configuration/ActivityTrackingHub.cs | Introduces the dedicated activity-tracking hosted hub configuration and routing registration. |
| src/MeshWeaver.Blazor/Infrastructure/UserContextMiddleware.cs | Stops silently swallowing synchronous post failures for login tracking; logs failures at Warning while keeping auth independent. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
Contributor
Contributor
Contributor
Contributor
Contributor
Contributor
…r identity; drop unused using Finding 1 (AccessContext timing): workspace.GetQuery(...) applies per-user RLS by capturing AccessService.Context EAGERLY at the GetQuery call (WrapWithPerUserRls in SyncedQueryDataSourceExtensions), on the workspace's hub — the tracking hub. The pipeline built GetQuery OUTSIDE the impersonation, so the RLS wrap captured the tracking hub's (empty/wrong) context, not the caller's. Fix: wrap the whole read+write pipeline in Observable.Using(Impersonate, _ => ...) so GetQuery is CALLED with the caller's context established on the tracking + root AccessServices. Fail-closed when callerCtx is null (empty userId => RLS yields no rows for the exact-path probe; the subsequent write posts context-null and PostPipeline rejects it). The inner per-write Observable.Using(Impersonate) remain — AsyncLocal does not flow across Rx scheduler hops, so each write re-establishes identity on its own emission thread. Test extended: asserts the created UserActivity node's CreatedBy == caller (not system/empty), confirming the read+write run under the caller's identity. Finding 2 (trivial): removed unused `using System.Reactive.Linq;` from ActivityTrackingHub.cs (no Observable operators there). Tests green WITH restore: ActivityTrackingHubTest + UserActivityTrackingTests + UserActivityDashboardQueryTests + RecentlyAccessedSearchTest (14); ApiToken + OAuthConnect + UserOnboardingServiceTests (81). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012wP3c7DLHutUczcBWgs4h8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem (MCP client "rejected on reconnect", root cause confirmed from live Loki)
After #620/#624, the OAuth exchange succeeds and a token is issued — but the client's reconnect wedges. Token validation is not the cause. The failure is activity tracking:
TrackActivityRequestwas registered viaWithGraphTypes → WithHandler<TrackActivityRequest>on every hub, soHandleTrackActivityran on whichever hub posted it — the top-level portal hub or a transient per-connection / MCP back-connection hub. ItsGetMeshNodeStream({userId}/_UserActivity/{userId}).Update(...)therefore resolved that hub'scache/{connectionId}, and the cross-hub write's sync initial-state response had to route back through that hub's transient, unregistered mesh rootmesh/{connectionId}:→ the round wedges → the reconnecting client is rejected. Fire-and-forget (
TrackLogin's swallowingcatch) buried it.Fix
ActivityTrackingHub(activity/_tracking), hosted off the mesh root, created lazily + idempotently. It registers no persistence of its own, soIMeshNodeStreamCacheresolves to the mesh root's shared, registered singleton cache — the write's sync responses route to a registered address, nevermesh/{connectionId}.HandleTrackActivityrelocated onto the tracking hub's workspace; services resolved from the mesh root; callerAccessContextre-established across each cold write viaObservable.Using(Impersonate)(AsyncLocal doesn't survive Rx scheduler hops). Fully reactive, fully observed, faults surfaced at Error — nothing fire-and-forget.UserContextMiddleware.TrackLogin: removed the swallowingcatch{}; post faults log at Warning; auth still never depends on tracking.Verification
ActivityTrackingHubTestpins the contract the old code violated (tracking hub stable/idempotent, hosted off the mesh root, resolves the SAME shared cache as the root; a track initiated from a distinct connection-style hub still lands). The exact 30s stall is a distributed phenomenon a monolith can't express (in-process routing ignores stream registration), so the monolith test pins the fix's contract; the stall's disappearance is verified against the live portal after deploy.dotnet build -c Release -warnaserrorclean; ActivityTrackingHubTest + UserActivity + ApiToken + OAuthConnect suites green (real runs).Note (separate decision)
This does NOT convert the deduped
_UserActivityaccess counter into append-only_Activity/ActivityLognodes — that's a semantic/data redesign (RecentlyViewed/dashboard/source:accesseddepend on the counter) tracked separately.🤖 Generated with Claude Code