Skip to content

Content-collection read request is issued from the router hub, so its GetDataResponse lands on the mesh router #5621

Closed as not planned

Description

@systemorph

What is failing

A GetDataResponse posted by the content-collections config handler was delivered to the root mesh hub (mesh/EfNT…) instead of the requesting actor. The root hub is the mesh's router and must not be an end of a work delivery; work landing on its action block competes with routing and can starve real SubscribeRequest traffic (the 2026-06-11 prod wedge).

Probable cause — medium-high confidence.

The report is a target-role ORIGIN line at ContentCollectionsExtensions.HandleCollectionConfigRequest (ContentCollectionsExtensions.cs:344): the handler is the answering half — it posts its response back to wherever the request came from (ResponseFor(delivery) / the request's sender). So the offending side is the requester that issued the collection-config/GetData request with the router (mesh/{id}) as sender — almost certainly a mesh-singleton holding the DI-injected IMessageHub, which in the root container is the router. This matches the known residue documented in Router Traffic Detection: "three content-collection reads" kept posting as mesh/{id} off a proven-router hub field after #4477 hopped only the lifecycle verb.

The fix is therefore not at the named call site (nothing to hop on a reply). It is to hop the requesting call site onto hub.ReadIssuingHub() (portal/reads-{meshId}) — the correct seam for a bounded one-shot read. Do not use StreamSubscribingHub() here, and never hop a subscription onto ReadIssuingHub() (it registers no handlers by design).

Impact

One occurrence on one pod, deduped per (role, type) per hub, so this is the first sighting of this shape — not an outage. But the class it belongs to is the known precursor to a portal-wide wedge: router-block work starving routing. Worth closing while it is one line rather than a burst.

Where to look

  • The reply side (named by the report, innocent): MeshWeaver.ContentCollections.ContentCollectionsExtensions.HandleCollectionConfigRequest (ContentCollectionsExtensions.cs:344).
  • The request side (the actual offender, to be identified): find the caller issuing CollectionConfigRequest/GetData for collections off the DI-injected IMessageHub — check MeshOperations and the content-collection readers noted in Router Traffic Detection as unhopped sibling exchanges.
  • Reference shape for the fix: MeshService hops both IssuingHub + NodeOperationTarget.

Related prior art in the same family: #4463, #4477, #4614/#4615/#4617, #4697. Not a duplicate — this is a new (role, call-site) pair, and unlike #4697 the requester-side offender is not yet identified.


Evidence

Fingerprint 8cdc3487631aca71
Category MeshWeaver.Messaging.MessageHub
Severity Error
Top frame MeshWeaver.ContentCollections.ContentCollectionsExtensions.HandleCollectionConfigRequest (ContentCollectionsExtensions.cs:344)
Namespace memex-cloud
Pods memex-portal-deployment-76cf776847-9p7z4
Occurrences 1
First seen 2026-09-23 20:56:19Z
Last seen 2026-09-23 20:56:19Z
Routing not determined — no configured route matches the category MeshWeaver.Messaging.MessageHub. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines
2026-09-23 20:56:19Z memex-portal-deployment-76cf776847-9p7z4 fail: MeshWeaver.Messaging.MessageHub[0]
      ROUTER_TRAFFIC ORIGIN: GetDataResponse was POSTED with the mesh hub as target (sender: Posts, target: mesh/EfNTPfFIy0WLbHUEEmk3jg). The mesh hub is the ROUTER and must not be an end of a work delivery. If the role above includes 'sender', this post LEFT the router: hop it onto the seam that matches what this delivery IS — MeshExtensions.NodeOperationIssuingHub() for a node LIFECYCLE write, MeshExtensions.ReadIssuingHub() for a bounded one-shot READ, MeshExtensions.StreamSubscribingHub() for a remote stream SUBSCRIPTION. The three are NOT interchangeable and the wrong one fails SILENTLY: ReadIssuingHub() registers no handlers by design, so a subscription hopped onto it stops reporting AND stops receiving data (#4614). If the role above includes 'target', ask where that target CAME FROM, because the two cases have opposite fixes: if this call site CHOSE the router as the destination, the call site is the offender — address the owning node instead (MeshExtensions.NodeOperationTarget()); if the target was read off an incoming request or subscription (request.Subscriber, ResponseFor(delivery) — SubscribeAck, DataChangedEvent, StreamErrorEvent and StreamEndedEvent all are), then this call site is the innocent answering half and the hub to move is the one that SUBSCRIBED or REQUESTED (#4697). 'sender AND target' means both apply. Reported once per role+type for this hub. Call site:
         at MeshWeaver.ContentCollections.ContentCollectionsExtensions.HandleCollectionConfigRequest (ContentCollectionsExtensions.cs:344)
         at MeshWeaver.Messaging.MessageHubConfiguration+<>c__DisplayClass65_0`1.<WithHandler>b__0 (MessageHubConfiguration.cs:312)
         at System.Reactive.Linq.ObservableImpl.SelectMany`2+ObservableSelector+_.OnNext
         at System.Reactive.Linq.ObservableImpl.SelectMany`2+ObservableSelector+_+InnerObserver.OnNext
         at System.Reactive.BasicProducer`1.SubscribeRaw
         at System.Reactive.Linq.Ob…[truncated]

Opened automatically from Admin/_LogIncident/8cdc3487631aca71. Recurrences are folded into this issue rather than opening new ones.

Activity

  1. added
    bugSomething isn't working
    sev:MMedium - secondary path broken, or easy workaround
    on Sep 23, 2026
  2. rbuergi commented on Sep 24, 2026

    @rbuergi
    Contributor

    Duplicate of #5620 — same log site MeshWeaver.Messaging.MessageHub: ROUTER_TRAFFIC[ ORIGIN]: {type} … with the mesh hub as sender/target; consolidated.

    Read before closing: The ORIGIN half of the SAME exchange #5620 reports: 2026-09-23 20:56:19Z, pod 76cf776847-9p7z4, router mesh/EfNTPfFIy0WLbHUEEmk3jg ↔ Posts, one millisecond apart.

    The fix, verification and any recurrence are tracked on #5620.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingsev:MMedium - secondary path broken, or easy workaround

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions