Per-user _Answers path resolution times out when a learner opens a quiz (the NotFound half was fixed by #3536; same fingerprint, different cause) #3517

Activity

1. rbuergi commented on Sep 7, 2026

   

   rbuergi

   on Sep 7, 2026

   Contributor

   More actions

   Located, and the "where the fix belongs" question splits cleanly across two repos

   The issue rates the mechanism high-confidence and the location medium. Here is the location, measured.

   piece	repo	path
   RadioGroupView	MeshWeaver.Plugins	src/MeshWeaver.Blazor.EntityViews/RadioGroupView.razor
   the binding seam GetMeshNodeDataContext	core	src/MeshWeaver.Data.Contract/LayoutAreaReference.cs:104

   So it is genuinely a two-repo question rather than an ambiguous one, and which repo owns the fix depends on which of two things is being asserted.

   🚨 The platform rule already decides the mechanism, and it is not "handle the absent case"

   Doc/Architecture/CqrsAndContentAccess and AGENTS.md are explicit, and this is the exact shape they name:

   A point read of an absent node is a framework defect, not merely slow: the owner answers a routing NotFound that terminates the stream AND opens the storm-breaker on that path — and the breaker fast-fails writes too, so the read suppresses the write it is waiting for.

   A node that may NOT EXIST YET needs BOTH halves — a scope:children listing for existence, then GetMeshNodeStream(target) for content.

   That reframes the fix. The view re-attempting every render pass is the symptom; the defect is that something is issuing a point read on a path that may not exist, which the rule forbids outright. 🚨 So a try/catch in RadioGroupView, or a null-check that swallows the DeliveryFailureException, would be a band-aid and would leave the storm-breaker open on that path — suppressing writes to it, which is the part that hurts and the part the log noise hides.

   The two candidate fixes, and how to choose between them

   1. The binding seam is wrong to point-read (core). If GetMeshNodeDataContext is meant to be usable for a node that may not exist yet, it needs the two-half shape, and every view bound through it is fixed at once — 473 occurrences across two unrelated spaces (a _Draft event's SendDocument form and a course quiz's _Answers) argues this is not one view's bug.
   2. The caller is wrong to bind before creating (Plugins, and per-site). Doc/GUI/DataBinding already says a node-bound editor must have its node created first — "NEVER … on an absent path (it NotFound-storms)". Both samples fit: one node deleted while the page was open, one never created before the form rendered. On this reading the fix is CreateOrUpdateNodeRequest at the two call sites, not in the view at all.

   These are not exclusive and the honest answer is probably both — (2) for the two known sites now, (1) so the next binding does not have to remember. What decides it is whether GetMeshNodeDataContext is specified to tolerate an absent node; that is a maintainer call about the seam's contract, not something to infer from the samples.

   🚨 A trap for whoever picks this up

   /Users/roland/code/MeshWeaver/src/MeshWeaver.Blazor.EntityViews/ exists on disk in the core checkout and is tracked by git in neither repo — 0 files under git ls-files, an untracked leftover of the Blazor extraction. Editing RadioGroupView.razor there would compile locally and be invisible to git, to CI, and to review. The real file is in MeshWeaver.Plugins. Check git ls-files --error-unmatch <path> before editing anything under a src/MeshWeaver.Blazor* directory in this checkout.

   Sequencing

   The Plugins half is in src/**, which is compiled into the image and is not synced into live portals by a registration — so the live-portal source-sync rule (Plugins#1430) does not gate it. The lane pin does: if it adopts a new core seam from fix (1), MW_PLATFORM_REF must carry it, checked with git cat-file -e <pin>:<path> and never by date.

   Not started tonight — fix (1) changes a contract in MeshWeaver.Data.Contract that every node-bound view depends on, which is not a 03:40 change. This comment is the analysis so the next reader starts at the decision rather than the search.
2. rbuergi

   added a commit that references this issue on Sep 7, 2026

   test(#3517): pin that the gate never blanks a node that EXISTS

   ...

   ffe792b
3. rbuergi commented on Sep 7, 2026

   

   rbuergi

   on Sep 7, 2026

   Contributor

   More actions

   Resolved in the SEAM (core), and the contract was not silent

   Being worked in #3536 — claiming it here so a second session does not start the same fix.

   The decider you named — "is GetMeshNodeDataContext SPECIFIED to tolerate an absent node?" — has an answer, but not on that method. GetMeshNodeDataContext (LayoutAreaReference.cs:104) is a pure string encoder and says nothing about node lifetime. The seam that actually reads is MeshNodeBindingExtensions.Bind (src/MeshWeaver.Mesh.Contract/), and its contract names this exact case:

   • the node is genuinely ABSENT, in which case the Where(node is not null) below filters every emission and this observable never emits or errors AT ALL …

   It degrades rather than errors, deliberately. … the budget emits the same null the binding emits for an absent field, so the control draws, AND the subscription stays live so a late value replaces it.

   So the seam was already specified to tolerate an absent node and draw the control empty. It got the mechanism wrong: absence does not arrive as a filtered null — routing mints an authoritative NotFound that terminates the stream and opens the storm-breaker window on the path. That is fix (1), and it is where this landed.

   And both samples independently rule the call sites out

   Sample	The "bind before create" reading	What the code says
   rbuergi/_Draft/Event_…	the form bound before its node existed	it did not. SendDocumentLayoutArea calls EmailDraftNodeType.EnsureExists — create-before-bind, already applied — and stormed anyway, because the _Draft subtree was deleted with the tab still open. No call-site discipline prevents that.
   roger.sas2026/_Answers/…/Quiz	the same mistake again	the learner's answers node is written by the first answer, on purpose. Creating it on render would write a node for everyone who merely looked at the page.

   So fix (2) is not the missing half — it was already done at one site and would be wrong at the other. Core-only; nothing to change in MeshWeaver.Plugins, and MW_PLATFORM_REF needs no move because the Plugins half is unchanged.

   What landed

   Bind now composes the shape Doc/Architecture/CqrsAndContentAccess → An OPTIONAL node requires: a live exact-path query (path:{x} ⇒ QueryScope.Exact, zero rows on absent, no NotFound, no breaker window) answers whether; the owner's stream answers what. Absent ⇒ the control draws empty and the binding stays subscribed, so the node appearing populates it — no .Take(1), no re-render.

   A catch in the view was explicitly not an option, and the doc now says why: it hides the fault and leaves the breaker open on the path, and the breaker fast-fails writes too — so the suppressed read goes on suppressing the write the form is about to make.

   Three arms on a real monolith mesh (NodeBoundBindingToleratesAnAbsentNodeTest), all green, arms 1–2 falsified RED against the pre-fix seam with the exact production message (No node found at 'TestData/…'. Closest ancestor is 'TestData'). Arm 3 pins the one way this fix could have been worse than the bug and silent: a gate that reported "absent" for a node that exists — asserted on a satellite path (ThreadComposerView's shape) and a partition root (SettingsLayoutArea's shape).

   One thing NOT fixed here, deliberately

   The quiz's write half. MeshNodeBindingExtensions.Write goes through GetMeshNodeStream(path).Update, which mutates and cannot create — so a learner's first answer still has nothing to land on until the _Answers node exists. That is a call-site concern (CreateOrUpdateNodeRequest at the quiz), it lives in in-mesh course content rather than either repo's src/, and it is a different defect from the read storm this issue reports. Worth its own issue rather than widening this one.
4. rbuergi

   added 3 commits that reference this issue on Sep 7, 2026

   fix(#3517): the existence gate reads the CHANGE STREAM, not the snaps…

   ...

   24fd359

   docs(#3517): say why the existence gate is not cached per path

   ...

   97d4451

   docs(#3517): correct two phrases the review round made stale

   ...

   76bfdbb
5. rbuergi commented on Sep 7, 2026

   

   rbuergi

   on Sep 7, 2026

   Contributor

   More actions

   Fixed and merged — #3536

   Closing by hand: the merge queue does not fire Closes links.

   Ancestry verified, not assumed — git merge-base --is-ancestor on both, plus the content itself:

   merge commit	0ed02badc515b4656517f0b24797a1730f7f68d1 — ancestor of origin/main ✅
   PR head	76bfdbb398dc944461539c3820fe80ca6e58c7db — ancestor of origin/main ✅
   MeshNodeBindingExtensions.cs on origin/main	carries the gate ✅
   NodeBoundBindingToleratesAnAbsentNodeTest.cs on origin/main	present ✅

   Which half, and what decided it. The seam, in core — and the decider was that
   MeshNodeBindingExtensions.Bind's own contract already named "the node is genuinely ABSENT" as a
   case it handles and specified the behaviour. It had the mechanism wrong: absence does not arrive as
   a filtered null, it arrives as a routing NotFound that terminates the stream and opens the
   storm-breaker. Both production samples independently ruled the call sites out — SendDocument already
   creates its draft (EmailDraftNodeType.EnsureExists) and stormed anyway because the node was deleted
   with the page open; the quiz's _Answers node is written by the first answer on purpose.

   Bind now composes the two-half shape: a live exact-path query (QueryScope.Exact — zero rows on
   absent, no NotFound, no breaker window) answers whether, the owner's stream answers what. Absent ⇒
   the control draws empty and the binding stays subscribed, so the node appearing populates it.

   Verified by falsification, both directions. With the gate removed, both #3517 shapes fail on their
   first assertion with the exact message from this issue's log samples (No node found at 'TestData/…'. Closest ancestor is 'TestData'). A third arm pins the one way the fix could have been worse and
   silent — a gate reporting "absent" for a node that exists — on a satellite path and a partition root.
   Review then caught a real instance of exactly that (the gate was on the snapshot query surface, which
   seeds .StartWith(empty) and so answers false first for every node); fixed, and the test
   strengthened to assert the first emission rather than a later one.

   The reasoning is committed, not just filed here: Doc/Architecture/CqrsAndContentAccess → An OPTIONAL
   node gained a section on the GUI binding, and Doc/GUI/DataBinding's "create it first" line is
   corrected — that rule is about writes; tolerating an absent node on the read side is the
   framework's job, not the author's.

   🚧 Deliberately NOT fixed here — worth its own issue

   The quiz's write half. MeshNodeBindingExtensions.Write goes through
   GetMeshNodeStream(path).Update, which mutates and cannot create, so a learner's first answer still
   has nothing to land on until the _Answers node exists. That is a call-site concern
   (CreateOrUpdateNodeRequest at the quiz), it lives in in-mesh course content rather than either
   repo's src/, and it is a different defect from the read storm this issue reports. It is no worse
   than before — previously the breaker was also fast-failing that write — but it is not fixed.
6. rbuergi

   closed this as completedon Sep 7, 2026
7. systemorph

   reopened this on Sep 15, 2026
8. systemorph commented on Sep 15, 2026

   

   systemorphBot

   on Sep 15, 2026

   Contributor

   More actions

   Reopened — this fault is happening again.

   Still occurring: 33 more since the last update (507 total).

   • Last seen: 2026-09-15 18:52:48Z
   • Pods: memex-portal-deployment-77cfc55bfc-txpk7, memex-portal-deployment-77cfc55bfc-bg4lk, memex-portal-deployment-5b9cbf46dd-5nqbz, memex-portal-deployment-5b9cbf46dd-gs6j6, memex-portal-deployment-58c859d4cc-b8pfg, memex-portal-deployment-7fd9668b69-jwzzl

   Recent log lines

   2026-09-07 03:53:59Z memex-portal-deployment-58c859d4cc-b8pfg fail: MeshWeaver.Blazor.EntityViews.RadioGroupView[0]
         Loading 'answers/biggest-shift-engineer' in area 'Quiz/2/1/Options1' in Area Quiz/2/1/Options1
         MeshWeaver.Messaging.DeliveryFailureException: No node found at 'roger.sas2026/_Answers/AgenticEngineering/Introduction/Quiz'. Closest ancestor is 'roger.sas2026' (remainder='_Answers/AgenticEngineering/Introduction/Quiz').
   2026-09-07 03:53:59Z memex-portal-deployment-58c859d4cc-b8pfg fail: MeshWeaver.Blazor.EntityViews.RadioGroupView[0]
         Loading 'answers/biggest-shift-engineer' in area 'Quiz/2/1/Options1' in Area Quiz/2/1/Options1
         MeshWeaver.Messaging.DeliveryFailureException: No node found at 'roger.sas2026/_Answers/AgenticEngineering/Introduction/Quiz'. Closest ancestor is 'roger.sas2026' (remainder='_Answers/AgenticEngineering/Introduction/Quiz').
   2026-09-15 18:52:48Z memex-portal-deployment-7fd9668b69-jwzzl fail: MeshWeaver.Blazor.EntityViews.RadioGroupView[0]
         Loading 'answers/agent-chatbot' in area 'Quiz/Options1' in Area Quiz/Options1
         MeshWeaver.Messaging.DeliveryFailureException: Path resolution for 'sglauser/_Answers/AgenticBusiness/01-MeetYourCoworker/Quiz' failed: The operation has timed out.
   
9. rbuergi commented on Sep 15, 2026

   

   rbuergi

   on Sep 15, 2026

   Contributor

   More actions

   The reopen is not a regression: #3536 held, and a DIFFERENT defect is folding into this fingerprint.

   The bot reopened on "still occurring", the quoted sample is from 2026-09-07 (before the fix merged at 06:25Z that day), and the counter did move (474 → 507). All three are true at once, so I read the incident record itself — Admin/_LogIncident/a577a67482869286 on the control instance, which is where these live (it is under a _LogIncident satellite segment, so a content query answers 0 for it even with coverage.partitions: ["admin"] — get by exact path is the only instrument).

   The exception CHANGED

   Of the ten stored samples, nine are the original shape and the newest is not:

   when	pod	exception
   2026-09-07 03:51–03:53Z (×9)	…-58c859d4cc-b8pfg	No node found at '<path>'. Closest ancestor is '<user>' (remainder=…)
   2026-09-15 18:52:48Z	…-7fd9668b69-jwzzl	Path resolution for 'sglauser/_Answers/AgenticBusiness/01-MeetYourCoworker/Quiz' failed: The operation has timed out.

   The absent-node shape this issue was filed for — and that #3536 fixed in MeshNodeBindingExtensions — has not been seen since 2026-09-07 03:53:59Z, i.e. 32 minutes before the fix merged. The one post-fix occurrence is a timeout in path resolution, which is a different failure with a different cause.

   The fix is genuinely deployed, so this is not a delivery question

   https://memex.meshweaver.cloud/api/version reports 3.0.0+c84c6c05…; c84c6c05 is core's 2026-09-12 09:30:22Z. git merge-base --is-ancestor 0ed02bad c84c6c05 → true, so #3536's seam fix is in the running image, and the absent-node shape stopped appearing the day it merged. That is the fix working, measured on the deployed build rather than assumed from the merge.

   Why it looked like a regression

   The fingerprint is computed from the normalised message plus the exception TYPE:

   category:  MeshWeaver.Blazor.EntityViews.RadioGroupView
   message:   Loading '{value}' in area '{value}' in Area {path}
   exception: MeshWeaver.Messaging.DeliveryFailureException
   

   "Node absent" and "path resolution timed out" are both DeliveryFailureException raised from the same Loading … log site, so they normalise to the same fingerprint and fold into one incident — and the stored samples keep the earliest lines, so the reopen comment quotes 09-07 evidence for a 09-15 event. Two distinct defects are therefore indistinguishable from the issue thread alone; that is the mechanism behind this close/reopen loop, not a flapping fix.

   What is actually open

   One occurrence, on one pod, of: per-user _Answers path resolution timing out for a learner opening a quiz. _Answers is not a configured satellite segment, so ordinary routing applies. I am deliberately not guessing the mechanism from a single sample — what would name it is the routing/hub-activation trace for that path at 2026-09-15 18:52:48Z, and whether the storm-breaker was open on it (a breaker opened by the old NotFound behaviour would fast-fail resolution on the same path, which would tie the two together, but that is a hypothesis and not a measurement).

   Suggested disposition: this issue's stated defect is fixed and verified on the deployed build; the remaining fault deserves its own title rather than inheriting one about NotFound-storms. Re-closing it unchanged will simply reopen on the next timeout.

   _{Measured 2026-09-15 by Claude Opus 5.}
10. rbuergi

    changed the title [-]RadioGroupView NotFound-storms when the node it is bound to is missing[/-] [+]Per-user _Answers path resolution times out when a learner opens a quiz (the NotFound half was fixed by #3536; same fingerprint, different cause)[/+] on Sep 15, 2026
11. rbuergi commented on Sep 15, 2026

    

    rbuergi

    on Sep 15, 2026

    Contributor

    More actions

    Closing as a duplicate of #1122 — both halves are now accounted for.

    • The NotFound half this issue was filed for is fixed by fix(#3517): a node-bound binding tolerates an absent node #3536 and verified in the deployed build (c84c6c05 = 2026-09-12; git merge-base --is-ancestor 0ed02bad c84c6c05 is true). It has not appeared in a sample since 2026-09-07 03:53:59Z, 32 minutes before that fix merged.
    • The surviving occurrence — Path resolution for 'sglauser/_Answers/AgenticBusiness/01-MeetYourCoworker/Quiz' failed: The operation has timed out. at 2026-09-15 18:52:48Z — is downstream of Per-node hub DataContext initialization times out at 120s, taking every hub under a user partition dark (root of #3517, #2886, #1313) #1122: five minutes later, on the same pod, DataContext initialization TIMED OUT for that same node path, leaving the hub in FAILED state. The per-node hub never initialises, so every read of the node times out; RadioGroupView is reporting the fault, not causing it.

    Tracking the remaining work on #1122, where the root is. Detail and evidence: #1122 (comment)

    🚨 If the recurrence bot reopens this: the fingerprint here is category + normalised message + exception TYPE, and both causes raise DeliveryFailureException from the same Loading … log site, so they are indistinguishable to it. A reopen is not evidence that #3536 regressed — compare the newest sample's exception text before acting.

    _{Closed 2026-09-15 by Claude Opus 5.}
12. rbuergi

    closed this as not plannedon Sep 15, 2026