A write into a Store-plugin partition false-passes: acknowledged, versioned, invisible — AgenticEngineering on systemorph has 371 durable nodes nobody can read #2899

Activity

1. rbuergi commented on Sep 1, 2026

   

   rbuergi

   on Sep 1, 2026

   ContributorAuthor

   More actions

   Re-measured on systemorph 2026-09-01 (~14:0xZ) — unchanged, and the stated root cause is falsified

   State is exactly as reported, 4 days on

   Seam	Query	Answer
   query index	search namespace:AgenticEngineering scope:descendants	2 — _GitSync (v1), _Policy (v1), both 2026-08-28
   children listing	get @AgenticEngineering/*	the same 2
   point read	get @AgenticEngineering/Introduction	Not found
   version list	get_versions AgenticEngineering/Introduction	v1, 2026-08-28T22:40:38Z, system-security, Edu/Lesson
   version content	get_version AgenticEngineering/Introduction 1	the complete node, state: "Active", full MarkdownContent
   the 08-31 probe	get_versions AgenticEngineering/WriteLaneProbe	v1, 2026-08-31T09:24:10Z, rbuergi, Markdown

   Two things ruled out by measurement

   1. It is NOT the two-read-seams-disagree defect (#2939 / #2970 / Plugins#1053). Both read seams agree the node is absent; it is the version store that disagrees with both. That had to be checked — the disagreeing-seams class is live and confirmed, and it produces a superficially similar "the node is there but you can't see it".

   2. It is NOT install-materialization shadowing a Store-plugin partition. This is the important one, and it costs the issue its stated root cause.

   The identical outcome exists on memex.meshweaver.cloud, in sglauser — a plain user partition. No Store/Plugin root, no installPaths, no install of any kind, nothing to shadow the durable layer with:

   • get_versions sglauser/MeshWeaverInstance/sglauser-local-3 → v1, 2026-08-31T15:44:49Z, system-security, MeshWeaverInstance
   • get @sglauser/MeshWeaverInstance/sglauser-local-3 → Not found
   • search namespace:sglauser scope:descendants → 3 nodes, all Markdown from April; search nodeType:MeshWeaverInstance → 22, every one under rbuergi/…
   • the sibling write of the same operation, Admin/_PluginGrant/sglauser-local-3 (3 s later, same system-security identity, different partition) → readable, v1, Active

   That is #2915: the registry authenticator cannot read that instance node, so instance-key resolution answers 503 forever with "This is NOT a statement about your key or your grant" — which is true about the key and the grant, and wrong about the word temporarily.

   Also ruled out

   • Not identity — both system-security and an interactive user (rbuergi) produced it, and the same system-security write into Admin in the same operation landed visibly.
   • Not "everything written that week" — rbuergi/MeshWeaverInstance/ci-crm, written 2026-08-28, reads back fine.

   The partition-index half of point 6 reproduces on memex too: autocomplete @/sglauser → 0 results while search namespace:sglauser scope:descendants returns its content. autocomplete @/rbuergi → the partition is listed as rbuergi@systemorph.com.

   What this changes about the asks

   Ask 1 ("a write into a shadowed/half-installed partition must FAIL") is right in spirit but scoped too narrowly — it is not shadowing and not install state. The rule that covers both instances is:

   An acknowledgement is only worth what the layer that reads can confirm. Any write that mints a durable identity someone will later resolve must read the node back through the seam the resolver will use, and fail loudly when that read comes back empty.

   Asks 2 (manifest must record only read-back-visible writes) and 3 (Succeeded + ImportedWithErrors must be impossible) stand unchanged and are independently actionable.

   Not repaired, deliberately

   The root question — why the node row is unreachable while its version row is not — needs the partition schemas inspected on the portal, which the MCP surface cannot do. Do not repair by restoring versions until the write lane is fixed: a restore takes the same write path and can land the same way. Both instances are still live and reproduce on demand, so the evidence has not decayed.

   This measurement is committed as Doc/Architecture/DurableButUnreadable (PR linked below) — the three-seam diagnostic, the two instances, and what each rules out.

   Nothing was mutated. All queries above are read-only.
2. rbuergi commented on Sep 1, 2026

   

   rbuergi

   on Sep 1, 2026

   ContributorAuthor

   More actions

   Cross-linking two findings from today, because they are about the same seam

   Recording a connection between this issue's measurement and one made independently a few hours earlier, since they constrain each other.

   This issue's finding (measured on both portals, now in Doc/Architecture/DurableButUnreadable): a node's version row is readable while the node row is not, on two unrelated partitions, with both read seams agreeing. sglauser is a plain user partition with no Store/Plugin and no install — which falsifies the plugin-install-materialization root cause this issue originally proposed.

   The independent finding, from a one-field patch on Provider/_Policy (memex, 12:27:54Z, recorded on #2896): the write surface reported

   the patch ... did not land within the confirmation window — ... retry.
   

   …and the index then showed the node had gone version 1 → 2. The write had landed. Point reads on that node answered Unavailable … no verdict within 10s for several minutes afterwards, while search answered instantly.

   Why they belong together. Both are the same asymmetry: the version/index side of a write completes while the node side does not answer. One shows it at write time (a landed write reported as failed); the other shows it at read time (a durable version whose node row is unreachable). If the write lane can acknowledge a version without the node row becoming readable, then "durable but unreadable" is what that state looks like afterwards — which would make these one defect observed at two moments rather than two defects.

   That is a hypothesis, not a conclusion — I have not shown that the same mechanism produced both, and the two occurred on different nodes and different partitions. But it is cheap to test and it changes where to look: instrument the write path's two-phase completion rather than hunting the read path for why a row is missing.

   It also reinforces the standing warning on this issue: do not repair by restoring versions until the write lane is understood. A restore takes the same path that produced the state, and on the evidence above that path can report failure while succeeding — so a "failed" restore may well have landed, and a retried one may double-write.
3. rbuergi commented on Sep 1, 2026

   

   rbuergi

   on Sep 1, 2026

   ContributorAuthor

   More actions

   Measurement: this is NOT the same mechanism as #2897 — and the discriminating read is one step

   Investigated together with #2897 and #2885 on the hypothesis that all three were one mechanism (the
   partition's payload outgrew the transport, so the delivery carrying the write could never land).
   That hypothesis is falsified for this issue. #2897/#2885 are one mechanism; #2899 is a different
   one that happened to be measured in the same window.

   The decisive measurement (systemorph, read-only, 2026-09-01)

   probe	answer
   get_versions AgenticEngineering/Introduction	v1, 2026-08-28T22:40:38Z, system-security, Edu/Lesson
   get @AgenticEngineering/Introduction	Not found: AgenticEngineering/Introduction
   search namespace:AgenticEngineering scope:descendants	2 results — _GitSync (22:40:22), _Policy (22:40:31)

   The two surfaces read different tables. get_versions reads the node history, written by a
   database trigger on the row itself. get and every scope:children listing read the
   current-state row through the live query path.

   So the accurate statement of the symptom is not "371 durable nodes nobody can read" but:

   a history row exists and the current-state read says not found. The write committed and the
   running mesh was never told.

   What that rules out

   • The A node whose NodeType has not compiled yet stays untyped FOREVER — the missed message behind the Pandas flake #2952/An instance whose NodeType has not been KICKED OFF yet binds the default config permanently — the area-not-found half of #2952 #3006 untyped-content degrade — RULED OUT. That degrade leaves the node in the
     listing with unusable content (view renders empty). It cannot remove a node from a children
     listing, and it cannot make a point read answer Not found. The 6 NodeType 'Systemorph/Event' is not registered failures are a real, separate problem (they are the 6 that did report failure),
     but they are not what hides the other 365.
   • The Message delivery to the AgenticEngineering message hub exceeds the Orleans 100 MB MaxMessageBodySize and cannot be delivered #2897 transport bound as the CAUSE — RULED OUT by dates. The invisible writes are dated
     2026-08-28 22:40. Message delivery to the AgenticEngineering message hub exceeds the Orleans 100 MB MaxMessageBodySize and cannot be delivered #2897's five 142 MB frames are 2026-08-31 09:06 — three days later, and
     inside the remediation attempt, not the original write. The oversized frame is a consequence of
     re-importing this course, not the reason its first import went invisible.
   • "Recycling the hub should have fixed it" — consistent. The state is durable, not
     per-activation, so a rebuilt hub re-derives the same missing announcement. Recycling changing
     nothing is evidence for this mechanism, not against it.

   The mechanism, and where it lives

   The durable write and the mesh-wide announcement are two separate steps. The row goes to
   storage; IMeshChangeFeed separately tells the running mesh the path exists. Everything that
   decides reachability keys off the announcement, not the row:

   • PathResolutionService caches path resolution — a path cached as a miss stays a miss, so
     routing answers No node found at '…' for the life of the process.
   • A live scope:children listing runs its SQL once, at Initial, and re-queries only on a
     change notification that passes the relevance gate; consumers share that snapshot via
     Replay(1). One lost notification freezes the listing permanently — which is exactly why a point
     read and a listing can disagree.

   MeshNodeTypeSource's class documentation already names this the #817/#824 announce-loss class.

   Ranked candidates, all code-cited, for where the announcement was lost here. The storage backend
   is in MeshWeaver.Plugins, so most of these are fixed there, not in core:

   1. public.top_level_index is never rebuilt on partition CREATE. It is a materialized view.
      Its only rebuild call sites are the run-once migration Job and DeletePartition.
      EnsureSchemaAsync registers the new schema in searchable_schemas and stops. This is
      item 6 of this issue, exactly — every partition created after the last migration is durable and
      permanently absent from @/ autocomplete. The comment at
      PostgreSqlCrossSchemaQueryProvider.cs claiming it is rebuilt on create is false in the code.
   2. n.state != 2. Every query hard-filters to Active. A row persisted at Transient/New is
      versioned, historied, acknowledged — and invisible to every query, with no log line.
   3. n.main_node != n.path. Note Introduction v1 carries mainNode: AgenticEngineering/Introduction
      while _Policy carries mainNode: AgenticEngineering; any is:main listing filters on this.
   4. Case-sensitive children scope. GenerateChildrenClause emits n.namespace = @scopeNs — the
      only text equality in that generator that is not case-folded, while schema routing lowercases
      and PathMatcher is OrdinalIgnoreCase throughout.
   5. The pg_notify trigger's dedup suppresses NOTIFY entirely for updates that touch only
      description/category/icon/display_order/… while the history trigger still writes a row —
      literally "durably versioned, no announcement".
   6. Best-effort announcement hops: the Orleans cross-silo change-feed broadcast logs-and-skips a
      failed broadcast; the cross-process pg_notify route silently increments a discard counter.

   A separate defect found on the way, worth its own issue: the two ExcludedSchemas lists are
   inverted on agent and auth — the runtime list excludes auth and deliberately keeps
   agent; the migration's list does the opposite. Every deploy wipes the registry and repopulates it
   in the state each side documents as a bug, until the runtime's next 30 s sync corrects it.

   Confirmed as stated in this issue

   The two asks that need no further diagnosis are confirmed by this measurement and should stand:

   • an import manifest must record per-file success on a read-back-visible write, never on the
     acknowledgement — otherwise a false-passing write latches the manifest and force can never
     retry it (this is what makes the loss permanent and invisible to the repair tool);
   • Succeeded + ImportedWithErrors in one wrapper activity must be impossible.

   Conserved

   The full field guide — the discriminating measurement, both mechanisms, and the rules that follow —
   is committed as a doc page rather than left in this thread:
   Doc/Architecture/AcknowledgedButInvisibleWrites (PR linked below).

   This issue stays open. The core-side half (the transport bound, #2897) is fixed in that PR; the
   announce-loss half diagnosed here is not, and its fix belongs in the storage backend.
4. rbuergi commented on Sep 2, 2026

   

   rbuergi

   on Sep 2, 2026

   ContributorAuthor

   More actions

   Re-measured 2026-09-02 (read-only): the write lane is not the defect

   The decisive probe. autocomplete @/AgenticEngineering/Introduction returns the node and six
   of its children, with correct names and node types (Edu/Lesson, Edu/Quiz, Edu/Page,
   Edu/CourseInvite, Markdown). A row no reader can see cannot be returned by autocomplete, so the
   371 nodes are durably present and reachable. The write landed. The read seam is what fails.
   @/AgenticEngineering/ likewise lists TDD, Slides, RiCube, Trainees, Training,
   DemoAgent, Exercises, Slides/S01…S39.

   Two hypotheses died on evidence that could have gone the other way — recording them so nobody
   spends the day again:

   Probe	Answer	Kills
   get_version AgenticEngineering/WriteLaneProbe 1	mainNode == path, node invisible	the is:main / n.main_node = n.path filter (#2939)
   get_version AgenticEngineering/_Policy 1	mainNode != path, node visible	…and inverts that correlation, so it cannot be the cause
   search namespace:AgenticPrimer scope:descendants	full content, same portal, byte-identical _Access shape (Public — Viewer + Anonymous — Viewer, nothing else)	the grant shape, the install date, a portal-wide query regression

   Also, contra the issue text and the doc: _GitSync and _Policy are not satellite-routed.
   SatelliteTableMapping.IsSatellitePath matches the exact segment list, which does not include
   them — they are ordinary mesh_nodes rows in the same table as the 371 invisible ones. So a
   table-routing split cannot explain the split in visibility.

   What survives, 5 of 5 cases including one that could have falsified it: a row in that partition
   passes the read filter iff its main_node is the partition root AgenticEngineering — which is
   exactly the prefix its two grants project at (COALESCE(main_node, namespace) in
   rebuild_user_effective_permissions) and exactly the column the per-schema access clause folds the
   caller's effective permissions against (n.main_node, not n.path). Settling it needs three
   columns read on that portal, against agenticprimer as the working control:

   SELECT id, path, main_node, state FROM agenticengineering.mesh_nodes ORDER BY last_modified;
   SELECT * FROM public.partition_access WHERE partition IN ('agenticengineering','agenticprimer');
   SELECT count(*) FROM agenticengineering.user_effective_permissions;

   Hypotheses (a) and (b) are both ruled out. (a) transport/#2897: a refusal is loud and terminal
   and leaves no version row, and the frames post-date the import by three days; #3018 is merged and
   the symptom is unchanged. (b) the untyped-content degrade (#2952/#3006): a degrade leaves the node
   in the listing with unusable content, and get_version returns fully-typed content here.

   One producer of this class was in core and is fixed — #3035

   StaticNodeStorageAdapter was the one of six serve seams that did not honour
   MeshNode.IsDefinitionOnly, and it is the seam PersistenceService reads. Because the static
   provider is fixed-namespace (first provider band) and read-only (absent from the write chain),
   Read and Write disagreed about which provider owns the path and only the write side was ever
   asked: acknowledged, version row written, unreadable forever. #3035 fixes it and adds
   AcknowledgedWriteIsReadableTest, which pins the invariant the platform had no test for — a write
   the storage layer acknowledged must be readable back through the seam readers use — and fails
   against the pre-fix code with this exact signature.

   #3035 deliberately does not close this issue. It is a different producer from the one live here,
   and closing would bury 371 still-unreadable nodes.

   Full RCA is committed at Doc/Architecture/DurableButUnreadable (this repo,
   src/MeshWeaver.Documentation/Data/Architecture/DurableButUnreadable.md) — the core mechanism, the
   falsifications above, the resolution of all three hypotheses, and the named residual.

   Still open here, untouched: repairing the partition, and the GitSync wrapper activity reporting
   Succeeded over a child that ended Warning. On the latter — the sibling half is already fixed
   (GitHubSyncService.MayAdvanceBaseline, #2229 item C); the status fold was left alone deliberately
   because Finish(v, override) is MAX(floor, fold) with dual writers that split-brain, and an
   unstudied change to terminal-status semantics inside a gate is worse than a named open item.
5. rbuergi

   added a commit that references this issue on Sep 2, 2026

   test(#2899): seal the awaits with .Await()

   ...

   5e67ef0
6. rbuergi

   closed this as completedin #3035on Sep 2, 2026
7. rbuergi

   added a commit that references this issue on Sep 2, 2026

   Merge pull request #3035 from Systemorph/fix/2899-write-lane-false-pass

   ...

   a866cf1