UserIdentityCache resolves the User TYPE node instead of a user instance: As<User> gets NodeTypeDefinition — ~600/h errors on prod #2425

Activity

1. rbuergi commented on Aug 26, 2026

   

   rbuergi

   on Aug 26, 2026

   ContributorAuthor

   More actions

   A write-side guard for exactly this landed today — and prod does not have it yet. That is checkable, and it changes what to do next.

   #2378 ("Refuse a NodeType declaration write that claims to be its own instance") merged at 15:03:41Z, commit 7c9a8310. The image both portals are running is 3.0.0-rc8.ci.5670 = 2699e4a05:

   git merge-base --is-ancestor 7c9a8310 2699e4a05  →  false
   

   The running image predates the fix. So the ~600/h you are seeing is the known defect still live in production, not a recurrence and not something the fix failed to catch. That matters because "we fixed it and it is still happening" and "we fixed it and it has not shipped" call for opposite next actions.

   What #2378 does and does not do

   It closes the write side: a create/update whose content IS a NodeTypeDefinition and whose own NodeType names itself is refused, so no future write can re-introduce the collision. Three built-in declarations (User, VUser, Partition) had shipped that way, which is why nodeType:User returned the User declaration alongside real accounts.

   🚨 It does not clean up rows that already carry the collision. Reads stay bad-data tolerant by design — an existing row keeps loading (degraded) until it is next written. So on a portal whose durable User declaration is already self-typed, rolling the image alone may not silence this: the row has to be rewritten too. Worth establishing before declaring victory:

   • Does the running portal's User node still carry nodeType: "User" (rather than NodeType or unset)?
   • If yes, the roll stops NEW collisions and this log line persists until that row is corrected.

   One narrowing that is cheap and worth doing

   The issue asks the right question — "worth knowing what that degrades". As<T> returning null is the fail-closed-and-silent shape: no exception, nothing to grep beyond this line, and the caller proceeds without a user. On the identity path that plausibly means an anonymous fallback. Since AccessService.CircuitContext wiped by an Rx hop → WellKnownUsers.Anonymous is a known production shape here (it presented as "you lack Update permission" for every viewer, admins included), it is worth confirming whether these two meet — i.e. whether a null here lands on the same anonymous fallback.

   Deploy context

   The image carrying 7c9a8310 has not rolled yet; a deploy chain is mid-flight now (manifest fix landed, main green, CD building). When it rolls, the write-side guard arrives. Re-measure the rate after the roll rather than before — and if it does not fall, the durable-row question above is the reason, not a failed fix.

   Also relevant and separately tracked: #2383 fixed _Policy satellites minted with MainNode pointing at themselves, which made them list as content. Different node type, same family of "a declaration/satellite is indistinguishable from an instance to a query".
2. rbuergi commented on Aug 28, 2026

   

   rbuergi

   on Aug 28, 2026

   ContributorAuthor

   More actions

   Folding in #2506 (closed as a duplicate of this issue): the same error was still firing at 599 occurrences/4 h on memex.systemorph.com and 69/4 h on memex.meshweaver.cloud, on image gen ci.6096 — which post-dates #2378's merge. That matches the prediction in the comment above exactly: the write-side guard stops re-introduction, but the already-self-typed durable declarations (User, VUser, Partition) keep loading degraded until they are rewritten.

   Remaining work, precisely: clean up the existing self-typed NodeTypeDefinition rows on both portals (rewrite or repair-migrate them), then confirm the UserIdentityCache error rate drops to zero on an image ≥ #2378.
3. rbuergi commented on Aug 28, 2026

   

   rbuergi

   on Aug 28, 2026

   ContributorAuthor

   More actions

   Still live on memex-cloud, and the culprit is exactly one row — measured 2026-08-28.

   The index is built from MeshQueryRequest.FromQuery("nodeType:User") (UserIdentityCache.cs:222). Running that query against the live mesh returns two nodes:

   path	name	nodeType	version	lastModified
   User	User	User	2	2026-06-02
   rbuergi	Roland Buergi	User	7	2026-08-27

   The first is the User type declaration, self-typed: its nodeType is User rather than NodeType. So the identity index does not merely resolve "a wrong node" in some general sense — it ingests the type declaration itself, whose content is a NodeTypeDefinition, and As<User> on it fails exactly as reported.

   This matches the prediction recorded when #2506 was closed as a duplicate of this issue: the write-side guard (#2378) stops NEW self-typed rows, but existing durable rows keep loading degraded until rewritten. That row's lastModified is 2026-06-02 — it predates the guard by nearly three months and has not been touched since, which is why the errors continue.

   What the repair is

   One field on one node: User.nodeType should be NodeType, not User.

   I have deliberately not made that change. It is production data, and retyping a type-declaration node is not obviously side-effect-free — the same query that feeds this index is what makes the node visible to type resolution, so the repair wants either a migration with a rollback story or an explicit decision from whoever owns the partition. Happy to do it on request.

   Worth checking beyond this mesh

   I measured memex-cloud only (the mesh this MCP is connected to). The same row will exist independently on every other install whose User declaration was written before #2378, so a fix that only repairs one mesh leaves the rest reporting the same ~600/h. That argues for the migration shape rather than a manual edit.
4. rbuergi

   closed this as completedin #2605on Aug 28, 2026