Write lane degraded on memex-cloud: create chain dies on a NotFound stream, a finished Store/Provision storms compile-state, plus a permanent-silent import order bug and a 2/s identity-cache loop

Four defects found while investigating a write-lane degradation on memex-cloud (2026-08-25, ~14:45–15:05 UTC). Reads and renders were healthy throughout; writes failed. Filed together because they were found in one incident, but they are independent and separately fixable — A and B are the outage, C and D are silent bugs the incident exposed.

Ruled out first, with evidence, so nobody re-treads it: this was not the fattened/draining pods. Both were wedged in Terminating (15 min past a 90 s app drain; terminationGracePeriodSeconds=1800, so they hold activations for half an hour) and were force-deleted — writes failed identically afterwards. It was also not the freshly-armed Modules:AutoRecycleOnStaleBuild (disproved separately on #2194).

A. `CreateNodeRequest` never replies: the create chain subscribes to the node's stream before the node exists

The primary user-visible failure. Three independent reproductions (create rbuergi/WriteProbe2224, different MCP session hubs), each a 60 s timeout, and the node genuinely absent afterwards. The message-flow trace names the whole chain:

AWAITING CreateNodeRequest → POSTED → RECEIVED runLevel=Started → ENQUEUED
→ ROUTED onTarget=True → HANDLER_ENTER
→ CREATE_CHAIN_SUBSCRIBED path=rbuergi/WriteProbe2224
→ HANDLER_EXIT state=Processed
   … a handler was entered and no reply, completion or fault has been recorded since …
[ROUTE] NotFound: No node found at 'rbuergi/WriteProbe2224'. Closest ancestor is 'rbuergi' (remainder='WriteProbe2224').
[SYNC_STREAM] OnError for 6LGxECQAzE2EKtMwdHqhOA (Reference=MeshNodeReference { Path =  }, Owner=rbuergi/WriteProbe2224)
MeshWeaver.Messaging.DeliveryFailureException: No node found at 'rbuergi/WriteProbe2224'.
System.TimeoutException: No response received in hub portal/mcp-rbuergi-04ecb166 within 00:01:00 for request CreateNodeRequest

The chain subscribes to the new node's sync stream, the route answers NotFound because it has not been persisted yet, the stream terminates with OnError, and the chain dies inside code that records no terminal stage — so the caller waits the full timeout and the create never lands. This is the "initial state never arrives" shape: the cure must be that the create chain does not depend on a stream that cannot exist yet (or tolerates absence until the persist lands), not a retry/watchdog around it.

[STALE-CALLBACK] diagnostics already name the gap precisely: "the work that owes the reply is either STILL RUNNING or terminated inside code that records no stage. Add hub.NoteRequestStage(request.Id, …) at that handler's terminal arms to split the two." Worth doing regardless — it is the difference between "hung" and "faulted" in every future incident.

B. A finished `Store/Provision` keeps re-racing its own compile-state node — a write storm that starves the fleet

Measured on the two serving pods: 39 and 95 MonotonicWriteGuard CONFLICTs per 2 minutes, versions racing 1277→1282 on Store/Provision/_Activity/compile-state, every loser rebased and retried. The guard's own log states the defect: "Find the writer that adopted a stale own-node snapshot; do not relax this guard." The provisioning request had already reached state: Done (7/7 phases) with requestedAction cleared — so a finished request was still driving compiles.

Collateral in the same window, on unrelated partitions:

[DeleteNode] timeout path=BusinessRules/Scope/_Access/Anonymous_Access stage=commit partial-deleted=0
System.TimeoutException: [DeleteNode:commit] the bottom-up delete … did not drain within 30s
[UpdateRemote] LATE_NACK_TERMINAL hub=cache/… target=BusinessRules/_Policy code=Unknown attempt=0
   msg=TimeoutException … the optimistically-acked write did NOT apply and is not auto-retryable

Recycling the request instance and the Store/Provision NodeType hub halved the rate (39→21, 95→46), which confirms the writer as the source but not the mechanism. Possible fuel worth checking in that order: an explicit compile of a type in the package, and two git_hub_sync update triggers, both issued after the provision reported Done.

C. Provisioning order makes an import failure PERMANENT and silent

A package repo that ships an instance of its own new type fails on import — the instance is imported in phase 4, its NodeType compiles in phase 7:

NodeType 'MyAi/Panel' is not registered

and the sync pointer advances anyway, so every subsequent retry is a no-op and the failure is permanent until the repo changes. That combination — wrong order, plus a pointer that advances past a failed item — will bite every package whose repo contains an instance of a type it introduces. Either the phases must order type-compile before instance-import, or a failed item must hold the pointer.

D. `UserIdentityCache` hot loop: 2/second, on every serving pod

fail: MeshWeaver.Blazor.Infrastructure.UserIdentityCache[0]
      As<User> for User: value is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible

360 occurrences per 3 minutes on each of the serving pods (one pod had 0, so it is triggered by something request-shaped, not by startup). The cache asks for a User and gets the NodeType definition named "User" — a path/identity collision between the type node and a user instance. .As<T>() is doing its job (returning null and saying why, rather than a silent miss), so the bug is the caller's resolution, and the loop means nothing backs off. Independent of A–C and cheap to fix.

Related: #2194 (same-day memory RCA — the flag was tested and cleared there), #2221/#2224 (registry static assets), the _App migration and MyAi provisioning that ran in this window.

Item B follow-up: recycling does not clear the writer — the storm regenerates.

Measured across the recycle of both the request instance (Admin/Provision/MyAi) and the Store/Provision NodeType hub:

time (UTC)	pod 72fmv	pod knbcm
14:58 (before recycles)	39 / 2 min	95 / 2 min
15:02 (just after)	21 / 2 min	46 / 2 min
15:05	44 / 2 min	52 / 2 min

So the dip immediately after the recycles was transient and the rate returned. Recycling the owning hub is therefore not a remedy, and the earlier reading of "halved" should not be taken as progress — whatever adopts the stale own-node snapshot re-establishes itself after the hub comes back. That points away from per-hub in-memory state and toward something re-driven from durable state (a request whose terminal write did not actually stop the driver, or a watcher re-arming from the persisted activity), which narrows where to look.

Nothing has fired a new provision since 14:45 and none is planned, so the ongoing rate is not new work — it is the same finished request still driving.

Item D rate for the record: the UserIdentityCache loop is steady at 383 per 2 minutes (~3.2/s) on both serving pods, unchanged across the same window — independent of B, as expected.

🚨 Correcting item B — I filed it wrong, twice over

Two new pieces of evidence, and both cut against my own framing.

1. The conflicts are not Store/Provision's, and never were. Grouping the last 4 minutes by node:

11 CONFLICT on SocialMedia/PostsHub/_Activity/compile-state
10 CONFLICT on SocialMedia/Profile/_Activity/compile-state
 9 CONFLICT on X/Dashboard/_Activity/compile-state
 8 CONFLICT on YouTube/Dashboard/_Activity/compile-state
 7 CONFLICT on SocialMedia/Post/_Activity/compile-state
 6 CONFLICT on Marketing/EventHub/_Activity/compile-state
 6 CONFLICT on Marketing/Campaign/_Activity/compile-state
 6 CONFLICT on LinkedIn/TileMigration/_Activity/compile-state
 …

Ten-plus types across five partitions. Independently corroborated: the identical pattern was observed at 09:15–09:30Z on Store/Catalog/_Activity/compile-state (versions 1650→1651, 1653→1654, 1657→1658, 1661→1662), 5.5 hours before the MyAi provision existed. So "a finished request that won't stop" is simply wrong — the rate tracks compile traffic, on whichever types are compiling.

2. More importantly: this race is documented, deliberate, and its loser is correct by design. NodeTypeCompilation.md → "What the claim does NOT fix":

NodeTypeBatchBake.WriteStamp is a read-modify-write at NextVersion, and the monotonic write guard bounces the loser … the other [writer] is the type's OWN per-node hub, which stamps compile state from the activation path, the sources watcher and the release watcher, and is legitimately live while a batch bake runs. The loser's outcome is already correct: the bytes are durable and content-addressed, the record stays pending, and the level-triggered probe … re-bakes and re-stamps on the next pass.

So the MonotonicWriteGuard CONFLICT lines are expected behaviour under concurrent compiles, not a defect, and my "write storm that starves the fleet" claim was not established — the write-lane failure is fully explained by item A alone, which reproduces with no provision running and no compile in flight.

What genuinely remains open from item B, and it is narrower:

[DeleteNode:commit] … did not drain within 30s on _Access nodes across unrelated partitions.
[UpdateRemote] LATE_NACK_TERMINAL … target=BusinessRules/_Policy … the optimistically-acked write did NOT apply and is not auto-retryable — an acknowledged write that silently did not apply is a correctness problem regardless of what else is true.
Whether the conflict VOLUME is proportionate to the compile traffic, or whether the level-triggered re-bake is doing more passes than it should. That is a cost question, not a correctness one.

Items A, C and D stand unchanged. Anyone acting on the original B framing — in particular pausing provisioning to "stop the storm" — should stop doing so: it will not change these numbers, and no provision has run since 14:45.

🚨 Item A upgraded — it is NOT "the create is lost". The create SUCCEEDS and the REPLY is lost.

This inverts the correct client behaviour, so it matters more than the original filing.

Evidence, three nodes, two independent investigators

node	client saw	reality
`rbuergi/WriteProbe2224` (mine)	3 × `CreateNodeRequest` 60 s timeout; a `get` right after said Not found	Exists, `createdDate 14:57:05`, `version 1`, content byte-identical to attempt 2
`MyAi/Settings`	3 × timeout, "absent afterwards"	Exists, `createdDate 14:54:36`, `version 1`, content as sent
`Admin/Migrations/AppTiles2`	"timed out after 60 s, absent afterwards"	Exists and EXECUTING — created 14:47:49, `state: Running`, log `["[start] RefreshAppTiles"]`, still running 25 min later

Consequences — "lost create" and "lost reply" demand opposite client behaviour

Retrying on this timeout is UNSAFE. It re-executes work that already ran. This nearly happened twice today: Admin/Migrations/AppTiles (Done 14:02) and AppTiles2 (Running) are two mesh-wide sweeps armed 40 seconds apart, the second only because the first "failed".
A control-plane request can be silently in flight while its requester believes it never started. That produced two wrong diagnoses in one afternoon — "the watcher never fires" and "writes are being dropped". The true symptom is: the response path is lost; the work is not.
The missing hub.NoteRequestStage(request.Id, …) at the handler's terminal arms is exactly what makes "never started" indistinguishable from "started, reply lost". That instrumentation is not a nicety — it is what lets a client decide whether retrying is safe.

And reads are unstable in BOTH directions — I asserted otherwise earlier and was wrong

I repeatedly reported "reads and renders are fine". They are not, in this window:

False negative: get rbuergi/WriteProbe2224 returned Not found at ~15:03 for a node created at 14:57. The route agreed — [ROUTE] NotFound: No node found at 'rbuergi/WriteProbe2224' logged at 15:02, five minutes after the node existed.
False positive: immediately after a delete that returned Deleted, a get still returned the node as state: Active (with a lastModified I never wrote); a second get seconds later returned Not found. So the delete had applied and the read was serving a superseded version.

That read instability is very likely the same fault as the create chain's: CREATE_CHAIN_SUBSCRIBED → the subscription's own read of the just-written node answers NotFound → [SYNC_STREAM] OnError → the chain terminates with no stage recorded → no reply. So the refined mechanism is a read-after-write visibility failure inside the routing/stream layer, not "subscribed before the persist" — the persist has happened; the reader cannot see it.

What this changes for anyone acting on this issue

Do not retry a timed-out create/control-plane request — check whether the node exists first.
Do not trust a single Not found during an incident; re-read before concluding absence.
Admin/Migrations/AppTiles2 has been Running for ~25 minutes with only [start] logged — either a long walk or a wedge; it is being left alone rather than re-armed, which is the right call given the above.

Read side, discriminated: the two read routes fail differently, and only one of them is a defect

Independent evidence (peer session, same session, seconds apart) separates them cleanly:

search path:rbuergi/_App scope:children name:*LinkedIn* → 1 result (only a stale 08-24 tile); two tiles minted at 14:44/14:45 absent.
get @rbuergi/_App/LinkedIn-RolandLinkedIn-LinkedIn → present, v2, correct content. Same for the second tile.

So:

route	behaviour	status
`search` / query	trails the durable store by minutes	documented and expected — the query index is eventually consistent (AGENTS.md → "CQRS — Never Query for a Single Node's Content"; CqrsAndContentAccess.md)
direct-path `get`	answered Not found five minutes after creation, with the route logging `[ROUTE] NotFound`	the defect — `get` is what every caller treats as ground truth

That narrows item A's read-side symptom to the routing/stream layer and removes the query lag from the bug's scope entirely — a distinction I did not draw when I first wrote "reads are unreliable", and the sharper statement is: the authoritative read route intermittently lies; the lagging route was always documented as lagging.

Practical rule (please apply it rather than distrusting reads wholesale): verify existence with a direct-path get, never with search; and if a get says Not found during an incident, confirm with a second read before acting.

🚨 A documentation/behaviour mismatch worth its own fix

AGENTS.md currently sanctions the unsafe pattern outright:

Valid query uses: listing children (path/*), searching by predicate, existence checks, autocomplete.

But an existence check answered from a lagging index is exactly how duplicate work gets armed: install-record checks, tile-mint duplicate guards, and the standard-packs AlreadyInstalled probe can each read "absent" for something that already exists and act on it. That is a plausible mechanical source of the duplicate mints and re-installs seen under load today — and it is doubly serious given the item-A finding above, since a caller whose create "timed out" and whose query then says "absent" has two independent reasons to wrongly conclude nothing happened, and to retry work that already ran.

Proposed: strike "existence checks" from the valid-query list in AGENTS.md and CqrsAndContentAccess.md, replacing it with "existence checks for a specific path → GetMeshNodeStream(path) / direct get; a query may only answer existence questions where a stale negative is harmless (autocomplete, listings)". Any existing guard that decides create-or-skip from a query should be treated as suspect.

Blast radius audited — the cost shape is duplicated WORK, not duplicated DATA. Keeping the severity honest, since "plausible source of duplicate mints" deserved checking rather than repeating.

The plugin-side existence checks that read through a query are all path-deterministic and idempotent, so a stale negative costs a redundant write rather than a second node:

AppTileRefresh.Missing decides "no record" from an _App children snapshot, then writes via CreateOrUpdateNode at the deterministic {viewer}/_App/{packageId} — a stale negative re-writes the same path.
StandardPacks.AlreadyInstalled reads the manifest to decide whether to install; the installer's CopyPlan skips by PATH, so a stale "not installed" re-runs an install that lands the same copies (that tolerance is deliberate — it is what makes re-install non-destructive to learner work).

So nobody needs to hunt for corrupted tile records. The real cost is the one already visible: two mesh-wide sweeps armed 40 seconds apart — expensive and confusing, not corrupting.

The dangerous variant, for anyone auditing further: a query-driven existence check whose write target id is minted per attempt rather than derived from the path. That shape turns a stale negative into genuine duplicate data. None found in the plugin repos; worth a look wherever a guard and its write don't share a deterministic key.

Doc fix in flight (core's AGENTS.md line plus a new section in CqrsAndContentAccess.md) with this severity split written into it, so the guidance says which case is merely wasteful and which is corrupting.

Severity note on item A's read side: the false negative is FREQUENT, and it has now corrupted two investigations

Three confirmed occurrences in one afternoon, on three different paths, by two different people:

#	path	`get` said	reality
1	`rbuergi/WriteProbe2224`	Not found	existed, v1, created 14:57:05
2	`MyAi/Settings`	Not found (×3)	existed, v1, created 14:54:36
3	`DefaultViews/Examples`	Not found	existed, created 14:14:38

That is not an oddity to note — it is a routine outcome, and it has already produced two wrong conclusions that were published before being caught:

"Writes are being dropped" (mine) — a create that had in fact succeeded, because the reply was lost and the confirming read returned a false negative. Two independent reasons to believe the same false thing.
"The stalled roll starved the catalog path of package content" (recorded on A landed module built against a newer platform aborted every new pod on the registry: optional-parameter ctor change is binary-breaking and raised no minMeshVersion floor #2234, now struck) — five packages reported missing that had landed hours earlier. The defect fooled the person documenting the defect.

The practical rule needs strengthening, because the earlier version was insufficient. "Verify by get, not search" is necessary but not enough: all three cases above WERE direct-path gets. The honest rule is:

Verify by get — and confirm a negative with a second read before acting on it, reporting it, or building an argument on it. A single Not found is not evidence of absence.

That applies to code as much as to humans: any guard that treats one NotFound as "does not exist" and then writes is exposed, which is the same hazard as the query-index one but on the route everybody trusts.

Given the frequency and the demonstrated capacity to derail diagnosis, item A's read side deserves the same priority as its write side rather than being read as a footnote to it.

Item A also fires on `memex` (namespace `memex` / memex.systemorph.com), not only memex-cloud

Found while triaging #2165–#2167, which turn out to be a symptom of this. The signature is item
A's, verbatim, on a different partition and a different node type — so it is not memex-cloud-specific
and not _App-specific.

Loki, pods …-z2xcv / …-97pjf, 2026-08-24. The incident node
Admin/_LogIncident/6e643370636ed34a was created at 17:54:18 and was at durable Version=3:

17:56:47  CreateNodeRequest → AWAITING …                                    (re-polled every 5 s)
17:57:16  [CreateOrUpdate] inner CreateNode faulted for Admin/_LogIncident/6e643370636ed34a
17:57:16  [ROUTE] NotFound: No node found at 'Admin/_LogIncident/6e643370636ed34a'. Closest ancestor is 'Admin'
17:57:16  [SYNC_STREAM] OnError … Owner=Admin/_LogIncident/6e643370636ed34a
17:57:48  CreateNodeRequest → AWAITING …                                    (again)
17:58:16  [CreateOrUpdate] inner CreateNode faulted …
17:58:49  Node created at Admin/_LogIncident/6e643370636ed34a by system-security
17:58:49  [UpdateRemote] ERROR hub=cache/… target=Admin/_LogIncident/6e643370636ed34a type=DeliveryFailureException
17:58:49  [UpdateQueue] FAILED path=Admin/_LogIncident/6e643370636ed34a seq=959 elapsedMs=0.71
17:58:56  [PostgreSqlStorageAdapter] write … at Version=1 was REFUSED; the durable row is at Version=3
17:58:56  [MonotonicWriteGuard] CONFLICT on Admin/_LogIncident/6e643370636ed34a

Two things worth adding to item A from this:

The false negative reaches persistence.Read, not only the route. HandleCreateOrUpdate
dispatched its inner create — which it only does when its direct persistence read returns null —
for a node that had been durable for 2.5 minutes. So the visibility failure is not confined to the
routing/stream layer.
MonotonicWriteGuard refuses the resulting Version=1 write, and the create still reports
success (Node created at … follows the CONFLICT). So a caller cannot tell a create that landed
from one the guard discarded — which is the same "the reply does not describe what happened"
property item A already names, at the other end of the write.

Also confirms item D independently: 6e643370636ed34a is the UserIdentityCache /
As<User> … NodeTypeDefinition incident, ticketed as #2163.

Unrelated but adjacent: the create-vs-fold decision in LogIncidentIngestService was inferred from a
point-read and treated a NotFound as proof of absence — exactly the inference this issue says is
unsafe. On the path where the route fails but persistence can see the node, that sent a
first-sighting payload through CreateOrUpdateNodeRequest's content-replacing merge and wiped the
live incident's draft, issue link and occurrence count. Fixed in
Systemorph/MeshWeaver.Plugins#663 (create-only, then fold — the store decides). That is a separate
defect; it does not address this one.

Review: target the exact grant, snapshot the failure list, bound the …

Status after #2267 — two of the four items are done, and item A has a named root cause with a live reproduction

Leaving this open deliberately. Here is what is closed and what is not.

Item A — one concrete root cause fixed, the item is NOT closed

PackageInstaller.WaitForGating pointed an exact-path GetMeshNodeStream at {root}/_Access/Public_Access — a node its own doc calls optional ("a partition whose node type does not gate never writes the grant, and that is a normal, common shape").

That is not a wait. Per Doc/Architecture/CqrsAndContentAccess.md → "An OPTIONAL node" (added by #2244, hours before):

the owner answers an authoritative routing NotFound, which terminates the stream with an error … that NotFound opens MeshNodeStreamCache's storm-breaker window on that path, and the breaker fast-fails WRITES too — so the read suppresses the very write it is waiting for.

So the method suppressed the gating write it existed to wait for, and every later read of that path was served a replayed NotFound for the breaker's backoff window. That is this issue's "read-after-write visibility failure", with a specific caller behind it rather than a framework-wide fault — and it matches the MeshWeaver.Education CI signature verbatim ([SYNC_STREAM] OnError … Owner=<Course>/_Access/Public_Access → No node found → CreateOrUpdateNodeRequest … target <unset> → 60 s TimeoutException → [ProvisionPlan] 'Edu' FAILED in phase 2/7 'Create home root').

Rewritten to the sanctioned composition — a live children listing for existence, empty-on-absent, never a point read of an absent path.

Why the item stays open: I fixed the caller I could reproduce. I cannot prove it is the only one. src/ has 451 GetMeshNodeStream( call sites, and a systematic audit for exact-path reads of optional nodes is real work that deserves its own change. In particular the memex-cloud rbuergi/WriteProbe2224 case is not separately reproduced here — the storm-breaker mechanism explains it, but a second caller with the same shape would too.

Two things worth keeping in mind for whoever picks that up, both learned while fixing this one:

select:path alone is a listing that can never match. MeshNode.Path is COMPUTED (Namespace + "/" + Id), so projecting only the path blanks both of its inputs and every row comes back with an empty path. Use select:path,id,namespace.
A listing is not free where a stale answer costs a bound. GatingSettleTimeout is also the "nothing to wait for" answer, so a wait that used to NotFound instantly now genuinely waits. Scope it to the cases where something is actually expected, or the fix trades a correctness bug for a latency one (measured here: 30.3 s → 0.74 s once scoped).

Item B — nothing to do

Retracted by the reporter as documented-by-design (NodeTypeCompilation.md → "What the claim does NOT fix"; the loser's outcome is already correct). No change made.

Item C — FIXED

The baseline guard tested the outcome literal "Failed", and a per-file failure reports "ImportedWithErrors" — a different string. So a partial import advanced LastSyncCommitSha past the commit whose nodes never landed, GitHubWebhookProcessor.SkipReason answered "already at this commit" for every later green build, and the Space's own UI read "up to date" throughout.

StaticRepoImportResult.Failed now carries the tally the importer had always computed, and the decision is a named, pinned predicate (GitHubSyncService.MayAdvanceBaseline, PartialImportHoldsBaselineTest, 6 tests).

The ordering half is deliberately not done. The issue offers "either the phases must order type-compile before instance-import, or a failed item must hold the pointer". The pointer guard is what makes the failure non-permanent, and it self-heals on the next sync of the same commit — the type node landed in the very pass that refused the instance. Reordering the import is a behaviour change on the path every deployment boots through and deserves its own change with its own pins.

Item D — FIXED by #2245 (merged)

Verified rather than assumed: UserIdentityCache is a DI singleton with one query subscription, and TryGetEmail rejects the NodeTypeDefinition row, so nothing wrong was ever stored. The collision lived in the query result — a NodeType declaration filed under the very type it declares — so removing that row makes the log line impossible at any emission rate, rather than merely rarer.

Remaining on this issue: item A's audit — every other exact-path read of an optional node, and whether rbuergi/WriteProbe2224 had a second cause.

Merge pull request #2359 from Systemorph/fix/2229-item-a-optional-nod…

Post-close recurrence, 2026-08-26 13:08Z — the plugin-gate on #2364, `Store/Coupon`

Compile succeeded, then the result write did not converge, then the gate marked it failed. Zero compiler diagnostics.

   ok  Store/Coupon [32 source(s), 9 dependency record entr(ies)]
warn MergeGuard Store/Coupon: refused stale/reordered cross-hub write to 'lastCompileSucceededAt'
warn MergeGuard Store/Coupon: refused stale/reordered cross-hub write to 'lastCompiledVersion'
warn MergeGuard Store/Coupon: refused stale/reordered cross-hub write to 'latestAssemblyPath'
warn MergeGuard Store/Coupon: refused stale/reordered cross-hub write to 'compiledFrameworkVersion'
warn MergeGuard Store/Coupon: refused stale/reordered cross-hub write to 'lastModified'
fail CompileWatcher [AdoptedSourceStamp] Store/Coupon: the adopted build's source stamp was
     committed for request 13:08:12 and the request is still standing with IsDirty=true —
     the write did not converge; the next release request will recompile this type
  RED Store/Coupon [compile:Failed render:Skipped tests:Skipped]
GATE FAILED — compile: Store/Coupon — 1 new failure(s)

Attribution: #2364's diff is three workflow files and one Documentation.Test guard — it cannot reach a plugin compile. #2380 ran the same gate against the same plugins tip (3df19b833) minutes apart and Store/Coupon was ok. So this is the race, sampled unevenly, not a regression in either tree.

Worth noting for the mechanism: every compile-status field was refused as stale in one burst — the writer's base was behind the live node for all five at once, which reads as the writer holding a stale snapshot of the whole node rather than losing a race on one field. That is the same shape as #2339's lastModified-only refusals, one notch worse. Not re-run; the PR is rebased onto a fresh tree per the "my diff cannot reach it" rule.

Second post-close occurrence, on a different node, with the exact log line — which sharpens the diagnosis.

MeshWeaver.Plugins PR #702, run 32973942631, Compile + render node repos (MeshWeaver from ACR):

GATE FAILED — compile: Store/Purchase — 1 new failure(s), 0 stale allow entr(ies).
…
[CompileStateMirror] Store/Purchase: satellite write failed (state stays on the node; next change retries).
System.TimeoutException: No response received in hub Store/Purchase within 00:01:00 for request
CreateOrUpdateNodeRequest (id=KCtJZTqnlU6Ob_tH8FQVyQ) → target portal/nodeops-pUvcwFOa0kKAcvayVc1MVg.
The request may have been undeliverable or the target hub was not found.

The compile did not fail — the RECORDING of it did

CompileStateMirror could not write the satellite carrying the compile state: a 60-second no-response on CreateOrUpdateNodeRequest to portal/nodeops-*. With the state never landing, the gate reads Store/Purchase as not-Compiled and reports compile: FAILED.

That is why these failures carry zero compiler diagnostics — there is nothing wrong with the code. It is the fail-closed-and-silent shape in its purest form: a successful compile, an unwritable result, and a gate that reports a compile error. Nobody files it as a write bug because it presents as a content bug, and the message names the node, not the lane.

Two occurrences, and what they share

when	node	shape
earlier today (#2364)	`Store/Coupon`	`[AdoptedSourceStamp] … the write did not converge; IsDirty=true` after five MergeGuard refusals in one burst
this run (#702)	`Store/Purchase`	`CreateOrUpdateNodeRequest` timed out after 60 s, "target hub was not found"

Different nodes and different proximate errors — a refusal storm versus a timeout — so it is not node-specific and probably not one bug. What they share is the outcome: the compile result cannot be persisted, and the gate translates that into a compile failure. Both are non-deterministic on identical input (#2380 passed the same gate on the same tip minutes after #2364 failed).

Worth noting about this run's timing

The HeartBeatEvent warnings immediately after show the mesh entering RunLevel=DisposeHostedHubs ~3 s after the gate verdict, so this write was pending across the run's teardown. I am not claiming teardown caused it: the request had already been outstanding a full 60 s when it timed out, i.e. it was issued long before shutdown began. But whether the gate concludes while writes are still in flight is worth checking — if it does, a slow write becomes a reported compile failure by construction.

What would make this actionable

The gate should distinguish "the compiler rejected this" from "we could not read/record the state". Right now they are the same red, and only the second one is a platform bug. A gate that says compile FAILED with zero diagnostics is reporting on something other than the compile.

Cross-reference: #2339 (CrossHubPatchAtomicityTest) has an agent on the MergeGuard side, briefed that per-field partial merge is intentional and that a previous atomic-patch fix was built and reverted.

Partial mechanism for the recurrence — one half explained, one half explicitly NOT

Root-causing #2339 turned up a delivery defect that plausibly reaches the Store/Coupon
"compile FAILED with zero diagnostics, IsDirty=true after five MergeGuard refusals in one burst"
recurrence — but only for one of the two write paths involved, so I want to be precise rather than
let it absorb the whole report.

The defect (fixed in #2391). DataExtensions.ChangeContainsStampedWrite — the write-identity
gate every cross-hub stream.Update waits on — serialized the entire node to read two scalars
(id, version). It runs once per still-pending patch on every emission of the owner's reduced stream,
so a burst of K concurrent writes to one node runs it up to K(K+1)/2 times, and it is
self-amplifying: the further emissions lag, the more patches are pending, the more full rebuilds
each next emission must pay before it can be delivered. Counted on a 288-write burst: 9 455
full-node rebuilds, at 0.9–1.9 ms each. The owner applied all 288 patches in 1.5 s while every
subscriber saw a ~2.7 s gap with no frames at all — on an idle 18-core box.

Why that can produce a non-converging write after repeated refusals

UpdateRemote's conflict recovery (RebaseSource) waits up to ConflictRebaseBound = 5 s for
the writer's own mirror to advance past the version the owner refused, then rebuilds the patch
against whatever it has. A stalled mirror has not advanced — STALE_MIRROR fires, the re-attempt is
rebuilt from the same base, and it is refused identically. That is a write that never converges,
produced by repeated refusal, and the stall that disarms the recovery is worst precisely when many
concurrent writers are hitting one node — i.e. exactly when refusals cluster into "five in one
burst". 2.5–2.7 s idle scales past 5 s on a loaded 4-vCPU pod without difficulty.

It also matches the "non-deterministic on identical input" signature: #2380 passing the same gate on
the same plugins tip minutes later is what you would expect when the discriminator is how many
writers happened to overlap.

Why it does NOT explain the line that was actually logged

[AdoptedSourceStamp] writes via workspace.GetMeshNodeStream() with no path — the own-node
path (UpdateOwn). That never goes through MergeGuard or UpdateRemote at all. So the five
refusals are evidence that concurrent cross-hub writers were active on that node in that window —
this defect's precondition — but I have not shown a causal chain from the delivery stall to that
own-node stamp write failing to converge.

So: this is a contributing mechanism, not a closure. If #2391 does not stop the recurrence, the
own-write half needs its own investigation, and the thing to instrument first is what clobbers or
loses the own-node commit that stampHighWater has already recorded — not the merge guard.

fix(#3051): do not claim "nothing was written" when the write had sta…

fix(mesh): a create whose store is unreachable answers Unavailable, n…

Navigation Menu

Repository navigation

Write lane degraded on memex-cloud: create chain dies on a NotFound stream, a finished Store/Provision storms compile-state, plus a permanent-silent import order bug and a 2/s identity-cache loop #2229

A. `CreateNodeRequest` never replies: the create chain subscribes to the node's stream before the node exists

B. A finished `Store/Provision` keeps re-racing its own compile-state node — a write storm that starves the fleet

C. Provisioning order makes an import failure PERMANENT and silent

D. `UserIdentityCache` hot loop: 2/second, on every serving pod

🚨 Correcting item B — I filed it wrong, twice over

🚨 Item A upgraded — it is NOT "the create is lost". The create SUCCEEDS and the REPLY is lost.

Evidence, three nodes, two independent investigators

Consequences — "lost create" and "lost reply" demand opposite client behaviour

And reads are unstable in BOTH directions — I asserted otherwise earlier and was wrong

What this changes for anyone acting on this issue

Read side, discriminated: the two read routes fail differently, and only one of them is a defect

🚨 A documentation/behaviour mismatch worth its own fix

Severity note on item A's read side: the false negative is FREQUENT, and it has now corrupted two investigations

Item A also fires on `memex` (namespace `memex` / memex.systemorph.com), not only memex-cloud

Status after #2267 — two of the four items are done, and item A has a named root cause with a live reproduction

Item A — one concrete root cause fixed, the item is NOT closed

Item B — nothing to do

Item C — FIXED

Item D — FIXED by #2245 (merged)

Post-close recurrence, 2026-08-26 13:08Z — the plugin-gate on #2364, `Store/Coupon`

The compile did not fail — the RECORDING of it did

Two occurrences, and what they share

Worth noting about this run's timing

What would make this actionable

Partial mechanism for the recurrence — one half explained, one half explicitly NOT

Why that can produce a non-converging write after repeated refusals

Why it does NOT explain the line that was actually logged

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Write lane degraded on memex-cloud: create chain dies on a NotFound stream, a finished Store/Provision storms compile-state, plus a permanent-silent import order bug and a 2/s identity-cache loop #2229

Description

A. CreateNodeRequest never replies: the create chain subscribes to the node's stream before the node exists

B. A finished Store/Provision keeps re-racing its own compile-state node — a write storm that starves the fleet

C. Provisioning order makes an import failure PERMANENT and silent

D. UserIdentityCache hot loop: 2/second, on every serving pod

Activity

rbuergi commented on Aug 25, 2026

rbuergi commented on Aug 25, 2026

🚨 Correcting item B — I filed it wrong, twice over

rbuergi commented on Aug 25, 2026

🚨 Item A upgraded — it is NOT "the create is lost". The create SUCCEEDS and the REPLY is lost.

Evidence, three nodes, two independent investigators

Consequences — "lost create" and "lost reply" demand opposite client behaviour

And reads are unstable in BOTH directions — I asserted otherwise earlier and was wrong

What this changes for anyone acting on this issue

rbuergi commented on Aug 25, 2026

Read side, discriminated: the two read routes fail differently, and only one of them is a defect

🚨 A documentation/behaviour mismatch worth its own fix

rbuergi commented on Aug 25, 2026

rbuergi commented on Aug 25, 2026

Severity note on item A's read side: the false negative is FREQUENT, and it has now corrupted two investigations

rbuergi commented on Aug 25, 2026

Item A also fires on memex (namespace memex / memex.systemorph.com), not only memex-cloud

rbuergi commented on Aug 25, 2026

Status after #2267 — two of the four items are done, and item A has a named root cause with a live reproduction

Item A — one concrete root cause fixed, the item is NOT closed

Item B — nothing to do

Item C — FIXED

Item D — FIXED by #2245 (merged)

rbuergi commented on Aug 26, 2026

Post-close recurrence, 2026-08-26 13:08Z — the plugin-gate on #2364, Store/Coupon

rbuergi commented on Aug 26, 2026

The compile did not fail — the RECORDING of it did

Two occurrences, and what they share

Worth noting about this run's timing

What would make this actionable

rbuergi commented on Aug 26, 2026

Partial mechanism for the recurrence — one half explained, one half explicitly NOT

Why that can produce a non-converging write after repeated refusals

Why it does NOT explain the line that was actually logged

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

A. `CreateNodeRequest` never replies: the create chain subscribes to the node's stream before the node exists

B. A finished `Store/Provision` keeps re-racing its own compile-state node — a write storm that starves the fleet

D. `UserIdentityCache` hot loop: 2/second, on every serving pod

Item A also fires on `memex` (namespace `memex` / memex.systemorph.com), not only memex-cloud

Post-close recurrence, 2026-08-26 13:08Z — the plugin-gate on #2364, `Store/Coupon`