fix(create): the post-creation verdict names what the leg waits on, and is the only verdict by rbuergi · Pull Request #6455 · Systemorph/MeshWeaver

rbuergi · 2026-10-10T21:17:09Z

What the stall is

With #6421 live, a reading on 2026-10-10 found 18 stalls on memex-cloud in 14 h, all in the first minutes of a pod's first process, all reading <path> was written, but its post-creation handlers had not finished within 30s. The targets are System-written creates: Admin/PlatformVersion/_Activity/startup-…, then Admin/_Notification/… about every 70 s.

Read for one of those pods (memex-portal-deployment-5499f76b76-ltlwc, governed read-only Logs actions on control, listed below):

The stalled create's trail ends CREATE_STAGE write (+311ms) → CREATE_CHAIN_EMITTED (+365ms) → CREATE_POST_HANDLERS_START (+365ms) and nothing after it. The node-operation hub's pump is idle.
Six creates stalled this way between 18:10:50Z and 18:16:13Z. The process wrote a crash dump at 18:16:19Z and was replaced. In the replacing process the same boot create completed in 11 ms.
The first process never logged Application started. The replacing one logged it 40 ms after its boot create.

No registered handler applies to these node types: the four in src/ are for User, Space, the sync config and non-static partition roots, and the three MeshWeaver.AI registers are all for User. For a node no handler applies to, the post-creation leg does two things only. It resolves every INodePostCreationHandler registration, and it asks each one's Matches. A handler a module registered is a forwarding proxy (ModuleServiceProxy), and every call on it, Matches included, first resolves the module's instance from the module's own container. So every create in the mesh, whatever its type, waits on every module's container until that module's handler exists.

The repro below produces the production line byte for byte on the code before this change.

What changes

The leg reports its stage as it enters each step, on the verdict, the log line and the fate trail: post-creation-handlers: resolving, … matching <handler> (i/n), … <handler> (i/n), … <handler> (i/n): additional nodes.
A module's proxy is named without resolving it (ModuleServiceProxy.Label): <registered type or interface> [module <name>, registration <index>]. Naming it by resolving it would wait on the same container. The validators stage (fix(create): a stalled validators stage names the validator it waits on (#6391) #6421) uses the same label, so a validator a module contributes is no longer named by a generated proxy class.
The deadline claims the answer before it writes the line. It used to read the once-only gate, write the Error line and only then claim. A handler chain finishing in between answered Ok under a line saying 'outcome unknown'.
The chain's own deadline is no verdict once the node was emitted. The leg runs inside the chain's OnNext, and Rx's Timeout holds an expiry back while an OnNext is in flight. A leg that returned late was therefore followed by a second Error line, reached no verdict — stalled at stage write, for a create that was written and already answered. It is now a trail entry (CREATE_CHAIN_DEADLINE_AFTER_EMIT).

The 30 s budget is unchanged. No timeout, retry or bound was touched.

What this does not establish

What holds a module's container during those boots. No log line carries a thread's stack, and the AnalyzeDump action that reads the dump those processes wrote is not on the control instance yet (its content schema refuses dumpPod). The same pods are the ones described in "a silo wedged from boot" (see #6432).

That the production wait is the resolve-or-match step is an inference by elimination: no handler applies, so nothing asynchronous is left to wait for. The refined stage on the next occurrence confirms or refutes it in one line and names the module. A change that lets the create answer without the module's handler would decide Matches on the handler's behalf, so none is made here.

Tests

ACreateWaitsOnAModuleContainerUnderConstructionTest (MeshWeaver.Compiler.Pipeline.Test). A real module compiled and installed into its own container. The construction of one of its singletons is parked, then a node of an unrelated type is created as System. The row is written, the line reads post-creation-handlers: matching INodePostCreationHandler [module MeshWeaver.Test.HeldContainer, registration 0] (4/4), the module's handler was never built, exactly one Error line is written for the create, and after the release the reply carries the same verdict and the next create is answered Ok.
- Negative control, measured on the unchanged src/: the test fails. The line reads … was written, but its post-creation handlers had not finished within 6s — answering 'outcome unknown', and reached no verdict within 6s — stalled at stage write follows it.
- Control in the suite: with nothing under construction the first create succeeds, and it built the module's handler.
LifecycleRequestsAlwaysReachAVerdictTest (MeshWeaver.Hosting.Test): a handler that never completes is named in the verdict (post-creation-handlers: SilentPostCreationHandler (), and the same handler armed for another path lets the create through.

Run locally on the final tree: Hosting.Test 14 of 14 for the four verdict classes, Graph.Test 10 of 10 for the post-creation classes, LocalizationTest 66 of 66, MeshWeaver.Documentation.Test 815 of 815. Release -warnaserror is clean for Mesh.Contract, Messaging.Hub, Documentation, Graph and the four test projects, one project per invocation.

ModuleRootServicesSwapTest.AModuleWithRootServices_SwapsLive_AndEveryConsumerFollowsTheNewGeneration fails in a local Debug run with and without this change (1 context(s) RETAINED), measured on the unchanged src/ in the same run as the negative control.

Docs

Doc/Architecture/WriteVerdictTotality carries the readings, the stage table, the two corrections and what is not established. Doc/Architecture/ReadingASiloEviction adds two readings to the boot fingerprint: the missing Application started, and what the create line says from this image on.

Deploy

Reaches a portal with its image. No address needs a recycle. The log line keeps its opening words, so post-creation handlers had not finished still finds it. After the roll, the next boot-time occurrence on memex-cloud is read for its waiting at clause.

Mirror-sync: MeshWeaver.Plugins runs npm run sync:i18n -- --ref <merge sha> after this lands — one re-worded value, activity.node.create.postCreationOutcomeUnknown gains {stage} in en and de; no key is added.

Governed reads behind the readings (control, read-only `Logs`)

Ops/Actions/logs-memexcloud-20261010-w14b-boot-ltlwc and …-ltlwc-2 through …-ltlwc-7.

🤖 Generated with Claude Code

…nd is the only verdict The boot-time stalls read "<path> was written, but its post-creation handlers had not finished within 30s" for creates no registered handler applies to. For such a node the leg only resolves the handler registrations and asks each one's Matches, and a handler a module registered is a forwarding proxy whose every call first resolves the module's instance from the module's own container. - The leg reports its stage as it enters each step: resolving, matching <handler> (i/n), <handler> (i/n), <handler> (i/n): additional nodes. On the verdict, the log line and the fate trail. - A module's proxy is named without resolving it (ModuleServiceProxy.Label): registered type or interface, module, registration index. The validators stage uses the same label. - The deadline claims the answer before it writes the line, so 'outcome unknown' is never logged for a create a finishing handler chain answered Ok. - The chain's own deadline is no verdict once the node was emitted. The leg runs inside the chain's OnNext, so a leg that returned late was followed by a second Error line, "stalled at stage write", for a create already written and answered. Repro: a real module in its own container, one singleton's construction parked, a create of an unrelated type. Fails on the code before (the line names nothing, the second line is written). Control: nothing parked, the first create succeeds. Refs #6391 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Copilot

🟡 Changes recommended

Additional-node discovery can still be reported under the wrong stage, and central verdict behavior lacks accurate documentation and race coverage.

4 open findings

What changed in this PR

Improves stalled post-creation diagnostics and ensures each create produces one consistent verdict.

Changes:

Tracks the exact post-creation stage and labels module proxies without resolving them.
Prevents duplicate or contradictory deadline verdicts.
Adds integration coverage, localized messages, and operational documentation.

File	Description
`src/MeshWeaver.Mesh.Contract/MeshExtensions.cs`	Refines stages and deadline verdict handling.
`src/MeshWeaver.Mesh.Contract/ModuleServiceForwarders.cs`	Adds non-resolving proxy labels.
`src/MeshWeaver.Messaging.Hub/Localization/strings.en.json`	Adds stage detail to English verdict.
`src/MeshWeaver.Messaging.Hub/Localization/strings.de.json`	Adds stage detail to German verdict.
`test/MeshWeaver.Hosting.Test/LifecycleRequestsAlwaysReachAVerdictTest.cs`	Tests stalled post-creation handlers.
`test/MeshWeaver.Compiler.Pipeline.Test/ACreateWaitsOnAModuleContainerUnderConstructionTest.cs`	Reproduces module-container stalls.
`src/MeshWeaver.Documentation/Data/Architecture/WriteVerdictTotality.md`	Documents stages and verdict semantics.
`src/MeshWeaver.Documentation/Data/Architecture/ReadingASiloEviction.md`	Updates operational diagnosis guidance.

🧠 Review effort: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

rbuergi · 2026-10-10T21:33:12Z

+                        // 🚨 CLAIMED FIRST, then said. This used to READ the gate, write the Error
+                        // line and only then claim through Respond — so a handler chain finishing
+                        // in between answered `Ok` under a line that says 'outcome unknown'. The
+                        // line is the instrument this failure is read through, so it is written
+                        // only for a verdict that IS the create's answer.
+                        if (!TryClaimResponse())


Agreed, the first push did not pin the ordering on purpose. Added PostCreationVerdictIsSaidOnceTest (MeshWeaver.Hosting.Test) in 5825364. It parks the leg inside a handler, releases it from INSIDE the write of the Error line (the point between the old gate read and the old claim) and gives it 3 s to finish and answer before the write returns. The reply must be the line's verdict.

Measured on main's src/ with the test in place: it fails with LINE: … answering 'outcome unknown' and REPLY: success=True. With the claim made first it passes: the reply is Unavailable and carries the same stage as the line.

…e; pin the claim-first order Review round on the post-creation verdict: - GetAdditionalNodes ran, and its result was enumerated, while the stage still named the handler's Handle. It is now `<handler> (i/n): discovering additional nodes`, the result is materialised once inside the try, and the stage returns to the handler when Handle is subscribed. - PostCreationVerdictIsSaidOnceTest makes the claim-before-log window deterministic: it releases a parked leg from inside the write of the Error line and gives it time to answer. The reply is the line's verdict. On main's order it is Ok under a line saying 'outcome unknown' (measured). - Docs: after the release the parked create carries the same Unavailable verdict and only the next create is Ok; `resolving` names no module container, only `matching … [module …]` does. Refs #6391 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…n-stage-names-handler

Copilot

🟡 Changes recommended

The response can remain unsent after being claimed, and proxy labels can become stale after live module swaps.

3 open findings

3 resolved since last review

🧠 Review effort: Balanced

…y's label follows a live swap Review round 2: - The deadline builds its verdict, claims, POSTS, and only then writes the trail and the Error line. Once the gate is claimed every later Respond is refused, so nothing may stand between the claim and the post. - ModuleServiceProxy's label is read from the module's CURRENT generation's registration on every call instead of being remembered at boot: the proxy outlives a live swap, which keeps route, service type and lifetime but may change the implementing class. The Create overload added for the remembered name is gone again. AProxysLabel_NamesTheCurrentGenerationsImplementation_AfterALiveSwap pins it; on the remembered name it reads "Greeter" after a swap to "RenamedGreeter" (measured). Refs #6391 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…n-stage-names-handler

Copilot

🟡 Changes recommended

The emitted-only timeout suppression can discard a fallback before any response has been claimed.

2 open findings

2 resolved since last review

🧠 Review effort: Balanced

rbuergi · 2026-10-10T21:58:41Z

+                    // "reached no verdict — stalled at stage write": a second Error line, for a
+                    // create that was written and already answered, naming a stage that had
+                    // finished. Nothing is owed here: the row is written and the answer was given.
+                    if (ex is LifecycleVerdictStalledException && System.Threading.Volatile.Read(ref emitted))


Not changing this, because in the case described the answer is still owned, and gating on responded would bring back the false verdict.

The branch does not cancel anything. Once emitted is set, the same OnNext arms the post-creation deadline (Observable.Timer(deadline).Subscribe(...)) BEFORE it subscribes the handler leg, and that timer is disposed in exactly one place: the leg's Finally, i.e. when the leg has terminated. Each terminal of the leg answers: completion posts Ok, a fault runs the compensation and posts its failure. So when the outer fallback is delivered after the callback returns with responded == 0, one of two things is true:

the leg is still pending asynchronously, and the post-creation timer is armed. Its due time is the same absolute deadline, already past, so it fires as soon as its scheduler runs it, and it posts the verdict that names the step;

or the leg terminated inside the callback, in which case it already claimed the gate.

There is no path where this fallback is the sole remaining answer. If it answered whenever responded is still 0, it would win exactly that race and say reached no verdict — stalled at stage write for a row that is written, which is the second, wrong verdict this branch exists to remove (ACreateWaitsOnAModuleContainerUnderConstructionTest asserts one Error line for the create). A delayed timer delays the correct verdict by the same delay; it does not lose it.

github-actions · 2026-10-10T22:49:34Z

Test Results

20 files - 4 20 suites - 4 54m 30s ⏱️ + 1m 11s
11 334 tests - 289 11 334 ✅ - 96 0 💤 - 193 0 ❌ ±0
11 348 runs - 289 11 348 ✅ - 96 0 💤 - 193 0 ❌ ±0

Results for commit f921766. ± Comparison against base commit 5467df1.

This pull request removes 335 and adds 22 tests. Note that renamed tests count towards both.


   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---, expected: True)
   --- End of inner exception stack trace ---, isDenial: True)
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
 ---> (Inner Exception #1) System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known<---
…

Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "50091c0c-2a13-4c40-83d4-07cd08e96a4e")
Memex.Portal.Shared.Test.SessionDenialIsAnAnswerTest ‑ OnlyAVerdictReadsAsADenial(shape: "the same verdict nested, as a late denial dispatch"···, failure: System.InvalidOperationException: write failed
 ---> System.UnauthorizedAccessException: Access denied
   --- End of inner exception stack trace ---, isDenial: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's FIRST member", fault: System.AggregateException: One or more errors occurred. (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's SECOND member", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall nested in an aggregate inside a wrapper", fault: System.InvalidOperationException: outer
 ---> System.AggregateException: One or more errors occurred. (Value does not fall within the expected range.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.ArgumentException: Value does not fall within the expected range.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---

   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall wrapped as InnerException", fault: System.InvalidOperationException: outer
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "an aggregate of unrelated faults", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Value does not fall within the expected range.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
, expected: False)
MeshWeaver.Graph.Test.ACreateWaitsOnAModuleContainerUnderConstructionTest ‑ WhileAModuleSingletonIsBeingBuilt_AnUnrelatedCreate_IsAnsweredNamingTheModulesHandler
MeshWeaver.Graph.Test.ACreateWaitsOnAModuleContainerUnderConstructionTest ‑ WithNothingUnderConstruction_TheFirstCreate_Succeeds
MeshWeaver.Graph.Test.ModuleRootServicesSwapTest ‑ AProxysLabel_NamesTheCurrentGenerationsImplementation_AfterALiveSwap
…

Copilot AI balanced review requested due to automatic review settings October 10, 2026 21:17

rbuergi mentioned this pull request Oct 10, 2026

CreateNode lifecycle stalls at the validators stage with no verdict — a validator never answers within its window during the platform-startup boot-activity create #6391

Closed

Copilot started reviewing on behalf of rbuergi October 10, 2026 21:17

View session

Copilot AI reviewed Oct 10, 2026

View reviewed changes

rbuergi and others added 2 commits October 10, 2026 23:31

Merge remote-tracking branch 'origin/main' into fix/6391-post-creatio…

7f5a541

…n-stage-names-handler

Copilot AI balanced review requested due to automatic review settings October 10, 2026 21:32

Copilot started reviewing on behalf of rbuergi October 10, 2026 21:32

View session

Copilot AI reviewed Oct 10, 2026

View reviewed changes

Comment thread src/MeshWeaver.Mesh.Contract/MeshExtensions.cs Resolved

Comment thread src/MeshWeaver.Mesh.Contract/ModuleServiceForwarders.cs Resolved

rbuergi and others added 2 commits October 10, 2026 23:48

Merge remote-tracking branch 'origin/main' into fix/6391-post-creatio…

f921766

…n-stage-names-handler

Copilot AI balanced review requested due to automatic review settings October 10, 2026 21:48

Copilot started reviewing on behalf of rbuergi October 10, 2026 21:49

View session

Copilot AI reviewed Oct 10, 2026

View reviewed changes

rbuergi merged commit dc9f15b into main Oct 10, 2026

rbuergi deleted the fix/6391-post-creation-stage-names-handler branch October 11, 2026 07:23

rbuergi mentioned this pull request Oct 11, 2026

Incident comment write stalls 30s then aborts when the incident hub cannot hydrate from persistence #4962

Closed

Navigation Menu

Repository navigation

fix(create): the post-creation verdict names what the leg waits on, and is the only verdict - #6455

fix(create): the post-creation verdict names what the leg waits on, and is the only verdict#6455
rbuergi merged 5 commits into
mainfrom
fix/6391-post-creation-stage-names-handler

rbuergi commented Oct 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

rbuergi Oct 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

rbuergi Oct 10, 2026

Uh oh!

github-actions Bot commented Oct 10, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

rbuergi commented Oct 10, 2026

What the stall is

What changes

What this does not establish

Tests

Docs

Deploy

Governed reads behind the readings (control, read-only Logs)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

rbuergi Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

rbuergi Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 10, 2026

Test Results

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Governed reads behind the readings (control, read-only `Logs`)