fix(#3478): a process its readiness validation refuses no longer JOINS the mesh by rbuergi · Pull Request #3491 · Systemorph/MeshWeaver

rbuergi · 2026-09-06T22:15:40Z

"not ready should NEVER run" · "validate before running" — maintainer, 2026-09-06

A readiness probe takes a pod out of the load balancer. It does not take the process out of the
mesh. On 2026-09-06 the NodeType bake gate on memex.systemorph.com correctly refused ci.7926,
Kubernetes correctly stalled the rollout — and the refused pod then ran for two hours inside the
mesh, stamping NodeType compile records with its own framework identity. Crm/Offer and
Crm/Opportunity came to name assemblies stamped sc273ee39f… while the serving replicas ran
s2f227642d…, and every deal and offer page on the client portal was dead until 21:50Z (#3472).

The protection produced the outage — not by misjudging the image, but by leaving the refused
process a participant. This makes the refusal real.

Where a process actually joins

"Do not join until validated" does not survive contact with the ordering: a portal becomes
addressable long before it can have validated anything, and the validation itself needs the mesh —
the bake sweep reads every NodeType and its sources through mesh queries. So membership-for-READING
cannot move after the verdict. Membership-for-WRITING can, and that is the half that does the
damage. The acts that carry this process's own build identity into the shared mesh are few and
enumerable:

act	site	what it says
NodeType compile stamp, batch path	`NodeTypeBatchBake.WriteStamp`	`CompiledFrameworkVersion`, `CompiledModulesHash`, `CompiledDependencies`, assembly coordinates
NodeType compile stamp, activation path	`NodeTypeCompilationHelpers` post-compile write-back	the same field-set — the two paths share it literally
module-set adoption	`ModuleSetStore.RecordAdoption`, from `MemexConfiguration`	"a replica is serving set N"

Assembly bytes are deliberately not gated: the store is keyed (nodeTypePath, version) with the
producing framework identity in the key, and NodeTypeBakeStatus.ClassifyDetailed cannot look for
bytes at all until the RECORD names a collection, a path and a version. Bytes nobody's record points
at are inert. The pointer is the poison, so the pointer is what is gated.

The mechanism

MeshPublicationGate (src/MeshWeaver.Mesh.Contract/Services/MeshPublicationGate.cs) — a
mesh-scoped singleton every one of those writes goes through, reading its verdict from the
registered IMeshAdmissionAuthority implementations (today exactly one: the bake gate).

admission	meaning	a publication
`Unarmed`	no authority armed	runs, unchanged
`Provisional`	armed, no verdict yet	held, released on admission
`Admitted`	validation passed	runs
`Refused`	validation failed	never built, never written

Publications are offered as a Func<IObservable<Unit>>, not an observable, so a refused one is never
constructed — no storage read is issued and no RequireSubscribeObservable is minted only to be
dropped. A held one is constructed at release time, so its compare-and-set reads the row it is about
to race rather than replaying a version read before the verdict existed.

Provisional membership, stated: a provisional member may read the mesh and compile; it may not
publish. That is what makes validate before running implementable without reordering startup — the
sweep runs exactly as before, but its stamps land after its verdict instead of during it.

One predicate, two consumers

The defect is a divergence — two verdicts, one enforced — so admission is derived FROM the readiness
predicate rather than being a second table:

public bool ReadinessGranted => Phase switch {
    BakePhase.NotStarted => true, BakePhase.Complete => true,
    BakePhase.Faulted => AllowUnprovenBake, _ => false };

public MeshAdmission Admission
    => !GatesReadiness ? MeshAdmission.Unarmed
        : Phase is BakePhase.NotStarted or BakePhase.Running ? MeshAdmission.Provisional
        : ReadinessGranted ? MeshAdmission.Admitted
        : MeshAdmission.Refused;

Invariant: Admitted ⟹ ReadinessGranted — pinned over the whole phase × flag space by
MeshAdmissionIsNeverMorePermissiveThanReadinessTest. It is deliberately less permissive in one
place: an armed gate at NotStarted, where readiness is granted (a probe cannot tell "switched
off" from "has not started") but no verdict exists. The incident's pod published through exactly that
window — its bake starts at ApplicationStarted, minutes after the process does.

🚨 Unarmed means unarmed. With PreWarm:GateReadiness off — the chart default, every dev host,
every test mesh — nothing consumes the bake verdict and nothing is enforced from it. This change is
byte-for-byte inert there.

A process that becomes unhealthy AFTER joining

The verdict is level-triggered, re-read at every publication rather than latched at admission: a
regression recorded hours after a clean sweep stops publication from that moment, and a
RetractRegression resumes it. Same mechanism, no second one. It never withdraws a pod that is
serving — withholding publication is not flipping readiness.

The second witness — and the regression this change had to close

Gating the activation-path stamp breaks #1214's self-healing retraction: it watched the type's
SHARED record, which a refused pod can no longer move, so a false-positive regression would become a
permanent stall instead of clearing itself. The retraction's question was always process-LOCAL, and
the record was only ever a proxy for it — so LocalNodeTypeBuilds publishes the fact directly (the
compile write-back raises it before the possibly-withheld stamp) and the watch takes whichever
witness answers first. The record watch stays: a type baked by a PEER on the same image is a real
recovery the local signal cannot see.

🚨 The decision this issue asked for: exit non-zero, or stay up inert?

A refused process stays up and INERT. It does not exit. Stated explicitly because this is a
maintainer-level call and the disruptive option must not be taken quietly.

The case for exiting is real: a non-zero exit turns a stalled rollout into a CrashLoopBackOff,
which is louder and arguably a more honest description of a process that will never serve.

Why inert is what ships:

The refusal is already real without it. Once publication is gated, exiting adds no protection —
it only changes how the stall is presented. Loudness is not the defect A process refused by its readiness gate still JOINS the mesh — readiness gates traffic, not membership, and a refused pod stamped module generations for 2h (caused #3472) #3478 describes.
It destroys the diagnosis surface. The refused pod's /health payload naming the regressed
types is the only place that verdict lives; a crash-looping pod has no readable health endpoint and
its logs rotate every restart.
Each restart re-runs a full cold bake (~2.4 s per NodeType, ~10 min on our largest mesh) to
learn the same verdict again, on a cluster we pay for.
It changes deployment behaviour on every path — Monolith, Aspire, the plugin tester — on
evidence this change is the first to produce. ModuleSetConvergence declined the neighbouring
options ("have a behind replica restart itself, or flip readiness") for the same reason and noted
they become decidable once the state is named. This is that state, now named.

It is decidable now: with admission derived from one predicate, "exit when Admission is Refused" is
a single call in DynamicTypePreWarmerHostedService's terminal handler. Deliberately not taken here.

Falsification — both arms, with numbers

Scenario, exactly as the issue specifies: two replicas on image A (records stamped
s2f227642d… / Offer|Opportunity|Feedback/v11-s2f22764-A.dll / module hash
8f251f57…), roll to image B, B's bake regresses on Feedback. Assertions are on the DURABLE rows,
read back through the mesh's own IStorageAdapter — the rows a peer replica reads. Real mesh
(MonolithMeshTestBase), production write (NodeTypeBatchBake.WriteStamp), no mocking.

arm	result
Fix present, gate armed, sweep regresses	0 of 2 stamps landed. Both records still read image A verbatim. `WithheldCount = 2`, `PassedCount = 0`, `ReleasedCount = 0`. `Admission = Refused`.
Fix present, gate armed, sweep clean (control)	mid-sweep `HeldCount = 3`, records unchanged; after `MarkComplete`, `ReleasedCount = 3`, `WithheldCount = 0`, all 3 records now name image B. Proves the test can see a stamp and the happy path still records what it compiled.
Fix present, gate UNARMED (the two-image window, kept in the suite)	the same regressing sweep stamps 2 records with image B — `PassedCount = 2`. The pre-fix behaviour, permanently observable, so a change that quietly disarms the gate cannot look like a pass.
Fix REVERTED (`Publish` forced to `Unarmed`, measured by hand, then restored)	3 of 4 cases fail: `Offer` → `(3366f713…, "", nodetype-cache, Offer/v22488-sc273ee3-B.dll, 22488)` where image A's stamp had been, `Opportunity` likewise, and `HeldCount` collapses 3 → 0. The two-image window reproduced verbatim. The one case that still passes under the revert is the unarmed arm — correctly, since it asserts the pre-fix behaviour.
Self-healing survives	a locally rebuilt condemned type retracts its regression through `LocalNodeTypeBuilds` although the record never moves; `Admission` leaves `Refused`.

Local verification (Release, -warnaserror, one project per invocation): MeshWeaver.Mesh.Contract,
MeshWeaver.Compiler.Pipeline, MeshWeaver.Hosting, MeshWeaver.PluginCatalog,
MeshWeaver.Documentation, Memex.Portal.Shared — 0 warnings, 0 errors each.
MeshWeaver.Hosting.Test 393/393, MeshWeaver.Documentation.Test 304/304 (link integrity,
What's New integrity and every ratchet guard), plus the 18 new cases.

Scope boundaries (deliberate)

It does not stop a bad image being CHOSEN — that is Roll selection must pick the latest platform version that ships ALL of an environment's plugins — completeness is a selection criterion, not a post-hoc verdict #3479, and the two are complementary:
Roll selection must pick the latest platform version that ships ALL of an environment's plugins — completeness is a selection criterion, not a post-hoc verdict #3479 stops the bad roll starting, this one stops a bad roll doing damage when one happens anyway.
Neither subsumes the other.
It does not gate Orleans cluster membership or hub activation. A refused process stays
addressable and its grains still activate — that is what lets the sweep read.
It does not change the stall posture. The old image keeps serving.

Also in this change

Doc/Architecture/MeshAdmission — the design, the join-point inventory, the exit-vs-inert
trade-off, and the scope boundaries. Cross-linked from ModuleSetConvergence, whose "rejected
alternatives" section this decides one of.
A What's New entry (Category: Fix).

No public type or member is removed, so no cross-repo counterpart is required.

🤖 Generated with Claude Code

…S the mesh Readiness gates traffic; it never gated membership. On memex.systemorph.com, 2026-09-06, the NodeType bake gate correctly refused ci.7926 and Kubernetes correctly stalled the rollout — and the refused pod then ran for two hours inside the mesh, stamping NodeType compile records with its own framework identity. Crm/Offer and Crm/Opportunity came to name assemblies stamped sc273ee39f… while the serving replicas ran s2f227642d…, and every deal and offer page on the client portal was dead until 21:50Z (#3472). The protection produced the outage, by leaving the refused process a participant. The join point is not process start and not Orleans membership — it is the first mesh-visible WRITE. A portal is addressable long before it can have validated anything, and the bake sweep must READ the mesh to have something to validate, so membership-for-reading cannot move after the verdict. Membership-for-writing can, and that is the half that does the damage: the NodeType compile stamp on both write-back paths, and the module-set adoption record. Assembly BYTES stay ungated on purpose — the bake probe cannot look for them until a RECORD names them, so bytes nobody points at are inert. MeshPublicationGate holds every such publication while this process is provisional, releases them when its validation passes, and never builds them when it does not. Admission is DERIVED from the readiness predicate (Admitted ⟹ ReadinessGranted), so "not ready" and "not admitted" cannot drift apart; it is level-triggered, which answers the become-unhealthy-after-joining direction with the same mechanism. Unarmed deployments — the chart default — are byte-for-byte unchanged. A refused process stays up INERT rather than exiting: the refusal is already real without a crash loop, exiting destroys the /health payload that is the only place the verdict lives, and each restart re-runs a full cold bake. That trade-off is stated in the PR body and in Doc/Architecture/MeshAdmission; the option is a single call away now that the state is named. Gating the activation-path stamp would have broken #1214's self-healing retraction, which watched the SHARED record a refused pod can no longer move — so LocalNodeTypeBuilds publishes the process-LOCAL fact the record was ever a proxy for, and the watch takes whichever witness answers first. Falsification (two replicas on image A, roll to image B whose bake regresses, real mesh, production write): with the fix, 0 of 2 stamps land and both records still read image A; reverted, both records name image B's assembly and the held count collapses 3 → 0 — the two-image window verbatim. The unarmed arm keeps that pre-fix behaviour permanently observable in the suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

🟡 Changes recommended

There is at least one compile-breaking typo and a concurrency/disposal hazard in the new process-local signal that should be fixed before merge.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Implements “mesh admission” so a pod whose startup validation (NodeType bake gate) refuses it can still read/compile, but cannot publish mesh-visible identity-bearing writes (NodeType compile stamps and module-set adoption), preventing a refused rollout from poisoning shared records.

Changes:

Introduces MeshPublicationGate + IMeshAdmissionAuthority and wires NodeTypeBakeGateState to derive mesh admission directly from the readiness predicate.
Gates NodeType compile-state stamping (batch + activation paths) and module-set adoption so refused/provisional pods cannot publish; adds LocalNodeTypeBuilds as a process-local witness to preserve #1214 self-healing.
Adds falsification tests and architecture/What’s New documentation for the admission model.

File summaries

File	Description
test/MeshWeaver.Hosting.Test/RefusedProcessDoesNotJoinTheMeshTest.cs	End-to-end falsification suite asserting refused/provisional processes cannot stamp shared NodeType records.
test/MeshWeaver.Hosting.Test/MeshAdmissionIsNeverMorePermissiveThanReadinessTest.cs	Pins admission/readiness invariants and `MeshPublicationGate` semantics (refuse/hold/release/discard).
src/MeshWeaver.PluginCatalog/ModuleSetAdoptionService.cs	Defers module-set adoption through `MeshPublicationGate` so refused pods don’t record “serving set N”.
src/MeshWeaver.Mesh.Contract/Services/MeshPublicationGate.cs	Core admission gate: holds/releases/refuses publications based on most-restrictive authority verdict.
src/MeshWeaver.Mesh.Contract/Services/LocalNodeTypeBuilds.cs	Process-local “usable build” signal used to retract false regressions even when shared stamps are withheld.
src/MeshWeaver.Hosting/Persistence/PersistenceExtensions.cs	Registers `MeshPublicationGate` and `LocalNodeTypeBuilds` as mesh-scoped singletons.
src/MeshWeaver.Hosting/NodeTypeBatchBake.cs	Routes batch-path compile stamps via `MeshPublicationGate` (factory-based, cold).
src/MeshWeaver.Hosting/NodeTypeBakeGate.cs	Makes `NodeTypeBakeGateState` an `IMeshAdmissionAuthority` and defines `ReadinessGranted` + `Admission` derivation.
src/MeshWeaver.Hosting/DynamicTypePreWarmerHostedService.cs	Calls `Reconsider()` on phase edges so held publications are settled promptly as verdict changes.
src/MeshWeaver.Hosting/DynamicTypePreWarmer.cs	Adds second witness (`LocalNodeTypeBuilds`) to recovery watch and triggers gate reconsideration on retraction.
src/MeshWeaver.Compiler.Pipeline/NodeTypeCompilationHelpers.cs	Gates activation-path stamp; emits `LocalNodeTypeBuilds` before possibly-withheld stamp.
memex/Memex.Portal.Shared/MemexConfiguration.cs	Switches module-set adoption from inline boot write to deferred hosted-service registration.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-07-a-portal-refused-by-its-own-checks-no-longer-joins-the-mesh.md	What’s New entry describing the operational behavior change.
src/MeshWeaver.Documentation/Data/Architecture/ModuleSetConvergence.md	Links convergence design to the new admission mechanism.
src/MeshWeaver.Documentation/Data/Architecture/MeshAdmission.md	New architecture doc defining join points, provisional admission, and inert-vs-exit trade-off.

Review details

Suppressed comments (1)

src/MeshWeaver.Mesh.Contract/Services/LocalNodeTypeBuilds.cs:60

Disposing the underlying Subject can race with RecordUsableBuild (it checks disposed then calls OnNext), leading to ObjectDisposedException during mesh teardown. Completing the subject is enough to drop later observations without throwing; avoid disposing it (or guard dispose/OnNext with a lock/Interlocked).

    public void Dispose()
    {
        if (disposed)
            return;
        disposed = true;
        built.OnCompleted();
        built.Dispose();
    }

Files reviewed: 15/15 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        // `hub.ServiceProvider`). Every phase transition below calls Reconsider(): that is the EDGE
+        // the gate's own level-triggered read cannot supply, and it is what releases the stamps a
+        // passing sweep held and discards the ones a failing sweep produced.
+        var admission = mesh.ServiceProvider.GetService<Mesh.Services.MeshPublicationGate>();


+    private readonly Subject<string> built = new();
+    private volatile bool disposed;
+


github-actions · 2026-09-06T22:25:44Z