fix(bake-gate): a refusal can only stall a roll — never-built types cannot regress, a serving image never refuses itself (#5544) by rbuergi · Pull Request #5725 · Systemorph/MeshWeaver

rbuergi · 2026-09-26T04:20:00Z

Refs #5544 · Refs #3883 · Refs #4684

What went wrong

After the V58 migration, memex.systemorph.com went fully down (nginx 503). The nodetype_bake gate refused readiness on the new ci.9332 pod. When the last pod of the serving ci.9218 image restarted, the gate refused that pod too. Both refusals named BinaryClickerV2/BinaryToggle. That type has failed CS1929 on every image since it was written (#3883).

There were two defects:

A type that never built was read as a regression. NodeTypeBakeEntry.WasHealthy counts NeverBuilt as healthy. That part is correct: a type nobody has built is not known to be broken. But the regression baseline was built from that field. So on any established instance, a failing never-built type was treated as a regression. The first-bake rule (fix: the first rollout of a new instance has no regression baseline #4472) only covered reports where every type is NeverBuilt.
- The record could never correct this. The Error stamp is a MeshPublicationGate publication, and the gate discards it on a refused pod. The failure refused the pod, so the refused pod could not record the failure. The next pod then read Ok again. That loop is why BinaryClickerV2/BinaryToggle NodeType cannot compile — layout source calls non-existent LayoutAreaHost.GetData<T>() #3883's record read Ok while every compile failed.
A restart of the serving image refused itself. Nothing told a roll candidate apart from a restart of the image the rollout falls back on.

The fix

Everything goes through the one shared stamp, DynamicTypePreWarmer.BaselineStamp. The sweep and BuildProtocolDriver.OutcomesOf both use it, so the GO and the gate still agree (#4496).

Rule 1, per type (NodeTypeBakeEntry.IsRegressionBaselineFor): a failure counts as a regression of this image only if both hold:
- a working build is on record (HadWorkingBuild: not NeverBuilt, not PreviouslyBroken);
- that build was produced by a different, not-newer platform build. The producer is ProducedByPlatformBuild, taken from CompiledPlatformVersion.
- If this same build produced it, the image demonstrably can build the type. If a newer build produced it, this pod is the OLD image of a roll.
- A record with no producer stamp keeps the strict reading.
Rule 2, per report (NodeTypeBakeReport.ThisBuildHasServed): a pod whose build has already been admitted here never refuses itself. Two witnesses establish that, and both are admission-gated publications:
- the durable marker ServedBuildWitness, a storage row at Admin/ServedPlatformBuilds/<build>. It is offered through MeshPublicationGate: held while the bake measures, written on admission, discarded on refusal. The report carries it as ServedBefore.
- a record whose working build this very build produced.
- The marker was added after review. Provenance alone cannot show that an image served: an ordinary roll compiles nothing, and prebuilt adoption keeps the producer's version.
A Faulted sweep on a served image no longer refuses. NodeTypeBakeGateState.ServedBefore relaxes that one verdict. This was the remaining way the gate could take a served instance down.
GateRelevant (the witness-unreadable path) applies both rules.
WasHealthyBeforeBake keeps its DynamicTypePreWarmer: a never-built type reports WasHealthyBeforeBake=false in the satellite lane — core green at the same sha, and it pins every satellite's platform ceiling #4496 meaning. The new answer is carried on HasRegressionBaseline.

Net effect: only a pod strictly newer than the producer of the working build it failed to reproduce can refuse, and only while no replica of its image has been admitted. The older image's pods never meet that condition. Failures that no longer gate are still named in WithoutBaseline. Once the pod is admitted, the held Error stamp is released, which ends the loop.

Tests

TheBakeGateCannotTakeAnInstanceDownTest has 8 deterministic cases, run from a bake report through the real stamp into an armed NodeTypeBakeGateState, with IsGatingFailure asserted alongside. It covers:

the incident shape on the new image;
the restarted old image;
a newer-producer record;
a same-producer record;
an unknown producer;
two negative controls in which a real regression still refuses.

Negative control: with BaselineStamp reverted to the old rule, 3 of the 8 tests fail (the never-built, restarted-old-pod and newer-producer cases). NodeTypeBakeStatusTest gained a probe case that checks the producer and the live build are carried.

AFirstRolloutHasNoRegressionBaselineTest.TheGosOwnProjection_UsesTheSameStamp changed one assertion: a PreviouslyBroken type on an established instance now reads HasRegressionBaseline == false. It was already non-gating through WasHealthyBeforeBake.

AServedBuildWitnessRecordsOnlyAnAdmittedBuildTest runs on real mesh storage. It checks that an admitted pod writes the witness, that a restart reads it, and that a refused pod never writes it. With the publication gate bypassed, both cases fail.

Run locally with Release -warnaserror builds:

Hosting.Test, bake/gate/admission classes: 138 passed.
Compiler.Pipeline.Test, bake classes: 55 passed.
Documentation.Test: 649 passed.

Docs

The new page Architecture/TheBakeGateOnlyStallsARoll is listed in the topic map and linked from Mesh Admission.

Not covered here

A Faulted sweep on a NEW, never-admitted image still refuses readiness. This is by design.
No marker exists yet for images that served before this ships. An image gets its marker once it is admitted with the gate armed.
The roll gate still reads through the startup probe. That wiring lives in the host and chart, not in core.
Re-arming the gate. It goes back on through the deployment records (Memex) once an image carrying this runs on memex and memex-cloud.

Addresses to recycle after deploy: none. This is boot-time sweep logic, so it takes effect on each pod's next start.

Pairs-with: none — no public type or member removed; additions only (init properties and one method on NodeTypeBakeEntry, two properties on NodeTypeBakeReport).

🤖 Generated with Claude Code

…annot regress, a serving image never refuses itself (#5544) memex.systemorph.com went fully down after the V58 migration: the nodetype_bake gate refused readiness on the new ci.9332 pod AND on the restarted pod of the serving ci.9218 image, on BinaryClickerV2/BinaryToggle — a type whose source has failed CS1929 on every image (#3883). Two defects: - A NeverBuilt entry counts as WasHealthy, and the baseline was built from that, so a type that never built read as a regression on any established instance. The Error stamp that would have corrected the record is a MeshPublicationGate publication and is DISCARDED on a refused pod, so the record stayed Ok forever. - Nothing distinguished a restart of the image the rollout falls back on from a roll candidate. Now, through the one shared stamp (BaselineStamp, used by the sweep and the GO): - per type: a failure regresses this image only when a working build is on record and was produced by a DIFFERENT, NOT-NEWER platform build (NodeTypeBakeEntry.IsRegressionBaselineFor, ProducedByPlatformBuild from CompiledPlatformVersion); - per report: an image whose build already stamped a working build here has served, and never refuses itself (NodeTypeBakeReport.ThisBuildHasServed). GateRelevant applies the same rules. WasHealthyBeforeBake keeps its #4496 meaning; the answer rides on HasRegressionBaseline. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Copilot

Copilot review overview

🟡 Changes recommended

The serving-image exemption can fail after adopting and restarting a compatible prebuilt, potentially allowing the serving image to gate readiness.

Review effort: Lite
Findings: 1

Open (1)

Serving-image exemption relies on non-durable producer provenance · New

What changed in this PR

This PR updates NodeType bake-gate regression detection so never-built types and serving images do not incorrectly block readiness.

Changes:

Adds producer-aware regression baselines and serving-image handling.
Expands deterministic rollout and restart test coverage.
Documents the revised gate behavior and limitations.

File	Summary
`test/MeshWeaver.Hosting.Test/TheBakeGateCannotTakeAnInstanceDownTest.cs`	Adds incident, restart, producer, and regression coverage.
`test/MeshWeaver.Hosting.Test/AFirstRolloutHasNoRegressionBaselineTest.cs`	Updates shared baseline assertions.
`test/MeshWeaver.Compiler.Pipeline.Test/NodeTypeBakeStatusTest.cs`	Verifies producer metadata propagation.
`src/MeshWeaver.Hosting/NodeTypeBakeGate.cs`	Updates gate diagnostics.
`src/MeshWeaver.Hosting/DynamicTypePreWarmer.cs`	Applies shared regression stamping.
`src/MeshWeaver.Documentation/Data/Architecture/TheBakeGateOnlyStallsARoll.md`	Documents the gate rules and limitations; unknown-producer exception needs clarification.
`src/MeshWeaver.Documentation/Data/Architecture/MeshAdmission.md`	Links the new architecture documentation.
`src/MeshWeaver.Documentation/Data/Architecture.md`	Adds the page to the topic map.
`src/MeshWeaver.Compiler.Pipeline/NodeTypeBakeStatus.cs`	Adds producer and serving-image tracking; the serving-image exemption is not durable for adopted prebuilt artifacts.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

rbuergi · 2026-09-26T04:32:06Z

+    public bool ThisBuildHasServed =>
+        !string.IsNullOrWhiteSpace(LivePlatformVersion)
+        && Entries.Any(e => e.HadWorkingBuild
+                            && string.Equals(
+                                e.ProducedByPlatformBuild, LivePlatformVersion, StringComparison.Ordinal));


Agreed, and it is worse than the adoption case alone. An ordinary roll compiles nothing, because the compatibility key is equal across every build of one epoch. So in the common case a serving image leaves no record naming itself, and provenance was the wrong witness.

Fixed in d5ecba3:

ServedBuildWitness writes a storage row at Admin/ServedPlatformBuilds/<build>. The row does not depend on per-type provenance.

The armed sweep offers it through MeshPublicationGate. The gate holds it while the bake measures, writes it when the pod is admitted, and discards it when the pod is refused. A bad image therefore cannot exempt itself on its next boot.

The row is read directly through IStorageAdapter, the same way as the claim lock: no hub, and no point read of a missing node. A failed read counts as "not served", which keeps the strict reading.

NodeTypeBakeReport.ServedBefore carries the witness into the shared stamp and into GateRelevant, on both the sweep's report and the build protocol's.

NodeTypeBakeGateState.ServedBefore also relaxes a Faulted sweep on such an image. That was the one remaining way the gate could take a served instance down, since a sweep that errored has no per-type evidence to judge.

Tests:

TheBakeGateCannotTakeAnInstanceDownTest.AnAdmittedImageWhoseOnlyWorkingBuildsAnOlderImageProduced_DoesNotRefuseItself is your scenario. It includes a control showing the same report without the marker still refuses.

AServedBuildWitnessRecordsOnlyAnAdmittedBuildTest runs on real mesh storage and covers two cases: an admitted pod writes the marker and a restart reads it, and a refused pod never writes it. With the publication gate bypassed, both cases fail.

Your second location (line 665, GateRelevant) is covered by the same change.

github-actions · 2026-09-26T04:25:16Z

Test Results (shard 0)

1 files ±0 1 suites ±0 3m 8s ⏱️ -1s
347 tests ±0 347 ✅ ±0 0 💤 ±0 0 ❌ ±0
351 runs ±0 351 ✅ ±0 0 💤 ±0 0 ❌ ±0