Self-heal a stuck compile overlay even when the type's version never moves — and tell admins when it can't by rbuergi · Pull Request #681 · Systemorph/MeshWeaver

rbuergi · 2026-07-27T21:19:05Z

The outage

memex went down at 20:07 UTC, 2026-07-27. A CD deploy (ci.1451) replaced both portal pods. Every NodeType had to recompile on first activation; Store/Plugin missed the 60s settle window, so sixteen plugin roots — AgenticEngineering, Claims, DataModelling, Reinsurance, Underwriting, Chess, … — bound the "build did not settle" overlay.

The recompile then succeeded at 20:47:35. The pages stayed broken anyway. Five manual recycles changed nothing. Only a scale-to-zero restored the site.

Root cause — one clause

&& (typeVersionAtOverlay is null || t.Version > typeVersionAtOverlay.Value)

The self-heal fires only when the NodeType node's version advances. But recompiling an already-Ok type does not rewrite its node — Store/Plugin sat at v796, last modified 08:34, untouched all day, while its assembly rebuilt fine.

So the watcher waited for a bump that could never come, and every re-activation re-armed the same unsatisfiable predicate. Sticky forever, by construction — and that is precisely why recycle was useless and only a pod restart worked.

The fix

The version gate was deliberate (it prevents a recycle → re-overlay → recycle hot loop), so it stays. A second route is added: a usable build at an unchanged version heals after a grace window.

Never instant — the delay is the anti-hot-loop property, so an instance that still cannot build recycles at most once per window instead of spinning. All four original contracts pass unchanged.

And it was invisible

The overlay text appears zero times in Loki across the entire window. No notification was raised. The first signal was a person reporting a dead page.

An instance still serving the fallback past the heal window is a platform fault, so it now raises one admin-scoped notification naming the type and the instance — anchored under the Admin partition, whose RLS scopes it to platform admins (the same anchoring StartupErrorNotifier and NodeTypeCompileParkRegistry already use). Wrapped whole: reporting can never throw back onto the render path.

Tests

Two new cases drive time with a TestScheduler (no sleeping) and pin both halves:

a usable build at an unchanged version heals after the grace — and not before;
a type still unusable when the window elapses is not recycled, so the overlay keeps telling the truth until a build actually lands.

638/638 Graph tests pass.

What this does NOT fix

Honest scope — this stops the stuck state, it does not stop the cold state:

No readiness gate. Pods still serve traffic while types are cold, so the first users after a deploy can still meet a fallback — now self-healing within the window instead of until a restart. A boot pre-warm that holds readiness until hot types have a usable build is the change that removes the window entirely.
SlowPathTimeout is a hard-coded 60s, which turns a legitimately slow compile into a broken page.
DeactivateOnIdle fires mid-activation — the activate → 60s subscribe timeout → deactivate → re-activate thrash (38 events in the window) is its own bug.
Compile events aren't logged at all, which is why the RCA needed the database rather than Loki.
Modelling this as a first-class Activity (rather than a background subscription plus a notification) is not done here.

🤖 Generated with Claude Code

…moves — and tell admins when it can't memex went down at 20:07 UTC today. A CD deploy replaced both portal pods; every NodeType had to recompile on first activation; Store/Plugin missed the 60s settle window, so sixteen plugin roots — AgenticEngineering, Claims, DataModelling, Reinsurance, Underwriting, Chess and the rest — bound the "build did not settle" overlay. The recompile then SUCCEEDED at 20:47:35. The pages stayed broken anyway, five manual recycles changed nothing, and only a scale-to-zero restored the site. The reason is one clause: && (typeVersionAtOverlay is null || t.Version > typeVersionAtOverlay.Value) The self-heal only fires when the NodeType node's VERSION ADVANCES. But recompiling an already-Ok type does not rewrite its node — Store/Plugin sat at v796, last modified 08:34, untouched all day. So the watcher waited for a bump that could not come, and each re-activation re-armed the same unsatisfiable predicate. Sticky forever, by construction. That version gate was deliberate: it stops a recycle → re-overlay → recycle hot loop. So this keeps it and adds a second route. A usable build at an UNCHANGED version now heals after a grace window. Never instant — the anti-hot-loop property is exactly the delay — so an instance that still cannot build recycles at most once per window instead of spinning. All four original contracts still pass unchanged. The second half is that the outage was INVISIBLE from the inside. The overlay text appears zero times in Loki across the whole window; no notification was raised; the first signal was a person reporting a dead page. An instance still serving the fallback past the heal window is a platform fault, so it now raises one admin-scoped notification naming the type and the instance — anchored under the Admin partition, whose RLS scopes it to platform admins, the same anchoring StartupErrorNotifier and NodeTypeCompileParkRegistry already use. Wrapped whole: reporting can never throw back onto the render path. Two new tests drive time with a TestScheduler and pin both halves: a usable build at an unchanged version heals after the grace and only then, and a type that is still unusable when the window elapses is NOT recycled — the overlay keeps telling the truth until a build lands. 638/638 Graph tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR addresses a production failure mode where the “build did not settle” overlay could become permanently sticky after a successful recompile that does not update the NodeType MeshNode version (i.e., recompiling an already-Ok type). It extends the overlay self-heal watcher with a grace-based healing path and adds an admin-scoped notification when self-healing fails.

Changes:

Extend ArmOverlaySelfHeal to heal via either version advance (existing behavior) or a grace-window re-read when the version never moves.
Add a delayed admin notification path when an instance remains stuck on the fallback beyond the healing window.
Add deterministic unit tests using TestScheduler to cover both “heals after grace” and “does not recycle while still unusable”.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
test/MeshWeaver.Graph.Test/OverlaySelfHealWatcherTest.cs	Adds scheduler-driven tests covering the grace heal path and the “still unusable → no recycle” contract.
src/MeshWeaver.Graph/Configuration/NodeTypeEnrichmentHelpers.cs	Implements grace-based self-heal and delayed admin notification for stuck overlays.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+            NotificationService.Dispatch(
+                instanceHub,
+                recipient: null,
+                mainNodePath: StartupErrorNotifier.AdminPartition,
+                title: $"Type '{nodeType}' is serving a fallback page",
+                message:
+                    $"The instance '{instance}' has been rendering the \"build did not settle\" "
+                    + $"fallback for over {StuckReportDelay.TotalSeconds:0}s and did not self-heal. "
+                    + "Its NodeType has no usable build on this pod. Users see a broken page until "
+                    + "the type builds or the instance is recycled.",
+                type: NotificationType.System,
+                targetNodePath: nodeType);
+            logger?.LogWarning(
+                "Overlay self-heal: instance '{InstancePath}' still stuck on NodeType '{NodeType}' after {Delay}s — platform admins notified",
+                instanceHub.Address, nodeType, StuckReportDelay.TotalSeconds);
+        }


github-actions · 2026-07-27T21:34:12Z