fix(compile): terminal-settle hung source snapshots + land forward release triggers over stale bases by rbuergi · Pull Request #583 · Systemorph/MeshWeaver

rbuergi · 2026-07-20T23:13:37Z

Compile pipeline wedge on GitSync source-update burst (memex-cloud 2026-07-20, Store/Plugin)

Incident

A GitSync burst (MeshWeaver.Plugins PR #109) updated the sources of many dynamic NodeTypes at once. Store/Plugin got stuck at compilationStatus: "Compiling": its compile activity stopped after exactly two messages ("Compile started" / "Invoking compiler…"), no Roslyn run and no error, ever. The CompileWatcher recovery (Compiling→Pending flip) fired and a second compile start was stamped — but that compile hung the same way. An explicit compile trigger during the stuck period produced no compile and no node write. A recycle during the burst didn't help; a recycle with the system quiet made the next compile succeed in 300ms.

Root cause

1. The hung source snapshot → absorbing Compiling wedge (the primary defect).

RunCompile creates the activity ("Invoking compiler…") and then subscribes the compile pipeline, whose first stage is a ONE-SHOT source-snapshot read: ResolveSources(...).Take(1) over the shared NodeSources.GetSources synced query (MeshNodeCompilationService.DiscoverSourceMaxLastModified / CompileCore). The synced query is a LIVE, never-completing collection gated on its Initial — when that Initial is lost racing a source-update burst, the bare .Take(1) waits forever: no timeout, no error path, no terminal write. And the wedge is absorbing:

the compile watcher needs a Pending transition (DistinctUntilChanged + Where(Pending));
InstallReleaseRequestWatcher gates on a settled status (not Pending and not Compiling), so every explicit trigger sits unprocessed forever;
the recovery kickoff is Take(1)-activation-one-shot — a recycle just re-rolls the dice (mid-burst it re-wedged; quiet it succeeded in 300ms).

That is exactly "no compile ever executed after the flip". Why Store/Order compiled (~45s) while Store/Plugin wedged: the wedge needs a specific per-subscription race — the compile's snapshot subscription missing the query's Initial mid-burst — not bad source content (Order compiled the same new shared sources fine).

2. The MergeGuard refused legitimate FORWARD compile triggers (the incident's dead explicit trigger).

The three-way MeshNodePatchMerge refuses every conflicting scalar (live changed since the writer's base). Correct for non-monotonic fields — but during a burst every mirror's base lags the owner, so a cross-hub RequestedReleaseAt patch carrying a genuinely newer trigger arrives with base ≠ live and is refused: the user's compile request is silently lost (the observed [MergeGuard] … refused stale/reordered cross-hub write lines are this mechanism firing on burst writes). Note the recovery flip itself is not MergeGuard-refused — it is a local UpdateOwn on the owner; the refusal killed the explicit remote trigger path.

Fix (no watchdogs, no retries)

The compile pipeline is now total — every dispatched compile reaches exactly one terminal status:

MeshNodeCompilationService.SnapshotSources — all four one-shot snapshot reads (DiscoverSourceMaxLastModified, DiscoverSourceVersionSnapshot, CompileCore, GetCompilationInputsAsync) are bounded (CompilationCacheOptions.SourceSnapshotTimeout, default 30s; the healthy path is instant off the Replay(1) cache) and error loudly, naming the dead source query. The error propagates through RunCompile's existing Catch to a terminal Status=Error write + bounded park accounting (non-deterministic failure ⇒ retry-bounded, then parked with a user notification). A fresh RequestedReleaseAt / Compile button retries cleanly.
NodeTypeCompilationHelpers.RunCompile — DefaultIfEmpty totality guards on the compile-outcome and release-create subscriptions: an empty completion settles the state machine with a terminal Error instead of silence. Same completion-guard contract MeshQuery.MergeProviderObservables already enforces for provider Initials.

The explicit trigger is a legal write again (rebased on current state):

DataExtensions.RebaseMonotonicTriggers — for the ONE contractually strictly-increasing control trigger (RequestedReleaseAt), the scalar conflict IS mergeable: a forward value is rebased onto the live state and lands; a backward (flapped) value is still dropped; every other scalar conflict (CompilationStatus, IsDirty, …) keeps the refusal that protects the compile single-flight.

Repro tests (red before the fix, green after)

CompileSourceSnapshotWedgeTest.HungSourceSnapshot_SettlesTerminalError_ThenFreshTriggerRecovers (MeshWeaver.Hosting.Monolith.Test) — a query provider withholds the Initial for exactly the NodeType's source query (the deterministic stand-in for the raced/lost synced-query Initial; MeshQuery.MergeProviderObservables gates its merged Initial on every provider by design). Pre-fix: the NodeType sticks at Compiling forever (assertion times out — the incident wedge). Post-fix: terminal Error naming the snapshot within the bound, then — after the query heals — a fresh explicit trigger runs a real compile that settles Ok with a release.
MonotonicTriggerMergeTest (MeshWeaver.Data.Test) — pins the monotonic merge on the exact internal seam the PatchDataRequest handler runs: forward-over-stale-base lands (pre-fix: refused/lost), backward drops, clean-base applies, non-monotonic scalar conflicts still refused.

Verification

Red run: reverted the src fix → wedge test fails with "did not emit within 40s" (stuck Compiling); ForwardTrigger_OverStaleBase_LandsNewerTrigger fails (trigger refused, live value kept).
Green run: both repro tests pass with the fix (wedge test 8s end-to-end).
dotnet build MeshWeaver.slnx -c Release -warnaserror: 0 warnings, 0 errors.
Surrounding suites (Release, single run): MeshWeaver.Data.Test, MeshWeaver.Query.Test, MeshWeaver.Graph.Test, and the Monolith compile-pipeline subset (CodeEditRecompile / ReleaseRequestWatcher / CompileLeaf / NodeTypeRelease / MeshHubRemoteStream / BrokenNodeTypeAccess).

🤖 Generated with Claude Code

…elease triggers over stale bases Root-causes the memex-cloud 2026-07-20 Store/Plugin compile wedge (GitSync burst, image 3.0.0-ci.1255): a dispatched compile flipped Pending->Compiling, created its activity ("Compile started" / "Invoking compiler..."), then parked FOREVER on its one-shot source-snapshot read - ResolveSources(...).Take(1) over the shared NodeSources.GetSources synced query - because the query's Initial was lost racing the burst. The wedge is ABSORBING: the compile watcher needs a Pending TRANSITION, InstallReleaseRequestWatcher gates on a SETTLED status, and the recovery kickoff is activation-one-shot, so no later trigger could ever start a compile again ("no compile ever executed after the flip"). Fix 1 - the compile pipeline is now TOTAL (wedges-to-zero): - MeshNodeCompilationService.SnapshotSources bounds every one-shot snapshot read (CompilationCacheOptions.SourceSnapshotTimeout, default 30s; healthy path is instant off the Replay(1) cache) and errors LOUDLY naming the dead source query. The error propagates to RunCompile's Catch -> terminal Status=Error write + bounded park accounting; a fresh RequestedReleaseAt / Compile button retries cleanly. - RunCompile gets DefaultIfEmpty totality guards on the compile-outcome and release-create subscriptions: any empty completion settles the state machine with a terminal Error instead of silence - the same completion-guard contract MeshQuery.MergeProviderObservables enforces for provider Initials. Fix 2 - the explicit trigger is a legal write again: - During the burst every mirror's base lagged the owner, so the three-way MergeGuard REFUSED cross-hub RequestedReleaseAt patches as conflicting scalars - silently losing legitimate NEWER compile triggers (the incident's dead explicit trigger + "[MergeGuard] refused stale/reordered" lines). DataExtensions.RebaseMonotonicTriggers resolves the ONE contractually strictly-increasing trigger monotonically: a forward value is rebased onto the live state and lands; a backward (flapped) value is still dropped; every other scalar conflict (CompilationStatus etc.) keeps the refusal. Repro tests (red before, green after): - CompileSourceSnapshotWedgeTest.HungSourceSnapshot_SettlesTerminalError_ ThenFreshTriggerRecovers - withholds the source query's Initial (the raced-burst stand-in), pre-fix the NodeType sticks at Compiling forever; post-fix it settles terminal Error naming the snapshot, then a fresh explicit trigger recompiles Ok once the query heals. - MonotonicTriggerMergeTest - pins forward-lands / backward-drops / clean-base / non-monotonic-still-refused on the exact internal merge seam. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Addresses a compile pipeline wedge observed during a GitSync source-update burst by making source snapshot acquisition and compile/release settling total (no silent hangs) and by allowing strictly-monotonic release triggers to merge forward even when the writer’s base is stale.

Changes:

Bound one-shot source snapshot reads with a configurable timeout and surface a loud terminal error instead of parking indefinitely at CompilationStatus=Compiling.
Add compile/release “totality guards” (DefaultIfEmpty) so empty completions still settle the NodeType to a terminal state.
Allow forward RequestedReleaseAt triggers to land over stale bases via owner-side rebase logic, with deterministic unit test coverage.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/MeshWeaver.Hosting.Monolith.Test/CompileSourceSnapshotWedgeTest.cs	Adds deterministic repro asserting hung snapshot now settles to terminal Error, then recovers on a fresh trigger.
test/MeshWeaver.Data.Test/MonotonicTriggerMergeTest.cs	Adds unit tests pinning forward-monotonic trigger rebase semantics at the owner merge seam.
src/MeshWeaver.Graph/Configuration/NodeTypeCompilationHelpers.cs	Adds `DefaultIfEmpty` guards to ensure compile/release pipelines always produce a terminal outcome.
src/MeshWeaver.Graph/Configuration/MeshNodeCompilationService.cs	Introduces bounded `SnapshotSources(...)` and routes one-shot source reads through it.
src/MeshWeaver.Graph/Configuration/CompilationCacheOptions.cs	Adds `SourceSnapshotTimeout` option (default 30s) to control snapshot bounds.
src/MeshWeaver.Data/Workspace.cs	Exposes merge internals to `MeshWeaver.Data.Test` via `InternalsVisibleTo` for seam-accurate tests.
src/MeshWeaver.Data/DataExtensions.cs	Makes merge seam internal, adds monotonic-trigger rebase step before three-way merge.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+            && baseVal is not null
+            && !System.Text.Json.Nodes.JsonNode.DeepEquals(baseVal, liveContent[triggerKey]))
+        {
+            logger?.LogInformation(


+    /// <summary>
+    /// 🚨 Monotonic-trigger resolution for the THREE-WAY merge path (base values carried).
+    /// <see cref="Serialization.MeshNodePatchMerge.Apply"/> refuses EVERY conflicting scalar
+    /// (live changed since the writer's base) — the right default for non-monotonic fields
+    /// (Status, IsDirty, …) where two concurrent writes are not mergeable. But for the ONE
+    /// strictly-increasing control trigger — <c>NodeTypeDefinition.RequestedReleaseAt</c>, which
+    /// every writer sets to <c>DateTimeOffset.UtcNow</c> and which contractually never moves
+    /// backward — the scalar conflict IS mergeable: the newest instant wins, regardless of what
+    /// base the writer diffed against. Refusing the FORWARD case silently swallows a legitimate
+    /// compile trigger whenever the writer's mirror lags the owner (guaranteed during a GitSync
+    /// burst) — the memex-cloud 2026-07-20 incident where every explicit Store/Plugin compile
+    /// trigger was dropped. Resolution, applied to the parsed patch/base before the merge:
+    /// <list type="bullet">
+    ///   <item>Patch BEHIND live → drop the key from the patch (the flap guard — identical
+    ///     outcome to the refusal, but with the precise monotonic log line).</item>
+    ///   <item>Patch AHEAD of live with a stale base → REBASE the writer's base value to the
+    ///     live value so the three-way merge sees "unchanged since base" and lands the newer
+    ///     trigger. This is the "make the flip a legal write — rebased on current state" fix,
+    ///     not a bypass: only this one contractually-monotonic field gets it.</item>
+    /// </list>
+    /// </summary>


github-actions · 2026-07-20T23:30:30Z