NodeType watchers ride SubscribeWithReEstablish — no more silent log-and-die control plane by rbuergi · Pull Request #927 · Systemorph/MeshWeaver

rbuergi · 2026-08-08T03:30:50Z

Fixes #891.

The hazard

The three NodeType control-plane watchers (InstallCompileWatcher, InstallSourcesWatcher, InstallReleaseRequestWatcher) subscribed the own stream with Subscribe(onNext, ex => log). One fault terminated the subscription permanently and silently: compiles stopped dispatching, source tracking froze (IsDirty dead, parked-type auto-retry dead), and the Compile button did nothing forever — a hub running half-alive, the wedges-to-zero violation.

Why option 1 (prove no OnError) is off the table

SynchronizationStream.OnError terminally faults the ReplaySubject on legitimate paths (unauthorized, delivery failure, write-path failure), and MeshNodeStreamHandle's content-typing boundary raises per-emission MeshNodeStreamException(Deserialization) faults by design (the GUI renders a typed error card off them).

The fix

All three watchers now subscribe through the canonical SubscribeWithReEstablish — the reviewed post-storm primitive WatchControlPlane/WatchSubmission already use (fault-driven re-establish with terminal discrimination; not a timer/poller watchdog — it reacts only to an actual OnError and stops on terminal classes). Extended with a third fault class:

Fault	Behavior
Poisoned own content (`Deserialization`)	Stop (replay would poison-loop at 1 Hz), log CRITICAL, surface via `onPoisonedContent` — the compile watcher parks the type via `OnCompileFailed(deterministic: true)`, so the failure is user-visible (park notification, diagnostics serve the cached error) instead of one warning line
Own node gone (routing NotFound)	Stop, as before (2026-06-10 storm guard)
Anything else	Transient — re-establish (1 s), fresh subscription

Replay safety on re-establish: compile dispatch is status-guarded (Pending→Compiling atomic), the sources snapshot write is idempotent (DictEquals), and the release watcher's high-water marks live outside the re-subscribed factory so committed triggers fail the IsPast gate — no double compile.

SubscribeWithReEstablish becomes public (it is the watcher primitive), with the fault classes documented.

Verification

New contract tests: PoisonedContent_DoesNotResubscribe_AndReachesTheVisibleSink, IsPoisonedContent_ClassifiesOnlyDeserializationFaults — 5/5 in ActivityControlPlaneResubscribeTest.
Existing watcher semantics pinned by ReleaseRequestWatcherHighWaterTest + CodeEditRecompileTest — 12/12 unchanged.
Release -warnaserror builds clean.
Known coverage gap, stated honestly: no end-to-end test poisons a real persisted NodeTypeDefinition (requires corrupt JSON on an FS-backed mesh); the classification + stop/park semantics are pinned at the helper level and the wiring is exercised by the existing compile-cycle tests.

No What's New entry — latent-hazard resilience fix, no observed user-facing behavior change on healthy paths.

🤖 Generated with Claude Code

…and-die control plane Fixes #891. InstallCompileWatcher, InstallSourcesWatcher and InstallReleaseRequestWatcher subscribed the per-NodeType own stream with a bare log-and-die OnError: one fault killed the subscription permanently and the hub kept running half-alive — compiles no longer dispatched, source tracking frozen, the Compile button silently dead — with a single warning line as the only trace. The issue's option 1 ("prove the stream cannot OnError") is disproven: SynchronizationStream.OnError terminally faults the ReplaySubject on legitimate paths (unauthorized, delivery failure, write-path failure), and the handle's content-typing boundary additionally raises per-emission MeshNodeStreamException(Deserialization) faults. All three watchers now subscribe through the canonical ActivityControlPlaneExtensions.SubscribeWithReEstablish — the reviewed post-storm primitive WatchControlPlane/WatchSubmission already use (on-fault re-establish with terminal discrimination; NOT a timer/poller watchdog) — extended with a third fault class: - Poisoned own content (MeshNodeStreamException/Deserialization): TERMINAL for re-establish purposes — the faulted stream replays the same emission, so re-establishing is a 1 Hz poison loop. Logged CRITICAL and surfaced through the new onPoisonedContent sink; the compile watcher routes it into NodeTypeCompileParkRegistry.OnCompileFailed(deterministic: true), so the type parks with a user-visible notification and diagnostics serve the cached error instead of hanging. Recovery = repair the content, recycle the hub. - Own node gone (routing NotFound): stop, as before (the 2026-06-10 storm guard). - Anything else: transient — re-establish after 1s with a fresh subscription. Replay safety on re-establish: the compile dispatch is status-guarded (Pending→Compiling atomic), the sources snapshot write is DictEquals- idempotent, and the release watcher's high-water marks live outside the re-subscribed factory so a committed trigger fails the IsPast gate. SubscribeWithReEstablish is now public (it is the watcher primitive, not an implementation detail); its doc comment names the three classes. Verified: new contract tests PoisonedContent_DoesNotResubscribe_AndReachesTheVisibleSink + IsPoisonedContent_ClassifiesOnlyDeserializationFaults (5/5 in ActivityControlPlaneResubscribeTest), and the existing watcher behavior pinned by ReleaseRequestWatcherHighWaterTest + CodeEditRecompileTest (12/12) is unchanged. Release -warnaserror builds clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Improves reliability of NodeType control-plane watchers by making their own-stream subscriptions self-healing across transient faults, while correctly treating poisoned content and own-node NotFound as terminal conditions.

Changes:

Route NodeType compile/source/release watchers through ActivityControlPlaneExtensions.SubscribeWithReEstablish instead of bare Subscribe(..., onError: log).
Extend SubscribeWithReEstablish to classify poisoned content (MeshNodeErrorCode.Deserialization) as terminal and optionally surface it via onPoisonedContent.
Add new contract tests covering poisoned-content classification and ensuring poisoned content does not resubscribe but reaches the visible sink.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
test/MeshWeaver.Hosting.Test/ActivityControlPlaneResubscribeTest.cs	Adds tests for poisoned-content terminal behavior and classification.
src/MeshWeaver.Mesh.Contract/ActivityControlPlaneExtensions.cs	Makes `SubscribeWithReEstablish` public, documents fault classes, and adds poisoned-content handling.
src/MeshWeaver.Graph/Configuration/NodeTypeCompilationHelpers.cs	Switches NodeType watchers to use `SubscribeWithReEstablish`, including parking on poisoned content for the compile watcher.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-08-08T03:58:53Z