DataContext's init watchdog is a subscription, not a Task-shaped timer race (#2528) by rbuergi · Pull Request #2606 · Systemorph/MeshWeaver

rbuergi · 2026-08-28T12:04:57Z

Closes #2528.

What

DataContext.OpenInitializationGate's watchdog was the banned timer-race shape: Task.WhenAny(allInit, Task.Delay(InitializationTimeout, cts.Token)).ContinueWith(...) plus a CancellationTokenSource disarm. The wait is now a subscription — Observable.Amb over three one-shot arms, Take(1), ObserveOn(TaskPoolScheduler.Default) (the same off-thread hop the old ContinueWith(TaskScheduler.Default) gave the body), Subscribe with a fluent error arm:

init settled — allInit.ToObservable().Materialize() (Task→IObservable is the sanctioned bridge direction; Materialize so a FAULTED/CANCELED init still runs the settle body instead of skipping to an error arm);
the time-box — Observable.Timer(InitializationTimeout). The bound stays (deleting it re-opens the 2026-06-26 path-resolution-storm wedge); its expiry fails fast and LOUD — fail-level log, InitializationError = TimeoutException, rejection handler — never "proceed as if initialized";
the disarm — an AsyncSubject<Unit> fired by Dispose(). Disarm means fire now, not unsubscribe (an Amb arm, never a TakeUntil): A gate that can never reopen, and an unbounded nested create, can strand a request forever #1270's FailGate answer still runs, Per-node hub DataContext initialization times out at 120s, taking every hub under a user partition dark (root of #3517, #2886, #1313) #1122's no-post-mortem-residue holds, and because the subject replays its terminal, a Dispose that lands before the gate arms now settles immediately (the old CTS shape armed a fresh, never-cancelled timer in that ordering).

The settle body is unchanged — the same four-way shutdown / timed-out / faulted / canceled / clean discrimination, read off allInit's actual state, ending in FailGate (shutdown) or the unconditional OpenGate.

Root cause found on the way: a FAULTED init was indistinguishable from a HUNG one

Writing #2528's requested faulted-init test exposed a pre-existing defect (verified red against main's DataContext too): a data source whose initial load throws never faulted IDataSource.Initialized. The fault stopped in the stream hub's HandleInitialize catch — hub FAILED state, but SynchronizationGate stayed shut, RunLevel never reached Started, Hub.Started never settled — so the owning DataContext saw nothing until its 120 s time-box expired, and then reported a TimeoutException instead of the real error.

Fix at the root: the Task-init arm of SynchronizationStream.Initialize routes its fault through the stream's own OnError — the path every other stream failure already takes — which classifies (teardown stays quiet), faults the store, calls Hub.FailStartup (so Initialized settles NOW with the actual exception) and opens SynchronizationGate. The error still propagates to the buildup's catch for the hub-level FAILED state. Consumed only by the data-source CreateStream path (GenericUnpartitionedDataSource:621), so the blast radius is exactly the path being fixed.

Tests (the four terminal outcomes, per the issue)

hung — DataContextInitTimeoutTest: time-box trips at the configured bound, request answered fast with "initialization failed … did not complete within", InitializationError is TimeoutException;
faulted — DataContextInitFaultedTest: answers in milliseconds carrying the specific init error (red on main: 15 s timeout, the conflation above);
shutdown/disarm — pre-existing DataContextDisposeDuringInitTest (Per-node hub DataContext initialization times out at 120s, taking every hub under a user partition dark (root of #3517, #2886, #1313) #1122) and DataContextShutdownGateAnswersDeferredTest (A gate that can never reopen, and an unbounded nested create, can strand a request forever #1270), both still green;
clean — every AddData round-trip in the suites below.

Verification

-c Release -warnaserror clean for MeshWeaver.Data, MeshWeaver.Graph and the three test projects. Full suites green locally: MeshWeaver.Data.Test 393/393, MeshWeaver.Messaging.Hub.Test 305/305, MeshWeaver.Persistence.Test 1023/1023.

What's New: Category: Fix entry included.

🤖 Generated with Claude Code

…r race (#2528) Task.WhenAny(allInit, Task.Delay(timeout, cts.Token)) + ContinueWith becomes Observable.Amb over three one-shot arms — init settled (Task->IObservable, Materialize'd so a faulted/canceled init still runs the settle body), the time-box (Observable.Timer; the bound STAYS — its expiry fails fast and loud, never 'proceed as if initialized'), and a disarm AsyncSubject fired by Dispose() (fire-now semantics, #1270; no post-mortem FAILED residue, #1122; replays its terminal, so Dispose-before-arm settles immediately instead of arming a never-cancelled timer). Root-caused along the way: a FAULTED initial load was indistinguishable from a HUNG one — the fault stopped in the stream hub's HandleInitialize catch, SynchronizationGate stayed shut, Hub.Started never settled, and the DataContext learned of it only when the 120s time-box expired, as a TimeoutException. The Task-init arm of SynchronizationStream.Initialize now routes its fault through the stream's own OnError (FailStartup + gate-open), so Initialized settles NOW with the real exception and the faulted branch answers in milliseconds. Tests: DataContextInitTimeoutTest (hung init trips the time-box, loud, fast, gate answered), DataContextInitFaultedTest (faulted init reaches the terminal FAILED state fast, carrying the specific error; red on main at 15s timeout). Dispose-mid-init (#1122) and shutdown-gate-answers-deferred (#1270) stay green. Closes #2528 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Refactors DataContext’s initialization watchdog to be fully reactive (subscription-based) rather than a Task.WhenAny / Task.Delay timer race, while preserving the existing time-box and shutdown/disarm behavior. It also fixes a related startup-fault propagation gap so faulted data-source initialization is surfaced immediately (instead of being misreported as a later timeout), and pins the intended terminal outcomes with new tests.

Changes:

Replaced the Task-shaped watchdog with an Observable.Amb-based subscription that races init-settled vs time-box vs Dispose-disarm and always settles the initialization gate.
Ensured a faulted Task-based stream initialization faults the stream (via SynchronizationStream.OnError) so IDataSource.Initialized settles with the real exception.
Added tests covering hung-init timeout and faulted-init fast failure, plus a What’s New entry documenting the change.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
test/MeshWeaver.Data.Test/DataContextInitWatchdogTest.cs	Adds regression tests pinning watchdog outcomes for hung and faulted initialization.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-28-reactive-init-watchdog.md	Documents the user-visible behavior and rationale for the reactive watchdog.
src/MeshWeaver.Data/Serialization/SynchronizationStream.cs	Routes Task-init faults through `SynchronizationStream.OnError` so startup settles with the real exception.
src/MeshWeaver.Data/DataContext.cs	Rewrites the initialization watchdog to a reactive `Amb` subscription with a Dispose-fired disarm signal and extracted settle logic.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-08-28T12:17:08Z