DataContext's init watchdog is a Task-shaped timer race (Task.WhenAny + Task.Delay + ContinueWith) — make the wait reactive without deleting the time-box #2528

Split out of #2488 / #2527, where it was assessed and deliberately NOT folded in.

The site

src/MeshWeaver.Data/DataContext.cs:449, in OpenInitializationGate:

watchdogCancellation = new CancellationTokenSource();
Task.WhenAny(allInit, Task.Delay(InitializationTimeout, watchdogCancellation.Token))
    .ContinueWith(_ => { … }, TaskScheduler.Default);

A timer race deciding whether initialization happened — the shape the /async skill's Rule 1a
rejects, and a Task-as-a-gate in src/.

🚨 It is NOT the #2301 deadlock — three independent structural reasons

Recorded so nobody re-attributes it later:

1. It blocks no thread. It hangs a ContinueWith on TaskScheduler.Default and returns
   immediately, so it cannot park a hub action block or a grain turn. Flake: OrleansGrainTeardownStragglerTest — the activation never leaves the silo catalog (4 hits on 2026-08-25, all unrelated diffs) #2301's mechanism was an
   inline ToTask() resumption onto the deactivating grain's own turn scheduler.
2. The budget does not match. InitializationTimeout is 120 s; every Flake: OrleansGrainTeardownStragglerTest — the activation never leaves the silo catalog (4 hits on 2026-08-25, all unrelated diffs) #2301 occurrence
   stalled at exactly 30 s, which is that test's own Timeout operator.
3. It is not in the trace. Flake: OrleansGrainTeardownStragglerTest — the activation never leaves the silo catalog (4 hits on 2026-08-25, all unrelated diffs) #2301's failures stall between "Hub disposal completed" and catalog
   departure, with no initialization gate involved.

…and the banned SHAPE is not (here) the banned BEHAVIOUR

Rule 1a's objection to a timer race is "what if the wait expired? and the tempting answer — do it
anyway — is exactly how a collectible AssemblyLoadContext gets unloaded out from under live code."
This site does not do it anyway. On expiry it records a TimeoutException, registers the
all-requests rejection handler, errors the data-source streams and opens the gate — the deliberate
wedges-to-zero treatment. So this is a conformance defect (a Task gate where the framework is
IObservable end-to-end), not a live incident.

What a fix must preserve — read the comments before changing the shape

Three production incidents shaped the current behaviour, and a naive Rx rewrite loses at least one
of them:

• Per-node hub DataContext initialization times out at 120s, taking every hub under a user partition dark (root of #3517, #2886, #1313) #1122 — the watchdog must be DISARMED by Dispose(). A transient probe hub
  ($model-probe / $schema-probe, alive for one read) previously had its 120 s timer fire 90 s
  after the hub was gone, stamping a fail-level "TIMED OUT … FAILED state" on nothing.
• A gate that can never reopen, and an unbounded nested create, can strand a request forever #1270 — 🚨 the disarm must still RUN THE BODY, not skip it. On Hub.IsShuttingDown the
  continuation calls FailGate(InitializationGateName, …), which is the terminal ANSWER to
  everything already parked behind that gate. Returning without it left the gate shut forever
  and stranded every deferred delivery until an unrelated 30 s deadline. So cancellation here means
  "fire now", not "unsubscribe" — an Amb over three sources, never a TakeUntil.
• 2026-06-26 — the time-box itself exists because a HUNG init (a Task.WhenAll that never
  completes) left the gate closed and every subsequent message deferred → NACKed → resubscribed, a
  path-resolution storm that GC-thrashed the portal. Do not delete the bound.

Plus the four-way discrimination the body performs today: shutting-down / timed-out / faulted /
cancelled / clean, each with a different log level and outcome, and the unconditional
Hub.OpenGate at the end.

Suggested shape

Amb over (a) allInit.ToObservable().Materialize(), (b) Observable.Timer(InitializationTimeout),
(c) a disarm subject fired by Dispose() — Take(1), then the existing body with the outcome as
data. Every arm produces a value, so there is exactly one terminal and no branch that can be missed.

Needs its own tests: a hung init, a faulted init, a dispose-mid-init (asserting FailGate fires and
no FAILED residue is recorded), and a clean init.

🤖 Generated with Claude Code