Thread wedges at StartingExecution forever when the _Exec commit is lost (e.g. NodeType release recycles the hub mid-claim) — no recovery on activation #539

Symptom

A thread round submitted via hub.StartThread(...) can wedge at Status = StartingExecution forever: the user message stays in pendingUserMessages, no cell is allocated, no agent round runs, and nothing ever recovers it. The thread only executes when its hub is explicitly re-activated (e.g. a DisposeRequest recycle of the main node followed by an access).

Production evidence (memex portal, UW Deepfield Industry News auto-update)

Two gather rounds (agent Assistant, model DeepSeek-V4-Flash, submitted through the canonical hub.StartThread surface from a layout-area click):

Trigger correlation — both wedged rounds were submitted ~60–70 s after a NodeType compile + release recycled the main-node hub:

• Release 20260716052635 at 05:26:35 → thread submitted 05:27:46 → wedged.
• Release 20260718214041 at 21:40:41 → thread submitted 21:41:11 → wedged.

(Opening the feed triggers the stale-assembly recompile/release, which recycles the per-node hub; the user's toggle click then lands right on that boundary.)

The model was exonerated: both rounds executed quickly and correctly on DeepSeek-V4-Flash once the hub was re-activated.

Root cause

The round dispatch is a two-edge handshake:

1. Claim — InstallSubmissionWatcher (on the thread hub's action block) writes Idle → StartingExecution.
2. Commit — the _Exec round watcher's first emission commits StartingExecution → Executing and launches the round.

When the commit edge is lost — here: the _Exec startup raced the release-recycle of the freshly claimed thread's hub environment — the claim survives as durable node state but no recovery path exists. InitializeThreadLifecycle (src/MeshWeaver.AI/ThreadExecution.cs) self-heals every other state on activation:

• Executing + ActiveMessageId → resume the interrupted round;
• Executing without a cell → reset to Idle so the submission watcher re-claims;
• Idle / Cancelled + pending → submission watcher claims;
• StartingExecution → deliberately no write: "the _Exec round watcher fires on its own first emission."

That assumption doesn't hold when the claim was made by a previous hub incarnation (or _Exec never came up): nothing activates _Exec, so the "first emission" never happens, and the thread is parked in a state that ThreadStateTransitions treats as in-flight — resubmission is refused, the submission watcher won't re-claim. The cancel path already acknowledges this exact window (the "No-CTS claim-window fallback" writes terminal Cancelled directly when Status is stuck at StartingExecution), but normal resumption has no equivalent.

Needed fix (root cause — no timers/watchdogs)

On thread-hub activation, StartingExecution must not be a no-op. Activation is proof that no round is actually in flight (executions die with their hub), so the recovery can act deterministically, mirroring the existing Executing branches:

• either drive the commit: ensure the _Exec hub is started and its round watcher observes the claimed state (so the claim → commit → launch chain completes), or
• roll the orphaned claim back (CAS-guarded, same idempotent shape as the Executing-without-cell reset): StartingExecution → Idle, letting InstallSubmissionWatcher re-claim the still-queued pendingUserMessages.

Care needed to not reintroduce the re-dispatch ping-pong documented in ThreadStateTransitions (Executing → StartingExecution stays illegal; this is only about an orphaned claim on a freshly activated hub, before any commit).

Deterministic repro to pin the fix (per WritingTests.md): submit via StartThread, dispose the hub after the claim write lands but before _Exec commits, re-activate the hub, assert the round executes and pendingUserMessages drains — observed via GetMeshNodeStream(threadPath).Where(...).FirstAsync().Timeout(...).

Impact

Any StartThread submission racing a hub recycle (NodeType release, redeploy, grain migration) silently never runs; from the product side it reads as "the agent ignored me". In UW Deepfield the feed's 6-minute stale-guard clears the UI, so the failure is doubly silent — the round just never happens. A 2.5-day wedge was observed in production with the page being reopened repeatedly in between; only an explicit recycle recovered it.

Pointers

• src/MeshWeaver.AI/ThreadExecution.cs — InitializeThreadLifecycle recovery branch table (the StartingExecution no-op), InstallSubmissionWatcher, _Exec round-watcher commit edge, No-CTS claim-window cancel fallback.
• src/MeshWeaver.AI/ThreadStateTransitions.cs — legal edges; StartingExecution counts as in-flight, so nothing re-claims it.
• Field diagnosis thread: memex UWDeepfield/IndustryNews auto-update, 2026-07-18/19.