Per-leaf delete of an _Access grant can park forever in the permission check — recursive space delete wedges (post-#894, 1-in-N)

Observed on PR #896 run 31161307727 shard 5 (SpaceMenuAndAccessTest.Space_ChildrenAreQueryable), commit includes #894 via main. Intermittent — main's own post-#894 run and #894's PR run were green.

Evidence (teardown snapshot from the failing run)

Hub TestSpace_b29902812e/_Access/Roland_Access RunLevel=Started Disposal=Pending
  Queue(buffer=1, deferred=0, exec=0, deliveryActionCompleted=False)
  Executing(DeleteNodeRequest, 12897ms)

Two per-leaf DeleteNodeRequests ({space}/Overview, {space}/_Access/Roland_Access) pending 10.4 s at the mesh hub; the access-grant leaf hub has been EXECUTING its delete for 12.9 s with the next delivery queued behind the single-threaded action block. The recursive space delete therefore never completes; the test's query wait starves and the class fails with leaked callbacks.

Why this path is new

Before #894, recursive delete never visited _Access satellites at all (the #839 leak) — this per-leaf shape (delete an AccessAssignment on its own per-node hub, mid-space-teardown) first became reachable when the authoritative plan started enumerating satellites. The last-admin validator refusal was fixed with CascadeRootPath (exempt, fast); the remaining park is downstream of validation.

Prime suspect

CheckDeletePermissionForNode → hub.GetEffectivePermissions(pathToCheck, userId).Take(1) where pathToCheck = node.MainNode ?? node.Path — for an access satellite, MainNode is the SPACE ROOT, i.e. the permission fold is evaluated against the very subtree being torn down, mid-delete, on the leaf's own hub. If any source stream of that fold (the $security-access:{scope} cached queries / scope walk) fails to produce its Initial in that state, Take(1) parks forever — the #539/#667 'mid-teardown owner never answers' family, now on the delete path itself.

Per the wedges-to-zero rule the fix is the non-emitting source (or bounding the permission check with an explicit, loud timeout that fails the delete rather than parking the hub) — not a widened budget.

🤖 Generated with Claude Code

Root cause narrowed to one line, with a LOCAL REPRO and a probe-proven stage trace

Local repro (no CI needed): run the WHOLE project — dotnet test test/MeshWeaver.NodeOperations.Test -c Release --no-build --no-restore — fails ~2-3 in 12. The class ALONE never fails (6/6 green, ~1 s), so it needs the other classes in the same process.

Not fixed by #896 — 3/12 failures with that fix merged in. Distinct defect.

Probe-proven stage trace (instrumented run, failing case)

FANOUT-ENTER   {space}                      desc=3
DELETEONE-CALLED {space}/Overview            (root={space})          ← parent posts leaf delete
DELETEONE-CALLED {space}/_Access/Roland_Access (root={space})
LDP-NEXT       {space}/Overview  n=0                                  ← leaf CollectPathsForDelete OK
FANOUT-ENTER   {space}/Overview             desc=1                    ← leaf's own fan-out
DELETEONE-CALLED {space}/Overview            (root={space}/Overview)  ← leaf deletes its OWN row
   …then NOTHING. No drain enumeration, no reply, forever.

Ruled OUT by evidence (all complete in milliseconds): the node read, CheckDeletePermissionForNode (GetEffectivePermissions), the validator chain, and CollectPathsForDelete (emits n=0). The drain-loop verification is never even subscribed.

The defect

The leaf's deleteOne → storage.Delete(path) never emits and never completes, so FanOutDeleteSubtree → DeleteSubtreeUntilDrained → the handler's SelectMany chain never runs and no response is ever posted — the delete can neither succeed nor fail. That is the wedge: the caller's Observe callback stays pending forever (mesh hub shows both leaf DeleteNodeRequests pending; leaf hubs show Executing(DeleteNodeRequest, …) with the next delivery queued behind the single-threaded action block).

Prime suspect, PersistenceService.Delete (src/MeshWeaver.Hosting/Persistence/PersistenceService.cs:158-169):

_writable
    .Select(p => p.Adapter.Read(path, …)          // ← containment probe, UNBOUNDED
        .SelectMany(existing => existing is null
            ? Observable.Return(false)
            : p.Adapter.Delete(path).Select(_ => true)))
    .Merge()
    .Aggregate(false, (any, d) => any || d)       // ← never emits unless EVERY leg completes
    .SelectMany(any => any ? Observable.Return(path) : Observable.Throw(...));

Aggregate emits only on completion of the merged sequence, so one writable provider whose Read never emits/completes parks the entire delete — silently, with no timeout and no error path. An empty completion would still yield false → the explicit Throw (a reply); only a non-completing Read produces the observed total silence.

Next step: identify which writable provider's Read fails to complete for a path that is mid-delete (tombstoned via RecentlyDeletedRegistry.MarkDeleted, per-node hub activating/disposing). Fix that source per wedges-to-zero; the containment probe must also never be able to leave a delete with no reply.

🤖 Generated with Claude Code

Escalating: this now reds unrelated PRs. SpaceMenuAndAccessTest.Space_ChildrenAreQueryable failed on shard 5 of BOTH open security PRs this hour — #905 (run 31168629021) and #907 (run 31168629021 / later run) — neither of which touches the delete path. Combined with the 2-3-in-12 local reproduction rate on the full MeshWeaver.NodeOperations.Test project, this is now a general merge blocker, not an isolated flake.

Priority note for whoever picks this up: the narrowing in my earlier comment stands (leaf deleteOne invoked → storage.Delete never emits and never completes → no reply ever posted → caller parks). Start at PersistenceService.Delete (src/MeshWeaver.Hosting/Persistence/PersistenceService.cs:158-169) — the per-provider Read containment probe under .Merge().Aggregate(...), where Aggregate emits only once EVERY leg completes, so a single non-completing Read parks the whole delete with no timeout and no error path.

🤖 Generated with Claude Code

Cross-linking #898 — same wedge, and the timeout raise there did NOT fix it.

#898 ("Space delete-cleanup wait sat below the delete operation's own 30s contract", MERGED) raised SpaceMenuAndAccessTest's wait 10s → 30s. That is a band-aid over this defect: the child DeleteNodeRequest does not complete at 32s either — it never completes, because two concurrent per-node deletes deadlock (see below). Raising the bound only changes how long the test waits before reporting the same wedge, which is why the failures continued on #905 and #907 after #898 merged.

Root cause is now proven and fixed in PR #910: changeFeed.Publish fans out SYNCHRONOUSLY on the publisher's thread, and CheckDeletePermissionForNode runs the entire delete inside GetEffectivePermissions' CombineLatest gate (its ReplaySubject sources emit during Subscribe). Two concurrent per-node deletes then cross-lock: each holds its own fold gate and waits on the shared synced-query Merge gate. dotnet-stack on the wedged process shows exactly those two blocked threads at Monitor.Enter under InProcessMeshChangeFeed.Publish. The lead recorded on #898 (the has-children query in the non-recursive re-entry) is disproven — CollectPathsForDelete emits n=0 in milliseconds.

Follow-up once #910 lands: the 30s wait introduced by #898 should be reverted to the original budget. With the deadlock gone the delete completes in milliseconds, and leaving an inflated bound in place would hide the next regression in exactly this path.

🤖 Generated with Claude Code

Merge pull request #918 from Systemorph/fix/899-permission-fold-gate

fix(catalog): sealing-key work runs outside the permission fold's gate

fix(graph): mail copy fails closed without a mesh at the call; permis…

feat(security): sealed secret envelope and the control instance's sea…

feat(graph): the provider-neutral opt-in mail copy lives in core (#7133)

fix(#7911): the account-lock register's standing read leases no modul…

Navigation Menu

Repository navigation

Per-leaf delete of an _Access grant can park forever in the permission check — recursive space delete wedges (post-#894, 1-in-N) #899

Evidence (teardown snapshot from the failing run)

Why this path is new

Prime suspect

Root cause narrowed to one line, with a LOCAL REPRO and a probe-proven stage trace

Probe-proven stage trace (instrumented run, failing case)

The defect

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Per-leaf delete of an _Access grant can park forever in the permission check — recursive space delete wedges (post-#894, 1-in-N) #899

Description

Evidence (teardown snapshot from the failing run)

Why this path is new

Prime suspect

Activity

rbuergi commented on Aug 7, 2026

Root cause narrowed to one line, with a LOCAL REPRO and a probe-proven stage trace

Probe-proven stage trace (instrumented run, failing case)

The defect

rbuergi commented on Aug 7, 2026

rbuergi commented on Aug 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions