Group-membership revocation never reaches the live node-open path — removed users keep reading protected data until portal restart #697

Summary — the governance problem

When an administrator removes a user from a group, that user keeps seeing the protected data — indefinitely, until the portal process is restarted.

• An HR manager removes an employee from the HRTeam group, believing access is revoked.
• The removed user can still open every HR record (dates of birth, home addresses) exactly as before — not for 30 seconds, not for 5 minutes: until someone restarts the portal.
• The situation is deceptive for the administrator: search and listings revoke immediately, and the Effective Access view shows the user as having no access. Everything the admin can check says "revoked" — while direct access to the records still works. Nothing signals that revocation didn't take.

This is a revocation failure in access control: granting access works, restricting a namespace works, but taking access away from a person does not reach the part of the system that serves the actual content. The system fails open. It bites in exactly the scenarios access control exists for: role changes, offboarding, a mistaken group addition.

Contract being broken

Doc/Architecture/AccessControl.md: permissions resolve from a live, reactive cache with change-notifier push — "no storage walks, no TTLs", "End-to-end propagation is on the order of the change-notifier tick (low milliseconds)". The evaluator resolves a user's groups from the process-wide synced query over all GroupMembership nodes ($security-memberships, PermissionEvaluator.ObserveAllMembershipNodes).

Reproduction

Setup (all shapes per Doc/Architecture/GrantingAccess.md; verified compliant — and identical to the platform's own store-gating pattern):

1. A partition readable to the user (e.g. Manufacturing, user holds Admin at partition root).
2. A restricted sub-namespace Manufacturing/HR:
   • Manufacturing/HR/_Access/Public_Access and Anonymous_Access — roles: [{role: "Viewer", denied: true}], mainNode: "Manufacturing/HR"
   • Manufacturing/HR/_Policy — {"$type": "PartitionAccessPolicy", "breaksInheritance": true}
   • Manufacturing/HR/_Access/HRTeam_Access — accessObject: "Manufacturing/HRTeam", role Editor
3. Group Manufacturing/HRTeam (content $type: AccessObject) with one membership: Manufacturing/HRTeam/{user}_Membership (member = login userId).
4. Protected nodes under Manufacturing/HR/… (custom nodeType, not a public-read type).
5. Restart the portal once so the boot snapshot contains everything. Confirm the user reads Manufacturing/HR/{record} (via the group; breaksInheritance discards the inherited root Admin).

Then:

6. Delete Manufacturing/HRTeam/{user}_Membership.
7. Immediately check Postgres (both correct within seconds):
   • SELECT count(*) FROM manufacturing.mesh_nodes WHERE path = 'Manufacturing/HRTeam/{user}_Membership'; → 0
   • SELECT count(*) FROM manufacturing.user_effective_permissions WHERE user_id = '{user}' AND node_path_prefix LIKE 'Manufacturing/HR%'; → 0 (trigger-driven rebuild fired)
8. Wait 6.5 minutes with zero access attempts — hands-off, because the per-scope security cache is a sliding Replay(1).RefCount window (AccessControl.md), and polling refreshes exactly the entry under test.
9. As the removed user, open Manufacturing/HR/{record} (single read).

Observed: the read succeeds — full record content returned, 6.5+ minutes after deletion, beyond the 30 s per-(path, user) probe cache (MeshNodeStreamCache) and the sliding 5-minute scope cache.
Expected: denied within the change-notifier tick, at worst within the documented cache windows.

Control (same session, same identity, proving the read path is gated and the identity correct): a read of Admin/_Access — a scope the user holds no grant on — is correctly denied (fail-closed as Not found).

Evidence by layer

Diagnosis

The removal delta is not delivered to (or not applied by) the process-wide $security-memberships query snapshot; PermissionEvaluator keeps counting the user as a group member until process restart re-materialises the snapshot. Grant-side behavior suggests the same delta-delivery gap for other freshly changed $security-* nodes (assignments, policies) — worth verifying while fixing. A corollary observed in the same session: re-creating the membership also has no live effect, so a remove-then-re-add sequence leaves the stale snapshot silently authoritative in both directions.

Impact

• Group-membership revocation has unbounded lag on node opens (the path that serves content) while listings already deny — the admin-visible surfaces confirm a revocation that is not in force.
• Fail direction is open.
• Scope: the live process only; any restart applies the correct state. But "revocation requires a portal restart" is not an acceptable security property, and nothing tells the admin a restart is needed.

Side anomaly (verified separately — do NOT assume the same root)

A freshly created GroupMembership node never appears in search nodeType:GroupMembership, while pre-existing membership nodes in another partition do appear. Verified in Postgres that this is not an access-control result: the row exists in {schema}.mesh_nodes, and the caller holds Read allow=true at the Manufacturing prefix — the longest matching prefix for Manufacturing/HRTeam/{user}_Membership — so the node gate passes.

Amendment (2026-07-29): when first filed this was described as "possibly same root". Follow-up investigation showed a superficially similar symptom elsewhere — freshly installed plugin partitions returning only _Policy from structured queries — is not a defect: it is the Store paywall working correctly (root grants Public Viewer, per-child _Access nodes deny it, and _Policy is a public_read type that bypasses the node gate). That investigation instead surfaced a separate fail-open defect on the direct-read path, filed separately. The GroupMembership anomaly above is unrelated to both and remains unexplained.

Environment

Local memex portal (Colima k3s, namespace memex), image memex-portal-ai-local@sha256:801e74d6071e5c09c16b4d480024b6bd28da3cb6a118e300a6055c4e2ca72a8f (locally built from latest, deployed 2026-07-28 ~17:35 CEST); also reproduced twice earlier the same day on the 2026-07-27 image (compiledFrameworkVersion f872ecab…). Postgres backend. Reproduction is deterministic: 3/3, including once on the latest build.