Stop stacking a LayoutAreaHost per write — the render leak that killed pods (#606) by rbuergi · Pull Request #916 · Systemorph/MeshWeaver

rbuergi · 2026-08-07T13:50:53Z

Summary

Fixes #606 — the portal leaked until the pod was killed (memex-cloud ballooned ~1 GB/min while serving).

Root cause: every write stacked another live render pipeline on the owner

Workspace.SubscribeToClient (Workspace.cs:661-665) had no registry and no dedup by StreamId. Every arriving SubscribeRequest unconditionally ran ReduceManager.ReduceStream<TReduced>(...), which for a LayoutAreaReference reaches LayoutExtensions.cs:114 → new LayoutAreaHost(...). The previously-created stream for that same StreamId was never disposed — only an UnsubscribeRequest disposes a server-side stream, and a resubscribe never sends one.

Retaining reference: each leaked LayoutAreaHost is held by its own SynchronizationStream<EntityStore> on the owner's sync/{ClientId} hub, and itself holds disposablesByArea (LayoutAreaHost.cs:784) — node-stream + permission subscriptions that register as SharedView subscribers on MeshNodeStreamCache entries, so the idle sweep can never release those upstream sync streams either. It also keeps rendering and keeps pushing frames.

The trigger — why it fires on every write: JsonSynchronizationStream.cs:615 resolves the mirror's staleness Version by name (GetProperty("Version", typeof(long))). EntityStore has no such property, so receivedVersion stays 0 forever while announcedVersion is fabricated as receivedVersion + 1 for every version-less event. The gate receivedVersion < announcedVersion is therefore permanently open for every layout-area stream, so every change-feed event on the owner path calls Resubscribe, re-posting SubscribeRequest with the same StreamId and no unsubscribe.

Net: one write to any node somebody is viewing = one more permanent LayoutAreaHost + full render graph. Untyped PluginContent did not create this; it made the pulses continuous (renders never reach a terminal state), which is why it presented as an untyped-content problem.

Controlled A/B: disarming only the change-feed resubscribe took per-write frames from 1,4,5,6,7,8,9,10 to a flat 1 Full + 4 menu patches on all 8 writes. Instrumentation reverted; no A/B code remains.

Fix

Workspace.cs — instance ConcurrentDictionary<(Subscriber, StreamId), ClientSubscription> with register/lookup. Instance, not static, so lifetime is the owner hub (an owner recycle correctly starts empty; a genuinely orphaned mirror still gets a fresh stream). ClientSubscription is a class, not a record: SynchronizationStream is a record whose generated GetHashCode/Equals recurse into StreamConfiguration and stack-overflow the process — the same trap already documented on _evictedRemoteStreams. Removal is pair-exact by reference identity.
JsonSynchronizationStream.CreateSynchronizationStream — a repeat SubscribeRequest for a (subscriber, StreamId) already served under the same reference now re-asserts the existing stream's state as a Full (delivered by the already-registered outbound forwarder, and the client's ExpectResubscribeFull latch is armed for it) and returns that stream instead of building a second host. It deliberately does not dispose the old stream: SynchronizationStream.Dispose tears down the shared sync/{ClientId} hub and HostedHubsCollection.GetHub has no disposal check, so dispose-then-recreate races a disposing hub.

Three secondary accumulators of the same class — subscriptions created inside a per-emission render body, appended under a synthetic key no area teardown reaps (DisposeExistingAreas/DisposeChildAreas only remove keys that StartsWith(context.Area)) — switched to the framework's existing LayoutAreaHost.ReplaceDisposable, documented for exactly this:

NodeMenuItemsExtensions.cs:101 — the worst: a global predicate renderer (WithRenderer(_ => true, RenderMenus)) appending under the constant key "$Menu" for every context. Now keyed menu-subscription:{area} (deliberately not the bare area name, since LayoutAreaHost.RenderArea:557 registers rendered controls under the area key and replacing that bucket would dispose the live menu control).
OverviewLayoutArea.cs:52 and MeshNodeLayoutAreas.cs:1781 — content projections; the comment at MeshNodeLayoutAreas.cs:1749 already notes the body "re-runs on every render".
GitHubSyncSettingsTab.cs:190 — same shape.

Honest attribution: the host-stacking fix is measured; the four ReplaceDisposable changes are code-reasoned and suite-verified, not independently measured (their accumulated subscriptions are silenced by their own DistinctUntilChanged, so they leak memory without producing countable wire traffic).

Verification

LayoutAreaResubscribeHostAccumulationTest.RepeatedWrites_DoNotStackLayoutAreaHosts_OnTheOwner — managed measurement only, no ClrMD, so it works on macOS where those guards now skip (#674). Every live server-side LayoutAreaHost emits its own Full per node change, so Full frames per write == live render pipelines.

Fails pre-fix: Full frames per write: [1, 4, 5, 6, 7, 8, 9, 10, 11, 12] → "Expected 12 to be less than or equal to 6".
Passes post-fix: [1, 2, 2, 2, 2, 2, 2, 2, 2, 2] (the 2 = 1 render Full + 1 resubscribe re-assert Full).

Suites on the final tree: Hosting.Monolith 478 passed / 0 failed (4 skipped) · Graph 820/820 · Layout 343/343 · Data 303/303. -c Release -warnaserror clean on Data, Graph, GitSync and the test project.

Residual risks

Rebuild-as-recovery is gone. A resubscribe no longer rebuilds the server-side render; if a host is genuinely stuck with Current == null, the guard returns it without asserting anything. The dedicated recovery path (ArmOverlaySelfHeal → DisposeRequest → recycle) still covers this, but it is a real behavioural change worth a reviewer's eye.
The inert staleness gate is untouched (JsonSynchronizationStream.cs:615). Layout-area streams still resubscribe on every owner-path change — now cheap (a Full re-assert) instead of leaking, but still one extra round-trip per write per viewer. Making the gate evidence-based for version-less reductions is a worthwhile follow-up.
Same defect class not swept exhaustively. Four further RegisterForDisposal-inside-a-render sites were found and deliberately not touched: EditLayoutArea.cs:193, DomainCatalogLayoutArea.cs:50 (also mints a fresh Guid data id per render), ThreadLayoutAreas.cs:221 and :285, PartitionSyncAdminLayoutArea.cs:286. Widening into the chat path without measurement was more blast radius than could be verified here.
Untyped-content half not fixed (separate finding): MeshNodeStreamCache.ConvertContentJsonElementToTyped and MeshNodeStreamExtensions.EnsureTypedContent return node with { Content = deserialized } over a freshly cloned JsonElement. JsonElement has no Equals override, so MeshNode record equality is always false for untyped content and every DistinctUntilChanged degrades to a no-op — including the persistence sampler's. The sibling GetQuery seam already does the right thing. Symmetry looks high-value; it needs a NodeType whose content type is registered only on its per-node hub to repro faithfully.

Closes #606

🤖 Generated with Claude Code

…d pods Fixes #606. memex-cloud ballooned ~1 GB/min while serving. Root cause: Workspace.SubscribeToClient had no registry and no dedup by StreamId, so every arriving SubscribeRequest unconditionally ran ReduceManager.ReduceStream, which for a LayoutAreaReference constructs a NEW LayoutAreaHost. The previous stream for that same StreamId was never disposed — only an UnsubscribeRequest disposes a server-side stream, and a resubscribe never sends one. Each leaked host is held by its own SynchronizationStream on the owner's sync/{ClientId} hub and holds disposablesByArea, whose subscriptions register as SharedView subscribers on MeshNodeStreamCache entries, so the idle sweep can never release those upstream streams either — and it keeps rendering. The trigger: JsonSynchronizationStream resolves the staleness Version BY NAME (GetProperty("Version", typeof(long))). EntityStore has no such property, so receivedVersion stays 0 while announcedVersion is fabricated as receivedVersion + 1 for every version-less event. The gate is therefore permanently open for every layout-area stream, so every change-feed event on the owner path resubscribes with the SAME StreamId and no unsubscribe. Net: one write to any node somebody is viewing = one more permanent LayoutAreaHost. Untyped PluginContent did not create this; it made the pulses continuous, which is why it presented as an untyped-content bug. Controlled A/B: disarming only the change-feed resubscribe took per-write frames from 1,4,5,6,7,8,9,10 to a flat 1 Full + 4 menu patches across all 8 writes. Instrumentation reverted; no A/B code remains. Fix: an INSTANCE registry on Workspace keyed (subscriber, StreamId) — instance so its lifetime is the owner hub; ClientSubscription is a class, not a record, because SynchronizationStream is a record whose generated equality recurses into StreamConfiguration and stack-overflows the process (the trap already documented on _evictedRemoteStreams). A repeat SubscribeRequest for a stream already served under the same reference now re-asserts that stream's state as a Full and returns it instead of building a second host. It deliberately does not dispose the old stream: Dispose tears down the shared sync hub and HostedHubsCollection.GetHub has no disposal check, so dispose-then-recreate races a disposing hub. Three secondary accumulators of the same class — subscriptions created in a per-emission render body under a synthetic key no area teardown reaps — switched to the framework's existing LayoutAreaHost.ReplaceDisposable: NodeMenuItemsExtensions (a GLOBAL predicate renderer appending under the constant key "$Menu" for every context), OverviewLayoutArea, MeshNodeLayoutAreas and GitHubSyncSettingsTab. The host-stacking fix is measured; these four are code-reasoned and suite-verified, not independently measured, because their own DistinctUntilChanged silences the duplicates. Test LayoutAreaResubscribeHostAccumulationTest uses a managed measurement only (no ClrMD, which now skips on macOS per #674): every live host emits its own Full per node change, so Full frames per write == live render pipelines. Fails pre-fix with [1,4,5,6,7,8,9,10,11,12] ("Expected 12 to be less than or equal to 6"); passes post-fix with [1,2,2,2,2,2,2,2,2,2]. Suites on the final tree: Monolith 478 passed, Graph 820, Layout 343, Data 303; re-verified after merging main. Release -warnaserror clean. Closes #606 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

This PR addresses #606 by preventing server-side layout-area resubscribe traffic from creating duplicate long-lived render pipelines (e.g., multiple LayoutAreaHost instances) for the same (subscriber, StreamId), which previously led to unbounded memory growth and pod OOMs under normal serving load.

Changes:

Add a per-workspace registry keyed by (subscriber, StreamId) to deduplicate server-side streams and unregister them on disposal.
On owner-side resubscribe, reuse the existing stream and re-assert its current snapshot as a Full instead of creating a second reduced stream/host.
Replace several RegisterForDisposal-inside-render callsites with ReplaceDisposable(...) to prevent stacking subscriptions across re-renders, and add a regression test that counts Full frames per write.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/MeshWeaver.Hosting.Monolith.Test/LayoutAreaResubscribeHostAccumulationTest.cs	New regression test that detects host stacking by counting `Full` frames per write.
src/MeshWeaver.Data/Workspace.cs	Adds an instance `(subscriber, StreamId)` → stream registry and lifecycle-unregistration.
src/MeshWeaver.Data/Serialization/JsonSynchronizationStream.cs	Reuses existing streams on resubscribe and re-asserts current state as a `Full`.
src/MeshWeaver.Graph/OverviewLayoutArea.cs	Switches a render-time subscription registration to `ReplaceDisposable` to avoid accumulation.
src/MeshWeaver.Graph/MeshNodeLayoutAreas.cs	Same `ReplaceDisposable` change for edit-node content projection.
src/MeshWeaver.Graph/Configuration/NodeMenuItemsExtensions.cs	Uses `ReplaceDisposable` with per-area keys to prevent global menu subscription stacking.
src/MeshWeaver.GitSync/GitHubSyncSettingsTab.cs	Uses `ReplaceDisposable` for a render-time “prefill once” subscription to prevent stacking.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+            logger.LogDebug(
+                "Owner {Owner} already serves stream {StreamId} for {Subscriber} — re-asserting the current snapshot instead of creating a second stream",
+                hub.Address, request.StreamId, request.Subscriber);
+            if (alreadyServing.Current is { Value: not null } snapshot)


+using System;
+using System.Collections.Concurrent;
+using System.Linq;


github-actions · 2026-08-07T14:05:49Z