Untyped live-compiled content (PluginContent) render hot-loop leaks memory until the pod is killed #606

Production impact

memex-cloud portal pods balloon ~1GB/min while serving and die (liveness 137 / previously Orleans membership suicide) whenever the untyped-content loop is hot. Log signature, repeating:

MeshNodeStreamCache.GetStream: Content for ClaimsDeepfield stayed an untyped JsonElement after deserialization (TypeRegistry lacks the $type discriminator) — downstream 'Content is X'/'as X' consumers will fail (renders empty, reactive waits time out). Raw: {"$type":"PluginContent", …}

plus PolymorphicTypeInfoResolver / ObjectPolymorphicConverter unregistered-type warnings and Orleans Watchdog starvation.

Mechanism (full trace, file:line on main @ 646b86f)

A recompile of a live NodeType (Store/Plugin) mints a NEW PluginContent CLR type in a new collectible assembly. The persisted node (ClaimsDeepfield etc.) still carries $type:"PluginContent". That type registers ONLY in the freshly built per-node hub's TypeRegistry (compiled HubConfiguration → WithContentType<PluginContent>(), applied in MeshNodeHubFactory.cs:21-41). There is no fan-out into the mesh hub, the deliberately domain-agnostic cache hub (MeshNodeStreamCache.cs:450-458), or circuit hubs — and the collectible-assembly guard (PolymorphicTypeInfoResolver.cs:95-120) intentionally refuses lazy adoption of dynamic types, so those hubs can never resolve the discriminator.

When such a hub deserializes the content (MessageHubGrain.cs:244 uses the mesh hub's options; portal renders use circuit options), ConvertContentJsonElementToTyped (MeshNodeStreamCache.cs:1626-1653) degrades to JsonElement, logs, and returns it as a successful emission. Because it is not an error:

• the storm-breaker/negative cache (GetStreamRaw, 1170-1214) never engages;
• AreaStreamRetry.RetryAreaWithBackoff's bounded retry/give-up never engages (error-only);
• NamedAreaView's give-up handler never engages (untyped content arrives on OnNext, renders empty);
• typed reactive waits (.Where(n => n.Content is T).Take(1).Timeout(…) — the framework's own shape, e.g. NodeTypeCompilationHelpers.cs:173/299/371/…) never see a typed value → time out → the outer render re-opens the wait → each cycle allocates a fresh un-memoized GetStream(path, options).Select subscription (MeshNodeStreamCache.cs:1559-1560 — unlike GetQuery, which memoizes via _optionsWrappedQueries, 1723-1731) plus fresh LayoutAreaHost/enrichment state registered for disposal against a host that never reaches a terminal render;
• the per-node grain amplifies with TryDeactivateOnIdle() churn (MessageHubGrain.cs:301/312) — activate→read→log→deactivate→reactivate per request → Watchdog starvation.

Nothing heals the mismatch; subscriptions + allocations accumulate until the pod dies.

Fix seams (in rising order of invasiveness)

1. (b) Stop the silent success — in ConvertContentJsonElementToTyped/DeserializeContent, when a $type was present but stayed JsonElement, record the path in the existing _negative/backoff cache and surface OnError so the already-built bounded retry + give-up UI engage. Caps the leak; area shows unavailable instead of looping. ⚠️ Regression risk: consumers that today TOLERATE the untyped emission (ContentAs recovery is "the default that cures the untyped-JsonElement read") would start seeing errors — needs a consumer audit before shipping.
2. (a) Read-side resolution fallback — on discriminator miss, resolve the type from the live compilation/assembly store (IAssemblyStore, NodeTypeDefinition.LatestAssemblyPath) as a read-side alias (mirror aliasByName, TypeRegistry.cs:62/86-91), respecting the collectible-assembly pinning concern. Restores typed rendering everywhere; the loop exits because the wait completes.
3. (c) Fan recompiled type registrations into live hubs — broadcast on the compile-completion write (NodeTypeCompilationHelpers.cs:1327-1370) over the existing change-feed (OnMeshChange, MeshNodeStreamCache.cs:487-518). Hardest to get right (new CLR identity per recompile × collectible unload).

Recommended order: (b) with a consumer-tolerance audit, then (a) as the real fix.

Context

Boot-storm half of the crash loop already fixed in #602 (lazy pre-warm/seed, read-first import skip, Orleans membership tolerance); this leak is the remaining crash driver on memex-cloud (2026-07-22, image ci.1301, pod killed 137 after ~26min at ~21GB).

🤖 Generated with Claude Code