An in-place update no longer costs the router a storage query per message by rbuergi · Pull Request #1179 · Systemorph/MeshWeaver

rbuergi · 2026-08-10T21:28:52Z

Root cause (issue #1172, duplicate #1174)

Every occurrence sits in a boot-window NodeType bake, and the "64 route dispatches in flight … not terminating" criticals and the ~10× bake inflation (per-type TimedOut, pods not-Ready 80 min–2 h 20 min) are one feedback loop:

Each compile writes its activity log line-by-line to {type}/_Activity/compile-{id} — every line a routed PatchDataRequest — plus compile-state mirror writes and heartbeats.
Every write published an Updated change event, and PathResolutionService.OnMeshChange removed the resolution-cache entry for that path. The NEXT routed message to the same activity paid a fresh storage query. Worse, on a hot-written path the cache could never even fill: an Updated landing during a fill dropped the claim, so nothing was ever committed — a query per message, forever.
In RoutingGrain.BuildGrainRoute the in-flight slot is held for the duration of that resolution (the grain hand-off is fire-and-forget), and the queries funnel through the bounded pg-read pool — at bake write rates the window saturates at 64.
The bake's own progress waits (DynamicTypePreWarmer's GetMeshNodeStream(typePath) subscriptions) ride the same router, time out behind the backlog, burn their full per-type budget as TimedOut, and stretch the bake — keeping the storm alive that much longer.

The violated invariant: an in-place Updated cannot change any path's resolution shape — the node existed at that path before the write and still does (moves publish Deleted+Created). Only the cached NODE snapshot goes stale, and only node-reading callers (grain activation, monolith hub creation, navigation) care.

Re the issue's own hypothesis: routes to a pending activation do NOT hold dispatch slots — DeliverToGrainWithRetry is fire-and-forget and pending deliveries park inside the reentrant MessageHubGrain on HubReady. The slot-holder is the per-message resolution, and the churn above is what made it a storage round-trip instead of a dictionary hit.

Fix

PathResolutionService entries become (Resolution, NodeStale). Updated stale-marks matching entries (dropping the cached Node so no superseded content is pinned) and stale-marks in-flight fill claims instead of dropping them, so hot-written paths commit their route shape. Created/Deleted remove entries and drop claims exactly as before.
IPathResolver.ResolveRoute (default: ResolvePath) — the route-shape view that may serve a stale-marked entry because its callers read only Prefix/Remainder. ResolvePath keeps full freshness: a stale-marked entry is a miss whose re-query repairs the entry for everyone.
RoutingGrain.BuildGrainRoute — which never reads Resolution.Node — switches to ResolveRoute.

Routing to a busy node is a dictionary hit again; the pre-warmer's waits complete; the boot window shrinks back to its designed size. This also holds under mid-bake Orleans placement (a joining silo receiving activations immediately): routes to not-yet-baked types no longer consume dispatch slots for a storage query per message.

Verification

Red verified against origin/main (interface default + tests only):

UpdatedNode_KeepsRouteResolutionWarm_WhileNodeReadersRefresh — RED before / green after; also pins that a stale entry carries no Node and that ResolvePath re-queries then repairs.
UpdatedDuringInFlightFill_StillCommitsTheRouteShape — RED before (the dropped claim = nothing ever cached for a continuously-written path) / green after.
CreatedAndDeleted_StillInvalidateRouteResolution — scope guard.

Suites: Hosting.Test 143/143 · PathResolution.Test 123/123 · Hosting.Orleans.Test 155/155 (incl. RoutingGrainTurnIsolationTest through the new ResolveRoute path) · dotnet build -c Release -warnaserror clean for the touched chain.

Fixes #1172
Fixes #1174

🤖 Generated with Claude Code

https://claude.ai/code/session_01UYLgoPK1qLtyxZ2ixdFtj5

…sage Prod (memex-cloud, 2026-08-10, issues #1172/#1174): during every boot NodeType bake the silo's RoutingGrain reported "64 route dispatches are in flight ... not terminating", pods sat Running/not-Ready for 80+ minutes (one 2 h 20 min), and the ~10-minute bake inflated ~10x with per-type "TimedOut" entries. The two symptoms were one feedback loop: 1. Each compile writes its activity log line-by-line to {type}/_Activity/compile-{id} - every line a routed PatchDataRequest, plus compile-state mirror writes and sync-stream heartbeats. 2. Every such write published an Updated change event, and PathResolutionService.OnMeshChange REMOVED the resolution-cache entry for that exact path - so the NEXT routed message to the same activity paid a fresh storage query. On a hot-written path the cache could never even fill: an Updated landing during the fill dropped the claim, so nothing was ever committed. 3. In BuildGrainRoute, a route's in-flight slot is held for the duration of that resolution (the grain hand-off is fire-and-forget), and the queries funnel through the bounded pg-read pool. At bake write rates the router's in-flight window saturated at 64. 4. The bake's own progress waits (DynamicTypePreWarmer's GetMeshNodeStream(typePath) subscriptions) ride the same router, timed out behind the backlog, burned their full per-type budget as TimedOut, and stretched the bake - which kept the storm alive that much longer. The invariant the sweep violated: an in-place Updated CANNOT change any path's resolution SHAPE. The node existed at that path before the write and still does - moves publish Deleted(old)+Created(new), deletes publish Deleted. Only the cached NODE snapshot (which grain activation reads) goes stale. Fix - split the cache's two freshness contracts instead of conflating them: - PathResolutionService entries are now (Resolution, NodeStale). Updated stale-MARKS matching entries (dropping the cached Node so no superseded content is pinned) and stale-marks in-flight fill claims instead of dropping them, so hot-written paths commit their route shape. Created/Deleted remove entries and drop claims exactly as before. - IPathResolver gains ResolveRoute (default: ResolvePath) - the route-shape view that may serve a stale-marked entry because its callers read only Prefix/Remainder. ResolvePath keeps full freshness: a stale-marked entry is a miss, and its re-query repairs the entry for everyone. - RoutingGrain.BuildGrainRoute - which never reads Resolution.Node - switches to ResolveRoute. Grain activation, monolith hub creation and Blazor navigation stay on the node-fresh paths. Routing to a busy node is a dictionary hit again; the router stops queuing on pg-read during the bake, the pre-warmer's waits complete, and the boot window shrinks back to its designed size. Tests (test/MeshWeaver.Hosting.Test/PathResolutionCachePoisonTest.cs), red verified against origin/main with only the interface default + tests applied: - UpdatedNode_KeepsRouteResolutionWarm_WhileNodeReadersRefresh - RED before (Updated swept the entry; ResolveRoute re-queried), green after; also pins that the stale entry carries no Node and that ResolvePath re-queries then repairs the entry. - UpdatedDuringInFlightFill_StillCommitsTheRouteShape - RED before (the dropped claim meant nothing was ever cached for a continuously-written path), green after. - CreatedAndDeleted_StillInvalidateRouteResolution - guards the fix's scope. Suites: Hosting.Test 143/143, PathResolution.Test 123/123, Hosting.Orleans.Test 155/155 (incl. RoutingGrainTurnIsolationTest through the new ResolveRoute path). dotnet build -c Release -warnaserror clean for the touched chain. Fixes #1172 Fixes #1174 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UYLgoPK1qLtyxZ2ixdFtj5

…tion-updated-churn

Copilot

Pull request overview

This PR addresses a router saturation feedback loop during the dynamic NodeType bake by preventing in-place Updated writes from evicting the route-shape cache used by Orleans routing, while still forcing node-reading callers to refresh stale node snapshots.

Changes:

Introduces IPathResolver.ResolveRoute as a route-shape-only resolution contract (Prefix/Remainder) that may serve stale-marked cache entries.
Updates PathResolutionService to stale-mark (not remove) cache entries and in-flight fills on Updated, dropping cached Node snapshots while preserving route shape.
Switches RoutingGrain.BuildGrainRoute to use ResolveRoute, adds targeted regression tests, and documents the user-visible improvement in What’s New.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
test/MeshWeaver.Hosting.Test/PathResolutionCachePoisonTest.cs	Adds regression tests covering Updated stale-marking behavior and Created/Deleted invalidation.
src/MeshWeaver.Mesh.Contract/Services/IPathResolver.cs	Adds `ResolveRoute` with default implementation and documents the route-shape freshness contract.
src/MeshWeaver.Hosting/PathResolutionService.cs	Implements stale-marking cache entries/claims on `Updated` while preserving remove-on `Created`/`Deleted`.
src/MeshWeaver.Hosting.Orleans/RoutingGrain.cs	Routes via `ResolveRoute` to avoid storage queries on hot-written paths.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-10-the-portal-comes-back-faster-after-its-daily-update.md	Adds a user-facing “Fix” entry describing the improved startup/rebuild behavior.

Suppressed comments (2)

src/MeshWeaver.Hosting/PathResolutionService.cs:378

The OnMeshChange XML doc still describes Updated as removing cached entries / staleness in a way that no longer matches the new behavior (Updated now stale-marks and drops the cached node snapshot, while Created/Deleted remove entries). Keeping the comment accurate matters because it documents the two freshness contracts that this service now implements.

This issue also appears on line 395 of the same file.

    /// Change-feed invalidation: any Created/Updated/Deleted event with path
    /// <c>P</c> removes every cached entry resolving to or under <c>P</c>
    /// (<c>key == P</c> or <c>key.StartsWith(P + "/")</c>). Created(P) can DEEPEN
    /// the resolution of P and of every descendant path (they previously resolved
    /// to a shallower ancestor); Deleted/Updated(P) staleness anything that
    /// resolved to or through P. Runs synchronously on the publisher's thread —

src/MeshWeaver.Hosting/PathResolutionService.cs:401

The XML doc for OnMeshChange still says the same sweep is applied to in-flight fill claims by “dropping its claim”, but Updated events no longer drop claims (they stale-mark them so the fill can still commit the route shape). Updating that doc will keep the explanation consistent with the new Updated behavior.

        if (change.Kind == MeshChangeKind.Updated)
        {
            // 🚨 An in-place update cannot change ANY path's resolution SHAPE — the node
            // existed at that path before the write and still does (moves publish
            // Deleted(old)+Created(new), deletes publish Deleted). Only the cached NODE
            // snapshot goes stale. Stale-MARK instead of remove, claims first so a fill
            // committing between the two sweeps cannot land a fresh entry the entry-sweep

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-08-10T21:42:46Z