Skip to content

AGENTS.md + docs: the grain serves old state until a DisposeRequest — step 1 of 3 - #4641

Merged
meshweaver-cloud[bot] merged 2 commits into
mainfrom
docs/dispose-request-stale-grain-state
Sep 17, 2026
Merged

meshweaver-cloud[bot] merged 2 commits into
mainfrom
docs/dispose-request-stale-grain-state

Conversation

@rbuergi

@rbuergi rbuergi commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Maintainer directive, 2026-09-17, verbatim across three messages:

"the grain keeps serving old state until we send dispose request" / "write this prominently in all repos" / "in pull request skill etc"

The mechanism, established from the code before a word was written

A per-node hub resolves its HubConfiguration once, while it is activating, and is then pinned by address. src/MeshWeaver.Graph/Configuration/MeshNodeHubFactory.cs:

"The binding above is made ONCE and then PINNED by address: routing short-circuits on GetHostedHub for an already-hosted address and never resolves the path again, so nothing re-reads the NodeType for the hub's whole lifetime."

So a merge, a seal, an image, a roll and even a pod restart change what the next activation would load, while an address already up keeps answering from what it bound — with no error, no log line and nothing to grep. That is the missing half of "merged and rolled is not loaded", of a NodeType recompiled while the hub still serves the previous assembly, and of a fix that is live in the image and invisible in the running mesh.

Two halves of the rule are load-bearing and both are stated:

  • Content is live. stream.Update and node CRUD route to the owning activation and MeshDataSource holds a long-standing own-node subscription — activation only seeds the workspace. Nobody should ever recycle to make a write visible; if they do, the write is the bug.
  • A dispose makes the activation RE-READ; it does not change what the re-read FINDS. The assembly store is keyed (nodeTypePath, LastCompiledVersion), a recompile of an already-Ok type does not rewrite its node, and the release request the recycle verb stamps adopts before it compiles (Modules:VersionStrictness defaults to Family) — so the same local copy, or the same prebuilt bundle sealed against another platform surface, lands the identical bytes again. Measured on memex over 30+ minutes and six recycles (A portal can serve STALE compiled code while reporting Ok/compiled — recycles, forced compile and the stale-build banner are all inert #2471), which is exactly why a same-path build mismatch is reported and deliberately NOT offered as a recycle. The remedy is a rebake and republish for this framework identity, or a forced recompile of the live source.

What lands here

  • AGENTS.md — a new ABSOLUTE section, wrapped as the shared-rule block dispose-to-serve-new-state and registered in .github/shared-rules.json with required-in naming only the hub. That is step 1 of 3 per SharedRuleBlocks.md: the id becomes known and the gate is green on its own, so no spoke marker is ever undeclared. Step 2 is the six spoke PRs; step 3 widens required-in to all seven.
  • Doc/Architecture/StaleStateUntilRecycle — the mechanism in one citable place, which it did not have: it was scattered over a comment in MeshNodeHubFactory, a class doc in NodeTypeRebindWatcher, a type doc in ServedBuildIdentity, an enum doc in BuildProvenance and four sections of a 2,660-line compilation page. It covers what an activation pins vs what stays live; the one framework surface (hub.RecycleNode — cold, the wait is a READ not a poll, install-lease deferral, the caller must outlive the target); the operator verb and the fact that the Compile button posts no DisposeRequest; the three posting rules (off-router per RecycleCore posts DisposeRequest from the root mesh hub instead of the NodeOperationIssuingHub seam #4463, the root mesh hub's outright refusal, carry a Reason per A package root is disposed while its own install is in flight (CD 7950): the creates' write-through streams lose their owner, no reply is ever sent, and the install runs out its bound — neither #3501 nor #3505 covers it #3510); the four automatic recyclers and the one config key (Modules:AutoRecycleOnStaleBuild, default OFF, so a portal is a mixture of old and new assemblies for as long as viewers do not click); and the four things a recycle does not do.
  • The skills — pullrequest first, as the directive named it ("…and MERGED is not LIVE", with the four-step verification and the PR-body obligation to name addresses a deploy must recycle), plus deployment, release, mesh-data and debug.
  • Cross-links from HubDisposalModel and NodeTypeCompilation, a topic-map line, and a What's New entry for the new page.

Not in this change, deliberately

Systemorph/MeshWeaver.Crm is not added to the register: it is absent from repos, so the gate never reads it, and registering it needs the read-only fleet-reader App installed there and Crm carrying every block already registered — an unreadable repo is RED here by design. Crm will carry this block's text anyway, so a later registration is a one-line change rather than a rollout. The register entry says so.

#4464 is not in this repo (git log --all --grep=4464 finds nothing about it); the off-router change is #4463, merged as #4477.

Verification

  • check-shared-rules.py --self-test — 18 cases, all correct.
  • check-shared-rules.py --local across all seven repos with the new block registered — ✅ All 5 shared rule block(s) are identical everywhere they are carried, across 7 repos.
  • dotnet build test/MeshWeaver.Documentation.Test -c Release -warnaserror — 0 Error(s), then the doc guards on that build: 12 tests, 12 passed, 0 failed (DocumentationLinkIntegrityTest, ArchitectureTopicMapTest, WhatsNewEntryIntegrityTest, NoConflictMarkersGuard, MigrationWorkloadModelGuard).

Pairs-with: none — documentation and agent instructions only; no src/ surface changes.

🤖 Generated with Claude Code

…age, and step 1 of the fleet block

Maintainer directive, 2026-09-17, verbatim across three messages: "the grain
keeps serving old state until we send dispose request" / "write this prominently
in all repos" / "in pull request skill etc".

A per-node hub resolves its HubConfiguration ONCE while activating and is then
pinned by address — routing short-circuits on an already-hosted address and never
resolves the path again (MeshNodeHubFactory). So a merge, a seal, an image, a roll
and even a pod restart change what the NEXT activation would load, while an address
already up keeps answering from what it bound, with no error, no log line and
nothing to grep. That is the missing half of "merged and rolled is not loaded", of
a NodeType recompiled while the hub still serves the previous assembly, and of a
fix that is live in the image and invisible in the running mesh.

What lands here:

* AGENTS.md — a new ABSOLUTE section, wrapped as the shared-rule block
  `dispose-to-serve-new-state` and registered in .github/shared-rules.json with
  `required-in` naming only the hub. That is step 1 of the three-step rollout
  SharedRuleBlocks.md prescribes: the id becomes known and the gate is green on
  its own, so no spoke marker is ever undeclared. Step 2 is the six spoke PRs;
  step 3 widens `required-in` to all seven.

* Doc/Architecture/StaleStateUntilRecycle — the mechanism in one citable place,
  which it did not have: it was scattered over a comment in MeshNodeHubFactory, a
  class doc in NodeTypeRebindWatcher, a type doc in ServedBuildIdentity, an enum
  doc in BuildProvenance and four sections of a 2,660-line compilation page. States
  what an activation pins and what stays live (content does — stream.Update reaches
  the owner and MeshDataSource holds a long-standing own-node subscription), the one
  framework surface (hub.RecycleNode: cold, the wait is a READ, install-lease
  deferral, the caller must outlive the target), the three posting rules
  (off-router per #4463, the root mesh hub's refusal, carry a Reason per #3510), the
  four automatic recyclers and the one config key
  (Modules:AutoRecycleOnStaleBuild, default OFF) — and, load-bearing, what a dispose
  does NOT change: the store is keyed (nodeTypePath, version) and the recompile
  adopts before it compiles, so the same local copy or the same prebuilt bundle
  lands the identical bytes again (#2471, measured over six recycles).

* The skills the directive named and the ones the mechanism reaches: pullrequest
  (MERGED is not LIVE — the four-step verification and the PR-body obligation),
  deployment (the right image is still not the right answer), release, mesh-data
  (content is live, the binding is not), debug (rule this out before tracing).

* Cross-links from HubDisposalModel and NodeTypeCompilation, a topic-map line, and
  a What's New entry for the new page.

Verified: check-shared-rules.py --self-test 18/18, and --local across all seven
repos green with the new block registered; Documentation.Test built Release
-warnaserror (0 Error(s)) and the doc guards run — 12 tests, 12 passed
(DocumentationLinkIntegrity, ArchitectureTopicMap, WhatsNewEntryIntegrity,
NoConflictMarkers, MigrationWorkloadModel).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 17, 2026 18:24
@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

429 tests  ±0   429 ✅ ±0   1m 2s ⏱️ +3s
  2 suites ±0     0 💤 ±0 
  2 files   ±0     0 ❌ ±0 

Results for commit 176734a. ± Comparison against base commit cc59e37.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

264 tests  ±0   264 ✅ ±0   2m 42s ⏱️ +12s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 176734a. ± Comparison against base commit cc59e37.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

533 tests  ±0   533 ✅ ±0   1m 0s ⏱️ -2s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 176734a. ± Comparison against base commit cc59e37.

♻️ This comment has been updated with latest results.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Multiple unresolved accuracy issues remain in the operational and architecture guidance.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR documents stale per-node activation state, recycle behavior, and deployment verification guidance.

Changes:

  • Adds the canonical architecture page and What's New entry.
  • Updates agent, release, deployment, debugging, pull-request, and mesh-data guidance.
  • Registers the shared rule for staged rollout.
File summaries
File Change Final review findings
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-17-why-a-page-can-keep-showing-the-old-version-after-an-update.md Adds user-facing stale-state guidance. Nit (1 vote): Qualify Recycle as a remedy because same-path build mismatches may persist.
src/MeshWeaver.Documentation/Data/Architecture/StaleStateUntilRecycle.md Adds the canonical activation and recycle model. Nit (2 votes): Qualify pod-restart behavior.
Nit (2 votes): Qualify RecycleNode read guarantees for root-hub callers.
Nit (1 vote): Limit release stamping and account for prebuilt adoption.
Nit (1 vote): Correct the initial-load echo explanation.
Nit (2 votes): Correct SettleRetypedRoot routing behavior.
src/MeshWeaver.Documentation/Data/Architecture/NodeTypeCompilation.md Adds a cross-link to stale-state guidance. —
src/MeshWeaver.Documentation/Data/Architecture/HubDisposalModel.md Adds lifecycle cross-links. —
src/MeshWeaver.Documentation/Data/Architecture.md Adds the architecture topic-map entry. —
AGENTS.md Adds the shared stale-state rule. Nit (2 votes): Qualify pod-restart behavior.
Nit (1 vote): Distinguish sender/origin from root-target refusal.
Nit (2 votes): Correct forced release compilation behavior.
Nit (3 votes): Limit operator recycling scope and account for prebuilt adoption.
.github/shared-rules.json Registers the shared rule for staged rollout. —
.claude/skills/release/SKILL.md Adds release verification guidance. Nit (3 votes): Account for the effective AKS auto-recycle setting.
Nit (1 vote): Distinguish operator recycle from dispose-only recycling.
.claude/skills/pullrequest/SKILL.md Adds post-merge live verification steps. Nit (2 votes): Qualify pod-restart behavior.
Nit (2 votes): Correct operator recycle compilation behavior.
.claude/skills/mesh-data/SKILL.md Clarifies content and binding freshness. Nit (3 votes): Limit release stamping to NodeType targets and account for prebuilt adoption.
.claude/skills/deployment/SKILL.md Adds deployment verification guidance. Nit (3 votes): Account for the effective AKS auto-recycle setting.
Nit (2 votes): Distinguish operator recycle from dispose-only recycling.
.claude/skills/debug/SKILL.md Adds stale-activation debugging guidance. Nit (2 votes): Qualify pod-restart behavior.
Nit (2 votes): Correct operator recycle compilation behavior.
Nit (2 votes): Qualify the isolated-test exception.
Review details

Suppressed comments (7)

.claude/skills/release/SKILL.md:297

  • This says any recycle leaves the bytes unchanged, but the operator recycle verb stamps a forced NodeType release and compiles the live source; that behavior is different from the dispose-only hub.RecycleNode extension. Clarify which surface the release procedure is prescribing.
name them. A recycle makes the activation re-read, it does not change what the re-read finds — so a
second one that answers the same is a delivery question, not an activation one. Full reference:

AGENTS.md:70

  • HandleDispose refuses only when the target hub has Address.Type == "mesh"; it does not refuse a DisposeRequest posted by the root mesh hub to a node. #4463 was exactly that accepted delivery, which is why the off-router hop is required. Reword this to separate sender/origin from root-target refusal.
**Tearing the activation down is the ONLY thing that makes it re-read**, and the next access re-creates the hub from its node — a fresh NodeType resolution, a fresh compile-or-adopt, fresh data loads. In code that is `hub.RecycleNode(path, reason: "…")`, the ONE framework surface: cold (it posts on Subscribe, so an unsubscribed chain tears nothing down), the wait is the framework's own READ rather than a poll or a sleep, and it defers while a package install holds that root. As an operator or agent it is the `recycle` verb — the MCP tool, a node's **Recycle** menu entry, `mw recycle <path>` — which also checks `Update` permission and stamps a release request BEFORE the dispose so the fresh activation recompiles instead of re-binding. (The **Compile** button writes that same release-request stamp and posts NO `DisposeRequest`: it rebuilds the type, it does not re-bind a live instance.) 🚨 The caller's hub must OUTLIVE the target: a dying hub cannot deliver its own last frame, so never recycle from the hub being recycled. 🚨 Issue it off the router (`hub.NodeOperationIssuingHub()` — #4463), never off the DI-injected root mesh hub, which `HandleDispose` refuses outright because disposing it over the bus wedged every node operation until the process restarted. 🚨 ALWAYS carry a `Reason` — an omitted one prints as *"reason not stated by the caller"*, and that blank cost #3510 six occurrences and four bake seals.

src/MeshWeaver.Documentation/Data/Architecture/StaleStateUntilRecycle.md:152

  • RequestedReleaseForce is the opposite of what this says. MeshOperations.RecycleCore writes it, and the compile watcher then skips on-demand prebuilt adoption and dispatches a live-source compile (NodeTypeCompilationHelpers.cs:382-402). The documented same-bundle outcome is therefore not the behavior of the operator recycle; it applies only to a dispose-only recycle.
**And the recompile can re-choose the same bundle.** The release request the `recycle` verb stamps
runs *adopt-before-compile*: the deployment's prebuilt bundle sources get one bounded chance to
supply the assembly first, and an adopted type then satisfies its release request without Roslyn
ever running. `Modules:VersionStrictness` decides what counts as a candidate, and its default is
**`Family`** — a bundle sealed for another identity of the same major line is adopted when its

src/MeshWeaver.Documentation/Data/Architecture/StaleStateUntilRecycle.md:92

  • RecycleLayoutArea explicitly drives the node menu through the circuit's hub.RecycleNode(path) flow, and that extension only posts DisposeRequest/waits; it does not stamp a release. MeshOperations.RecycleCore stamps only when IsNodeTypeNode(curr) is true, so an instance/menu recycle does not recompile its NodeType. This paragraph conflates the surfaces and promises a compile that does not happen.
The **`recycle` verb** (`MeshOperations.Recycle`) backs the `recycle` MCP tool, a node's **Recycle**
menu entry and `mw recycle <path>`. It is more than a dispose: it checks `Update` permission on the
target (a refusal is an *answer* rendered in the operation's envelope, not a fault), waits on the
same install lease, **stamps a release request on the NodeType before the dispose** — so the fresh
activation recompiles rather than re-binding — and broadcasts a cache invalidation on the change

src/MeshWeaver.Documentation/Data/Architecture/StaleStateUntilRecycle.md:95

  • MeshOperations.Recycle only stamps a release request when the target node is itself a NodeType definition (IsNodeTypeNode); recycling an ordinary instance/page does not stamp its NodeType. Also, the release path can adopt a compatible prebuilt bundle without running Roslyn. This description overstates both the scope and effect of the operation.
same install lease, **stamps a release request on the NodeType before the dispose** — so the fresh
activation recompiles rather than re-binding — and broadcasts a cache invalidation on the change
feed. The stamp is *sequenced* before the dispose, never merely issued alongside it: when the dispose
won that race, the reactivated hub re-ran its source query against a half-invalidated state, matched
zero `Code` nodes and recompiled the **pre-fix** source.

src/MeshWeaver.Documentation/Data/Architecture/StaleStateUntilRecycle.md:176

  • MeshDataSource deliberately suppresses the initial-load echo: a node just loaded from persistence is already durable and must not be written back. Thus the reason a recycle clears no data is a fresh read/no delete, not that the own-node source re-saves the loaded node; this sentence currently describes the opposite behavior.
- **It clears no data.** Reactivation re-loads from the store and the own-node source re-saves what
  it loaded, so a recycle is not a delete: a node removed underneath the mesh comes back on the next
  activation, and clearing in-memory-only state takes a process restart.

src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-17-why-a-page-can-keep-showing-the-old-version-after-an-update.md:21

  • This presents Recycle as a universal remedy, but the page itself documents same-path build mismatches where a fresh activation finds the same bytes. Qualify the user-facing guidance so readers do not repeatedly recycle a node when the delivery/build artifact—not the activation—is stale.
answering exactly as it did before, with nothing anywhere reporting a problem. The remedy is the
**Recycle** action on that node: it stops the worker, and the next visit starts a fresh one that
reads everything again.
  • Files reviewed: 12/12 changed files
  • Comments generated: 15
  • Review effort level: Lite

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .claude/skills/debug/SKILL.md Outdated
Comment on lines +94 to +97
binds its configuration ONCE while activating and is then pinned by address — routing never resolves
that path again. Merging, sealing, rolling the image and restarting the pod all change what the NEXT
activation would load; an address that is already up answers from what it bound, promptly and
wrongly, with **no error, no log line and nothing to grep**.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, and it was the worst error in the change — thanks. Confirmed: a restart tears down every activation the pod hosts, so those addresses re-activate and re-read on next access, and a roll does the same for the pods it replaces. The claim now reads "every change a portal absorbs WHILE IT KEEPS RUNNING", with an explicit note that a restart or a roll is NOT the gap — what a roll leaves open is only what the FRESH activation then chooses, plus what is pinned at process start (the store modules). Fixed in AGENTS.md, StaleStateUntilRecycle.md (which now carries the restart case as its own 🚨 paragraph), and the pullrequest, deployment, release and debug skills.

Comment thread .claude/skills/debug/SKILL.md Outdated
Comment on lines +104 to +111
🚨 **But a recycle makes the activation RE-READ; it does not change what the re-read FINDS.** If the
second attempt answers the same way, the activation was never the problem and a third recycle will
find nothing: the assembly store is keyed `(nodeTypePath, version)` and the recompile adopts before
it compiles, so the same local copy or the same prebuilt bundle lands the identical bytes again
(#2471 — measured over 30+ minutes and six recycles, every surface reporting success). At that point
go and read what the pod actually loaded (`[ModuleLoad] … (written=…)`), not the message flow. Full
reference:
[StaleStateUntilRecycle.md](../../../src/MeshWeaver.Documentation/Data/Architecture/StaleStateUntilRecycle.md).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, and I had it backwards. Verified in code: MeshOperations.WithReleaseRequest stamps RequestedReleaseAt AND RequestedReleaseForce = true, and NodeTypeCompilationHelpers short-circuits on exactly that flag — "A FORCE MEANS BUILD THE LIVE SOURCE, NOT SERVE ME WHATEVER A BUNDLE STILL RESOLVES" (#2818, which exists because a force used to re-adopt the very bytes the operator was replacing, #2813). So the operator recycle of a NodeType is the rebuild-from-source remedy, not a re-adoption. The same-bytes text is now scoped to (a) a dispose-only recycle, which compiles nothing and re-binds the same store key, and (b) an UNFORCED trigger or a Modules:RequirePrebuilt mesh. Fixed everywhere it was copied.

Comment thread .claude/skills/debug/SKILL.md Outdated
Comment on lines +113 to +114
**None of this applies to a TEST**, where the mesh is built fresh per fixture — there, go straight to
the procedure above.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid — the blanket exception was wrong and TestStateIsolation.md says so directly: with a shared TestCluster per xUnit collection "the grain caches its config, Test B activates a different node at the same path, reads stale state from Test A". The debug skill now says a test is NOT automatically exempt — a fixture that builds and tears down its own mesh is, a shared one is not — and links TestStateIsolation.md. The Architecture page carries the same qualification in its pinning section.

Comment thread .claude/skills/deployment/SKILL.md Outdated
Comment on lines +161 to +165
- **`Modules:AutoRecycleOnStaleBuild` is OFF by default.** The stale-build watcher publishes a
*banner offering* a recycle instead of converging, so a portal is a mixture of old and new
assemblies **for as long as viewers do not click**. That is a deliberate default (the
unconditional self-recycle it replaced made publication frequency equal restart frequency), and it
is the operative fact when you are asked "is the fix live?". The 2026-08-25 Store outage is the

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid — I quoted the code default as if it were the fleet default. Verified: PrebuiltAdoptionPolicy-side the runtime default is off, and deploy/aks/values.aks.yaml:342 sets Modules__AutoRecycleOnStaleBuild: "true" fleet-wide (deploy/helm .../deployment.yaml also calls convergence "on fleet-wide"). The text now states BOTH and tells the reader to read the EFFECTIVE value, keeping the off-case consequence (banner only, viewers must click) separate from the configured-true case, and notes that neither setting fires for a same-path byte change.

Comment thread .claude/skills/deployment/SKILL.md Outdated
Comment on lines +171 to +177
- 🚨 **A recycle makes the activation RE-READ; it does not change what the re-read FINDS.** A third
recycle finds nothing a second did not: the assembly store is keyed `(nodeTypePath, version)` and
the recompile adopts-before-compiling, so the same local copy / the same prebuilt bundle lands the
identical bytes again (#2471 — measured over 30+ minutes and six recycles). The remedy there is a
rebake and republish for THIS framework identity, or a forced recompile of the live source. And a
recycle cannot fix a package that never reached the pod — read the
`[ModuleLoad] <Assembly> ← … (written=…)` line before recycling anything.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, and I had it backwards. Verified in code: MeshOperations.WithReleaseRequest stamps RequestedReleaseAt AND RequestedReleaseForce = true, and NodeTypeCompilationHelpers short-circuits on exactly that flag — "A FORCE MEANS BUILD THE LIVE SOURCE, NOT SERVE ME WHATEVER A BUNDLE STILL RESOLVES" (#2818, which exists because a force used to re-adopt the very bytes the operator was replacing, #2813). So the operator recycle of a NodeType is the rebuild-from-source remedy, not a re-adoption. The same-bytes text is now scoped to (a) a dispose-only recycle, which compiles nothing and re-binds the same store key, and (b) an UNFORCED trigger or a Modules:RequirePrebuilt mesh. Fixed everywhere it was copied.

Comment thread AGENTS.md Outdated

**What stays live:** every write that goes THROUGH the mesh — `GetMeshNodeStream(path).Update(...)` and node CRUD route to the owning activation, whose own-node subscription picks them up, which is why editing and every data-bound view need no recycle. **What is bound ONCE and never re-read:** the NodeType binding the hub was born with — and with it the compiled assembly, its collectible load context and everything that configuration's closure captured — plus anything read from outside the mesh (the image's files, the pinned store modules). **"Published" therefore does not imply "every instance is running it"**: the stale-build watcher OFFERS a recycle by default (convergence is the opt-in `Modules:AutoRecycleOnStaleBuild`), so after a package publishes a new build a portal is a mixture of old and new assemblies for as long as viewers do not click.

**Tearing the activation down is the ONLY thing that makes it re-read**, and the next access re-creates the hub from its node — a fresh NodeType resolution, a fresh compile-or-adopt, fresh data loads. In code that is `hub.RecycleNode(path, reason: "…")`, the ONE framework surface: cold (it posts on Subscribe, so an unsubscribed chain tears nothing down), the wait is the framework's own READ rather than a poll or a sleep, and it defers while a package install holds that root. As an operator or agent it is the `recycle` verb — the MCP tool, a node's **Recycle** menu entry, `mw recycle <path>` — which also checks `Update` permission and stamps a release request BEFORE the dispose so the fresh activation recompiles instead of re-binding. (The **Compile** button writes that same release-request stamp and posts NO `DisposeRequest`: it rebuilds the type, it does not re-bind a live instance.) 🚨 The caller's hub must OUTLIVE the target: a dying hub cannot deliver its own last frame, so never recycle from the hub being recycled. 🚨 Issue it off the router (`hub.NodeOperationIssuingHub()` — #4463), never off the DI-injected root mesh hub, which `HandleDispose` refuses outright because disposing it over the bus wedged every node operation until the process restarted. 🚨 ALWAYS carry a `Reason` — an omitted one prints as *"reason not stated by the caller"*, and that blank cost #3510 six occurrences and four bake seals.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid. MeshOperations.RecycleCore stamps through Update(curr => IsNodeTypeNode(curr) ? WithReleaseRequest(curr, ...) : curr), so an ordinary page or instance is a plain teardown that recompiles nothing. The text now says "only when the target is a NodeType node" and states that recycling an ordinary node stamps nothing. Combined with the RequestedReleaseForce correction above: on a NodeType the verb forces a live-source compile; on anything else it is dispose-only.

Comment thread AGENTS.md Outdated

**Tearing the activation down is the ONLY thing that makes it re-read**, and the next access re-creates the hub from its node — a fresh NodeType resolution, a fresh compile-or-adopt, fresh data loads. In code that is `hub.RecycleNode(path, reason: "…")`, the ONE framework surface: cold (it posts on Subscribe, so an unsubscribed chain tears nothing down), the wait is the framework's own READ rather than a poll or a sleep, and it defers while a package install holds that root. As an operator or agent it is the `recycle` verb — the MCP tool, a node's **Recycle** menu entry, `mw recycle <path>` — which also checks `Update` permission and stamps a release request BEFORE the dispose so the fresh activation recompiles instead of re-binding. (The **Compile** button writes that same release-request stamp and posts NO `DisposeRequest`: it rebuilds the type, it does not re-bind a live instance.) 🚨 The caller's hub must OUTLIVE the target: a dying hub cannot deliver its own last frame, so never recycle from the hub being recycled. 🚨 Issue it off the router (`hub.NodeOperationIssuingHub()` — #4463), never off the DI-injected root mesh hub, which `HandleDispose` refuses outright because disposing it over the bus wedged every node operation until the process restarted. 🚨 ALWAYS carry a `Reason` — an omitted one prints as *"reason not stated by the caller"*, and that blank cost #3510 six occurrences and four bake seals.

🚨 **A dispose makes the activation RE-READ; it does NOT change what the re-read FINDS — so it is never itself a fix, and a second recycle proves nothing the first did not.** The assembly store is keyed `(nodeTypePath, version)` and a recompile of an already-`Ok` type does not rewrite its node, so a recycle **re-resolves the same key from the same local copy**; the release request it stamps adopts-before-compiling, and where the only candidate is a prebuilt bundle sealed against another platform surface (`Modules:VersionStrictness` defaults to `Family` — same major line, floor satisfied, links resolve) the identical bytes land again. That was measured on memex over 30+ minutes and six recycles (#2471), which is why a same-path build mismatch is reported and deliberately NOT offered as a recycle. The remedies are a rebake and republish for THIS framework identity, or a forced recompile of the live source — never more recycles. A recycle also clears no data (reactivation re-loads and re-saves what the store holds, so it is not a delete), touches ONE address and not the process, the other replicas or any process-wide cache, and cannot fix a package that never reached the pod — read the `[ModuleLoad] … (written=…)` line before recycling anything.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, and I had it backwards. Verified in code: MeshOperations.WithReleaseRequest stamps RequestedReleaseAt AND RequestedReleaseForce = true, and NodeTypeCompilationHelpers short-circuits on exactly that flag — "A FORCE MEANS BUILD THE LIVE SOURCE, NOT SERVE ME WHATEVER A BUNDLE STILL RESOLVES" (#2818, which exists because a force used to re-adopt the very bytes the operator was replacing, #2813). So the operator recycle of a NodeType is the rebuild-from-source remedy, not a re-adoption. The same-bytes text is now scoped to (a) a dispose-only recycle, which compiles nothing and re-binds the same store key, and (b) an UNFORCED trigger or a Modules:RequirePrebuilt mesh. Fixed everywhere it was copied.

Comment on lines +20 to +25
A merged pull request, a published bake, a new image, a rolled deployment and even a pod restart all
change what the **next** activation would load. An address that is already up is untouched by every
one of them, and the failure has no signal of its own: no error, no warning, no log line, nothing to
grep. The address answers promptly and answers the old thing. That is the missing half of three
recurring stories — *"merged and rolled is not loaded"*, *"the NodeType recompiled but the hub still
serves the previous assembly"*, and *"the fix is in the image and invisible in the running mesh"*.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, and it was the worst error in the change — thanks. Confirmed: a restart tears down every activation the pod hosts, so those addresses re-activate and re-read on next access, and a roll does the same for the pods it replaces. The claim now reads "every change a portal absorbs WHILE IT KEEPS RUNNING", with an explicit note that a restart or a roll is NOT the gap — what a roll leaves open is only what the FRESH activation then chooses, plus what is pinned at process start (the store modules). Fixed in AGENTS.md, StaleStateUntilRecycle.md (which now carries the restart case as its own 🚨 paragraph), and the pullrequest, deployment, release and debug skills.

Comment on lines +70 to +73
- **The wait is a read, not a poll.** `GetMeshNode` already treats a `ShuttingDown` NACK as
*"recycling, NOT absent"* and re-probes on its own paced loop inside the caller's budget, so the
method emits the node as served by the **re-activated** address. No timer, no watchdog, no sleep;
over budget it errors with `AddressRecyclingException` rather than emitting a stale nothing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid. HubRecycleExtensions states the exception itself: from the ROOT hub the dispose and the read leave different off-router hubs (portal/nodeops-* and portal/reads-*), whose action blocks are independent, so the read can overtake the teardown and be answered by the still-live activation. The guarantee is now scoped to "every caller this class documents", with the root-hub case written out as the one documented exception — and a one-line form of it is in the AGENTS.md block, since that is where a reader decides which hub to call from.

Comment on lines +198 to +201
All four post to **their own hub**, which is the sanctioned shape and the reason `DisposeRequest`
carries a `Reason` at all. The stale-build case defaults to an **offer** — a banner over the
instance's still-working content — because the unconditional self-recycle it replaced made
publication frequency equal restart frequency and tore hubs out from under users mid-edit. The cost

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid. PackageInstaller.SettleRetypedRoot posts via hub.NodeOperationIssuingHub() at the package ROOT and then waits for it to answer, so it is a routed teardown, not a self-post; only the rebind watcher, the overlay self-heal and the stale-build convergence post to their own instance hub. The page now says "the first three post to their own instance hub" and names SettleRetypedRoot as the exception, and the router-exception sentence no longer claims all four are self-directed.

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

    3 files  ±0      3 suites  ±0   5m 39s ⏱️ -19s
2 341 tests ±0  2 341 ✅ ±0  0 💤 ±0  0 ❌ ±0 
2 342 runs  ±0  2 342 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 176734a. ± Comparison against base commit cc59e37.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

1 496 tests  ±0   1 494 ✅ ±0   7m 33s ⏱️ -6s
    4 suites ±0       2 💤 ±0 
    4 files   ±0       0 ❌ ±0 

Results for commit 176734a. ± Comparison against base commit cc59e37.

♻️ This comment has been updated with latest results.

…RCED release is not an adoption

Every one of the automatic review's findings was valid; each was re-verified against
the code or the chart before a word changed.

1. A pod restart / roll was listed with the operations that leave a live activation
   in place. It is not: a restart tears down every activation the pod hosts, so those
   addresses re-read on next access. The claim is now "every change a portal absorbs
   WHILE IT KEEPS RUNNING", with the restart case written out as its own paragraph —
   what a roll leaves open is what the FRESH activation then chooses, plus what is
   pinned at process start.

2. The operator `recycle` was described as adopt-before-compile. It is the opposite:
   MeshOperations.WithReleaseRequest stamps RequestedReleaseForce = true, and
   NodeTypeCompilationHelpers short-circuits on that flag — "A FORCE MEANS BUILD THE
   LIVE SOURCE, NOT SERVE ME WHATEVER A BUNDLE STILL RESOLVES" (#2818, which exists
   because a force used to re-adopt the very bytes the operator was replacing, #2813).
   So the verb is the rebuild-from-source remedy on a NodeType. The same-bytes text is
   re-scoped to a dispose-only recycle (which compiles nothing and re-binds the same
   store key) and to an UNFORCED trigger or a Modules:RequirePrebuilt mesh.

3. That stamp lands ONLY when the target is a NodeType node
   (`IsNodeTypeNode(curr) ? WithReleaseRequest(...) : curr`). Recycling an ordinary
   page is a plain teardown that recompiles nothing — now stated.

4. Modules:AutoRecycleOnStaleBuild was quoted at its code default (off) as if that
   were the fleet's: deploy/aks/values.aks.yaml:342 sets it "true" fleet-wide. Both
   are now stated, with the instruction to read the EFFECTIVE value, and the note that
   neither setting fires for a same-path byte change.

5. "None of this applies to a TEST" was wrong. TestStateIsolation.md says a shared
   TestCluster keeps activations between tests — "the grain caches its config, Test B
   … reads stale state from Test A". The exemption is now a fixture that builds and
   tears down its own mesh, not "it is a test".

6. RecycleNode's "the read is answered by the reactivated hub" carried no caveat. The
   implementation documents one: from the ROOT hub the dispose and the read leave
   different off-router hubs, so the read can overtake the teardown. Scoped and stated.

7. PackageInstaller.SettleRetypedRoot is not a self-post — it issues through
   NodeOperationIssuingHub() at the package root. Only three of the four automatic
   recyclers post to their own hub.

Re-verified: check-shared-rules.py --self-test 18/18; --local across all seven repos
with required-in widened to all seven (the step-3 shape) green; Documentation.Test
Release -warnaserror 0 Error(s), full suite 562 passed / 0 failed / 0 skipped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

    6 files  ±0      6 suites  ±0   11m 23s ⏱️ -6s
3 199 tests +5  3 007 ✅ +5  192 💤 ±0  0 ❌ ±0 
3 203 runs  +5  3 011 ✅ +5  192 💤 ±0  0 ❌ ±0 

Results for commit 176734a. ± Comparison against base commit cc59e37.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results

   17 files  ±0     17 suites  ±0   29m 23s ⏱️ -16s
8 262 tests +5  8 068 ✅ +5  194 💤 ±0  0 ❌ ±0 
8 267 runs  +5  8 073 ✅ +5  194 💤 ±0  0 ❌ ±0 

Results for commit 176734a. ± Comparison against base commit cc59e37.

@meshweaver-cloud
meshweaver-cloud Bot added this pull request to the merge queue Sep 17, 2026
Merged via the queue into main with commit 3a48c3f Sep 17, 2026
37 of 39 checks passed
@rbuergi
rbuergi deleted the docs/dispose-request-stale-grain-state branch October 10, 2026 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants