Prebuilt adoption stamps the LIVE source snapshot onto bytes compiled from an OLDER one — a stale bundle reports compiledSources == currentSourceVersions and runs old code as current

Measured (memex-cloud and memex.systemorph.com, 2026-08-29 ~21:00, plugins session)

After Plugins #868 merged (Edu/CourseCatalog source changed: a new ObserveCourses lane + 2 new test cases), git_hub_sync update on Edu logged:

Recompiling 1 NodeType(s): Edu/CourseCatalog
Adopted 1 prebuilt assembly(ies) — the rest compile

The node now reads compilationStatus: Ok, latestAssemblyPath: Edu_CourseCatalog/v2509-s770d75c-2a34a2c6efc5.dll (mvid 40ff945b…), and compiledSources == currentSourceVersions (CourseCatalogLayoutAreas = 639236350605419590) — yet the LIVE Tests area (after recycling the type and the Edu/Courses instance) lists only the OLD cases and the catalog still says "No courses published yet". The last sealed publication for that identity is from run 33261701592 @ 90c6be13, before #868. The gate adopted bytes compiled from the old source set and recorded them as compiled from the new one.

Not a shortfall (#2675/#2697/#2698): a wrong-bytes adoption that reports as current. Nothing downstream can detect it — the node's own record says the live source set is what's running.

Mechanism (`src/MeshWeaver.Graph/Configuration/PrebuiltAssemblySeeder.cs`, the adopt path ~l.300–400)

Adoption stamps LastCompiledVersion = node.Version, LatestAssemblyMvid = OfBytes(assemblyBytes), CompiledFrameworkVersion, and RequestedSourceStampAt = now — and, per its own comment, "the source snapshot is stamped BY THE OWNER, not here … 'bytes correspond to the LIVE source set'". That is exactly the assumption that broke: the adopted bytes correspond to the source set the bundle was sealed from, which is the live set only while nobody has changed a source since the seal. The adoption check keys on identity + type version; it never compares the bundle's recorded source snapshot with the node's CurrentSourceVersions.

Fix (root cause)

The bundle must carry the per-source snapshot it was compiled from (the bake already writes source-commit.txt per set; the per-NodeType record needs the compiledSources map — the same one the owner would stamp).
Adopt only if that snapshot equals the node's CurrentSourceVersions; otherwise DECLINE naming the source(s) that moved (Edu/CourseCatalog/Source/CourseCatalogLayoutAreas: bundle 6392363…, live 6392363…+) — the same DECLINED shape fix(gate): a bake shortfall names the assembly that declined #2675 introduced — and compile.
On adoption, stamp CompiledSources from the bundle's snapshot, never from the live set. compiledSources == currentSourceVersions must be a conclusion the comparison earned, not an assertion the stamp makes.

Interim on the fleet: the plugins session forced a real recompile via the compile tool — a stopgap, said as such; every NodeType whose source moved after its last seal is in this state until a bake re-seals it or the gate compares snapshots.

Related: #2675 (naming), #2697 (count), #2698 (module MVID skew — the same family: adoption trusts a key that does not cover the axis that changed).

Addendum: the stopgap does not exist — adoption wins over a forced recompile, every time

Measured on memex-cloud (plugins session, 21:21Z): compile on Edu/CourseCatalog set requestedReleaseForce=true, lastCompileSucceededAt moved to 21:21:56 — and the watcher re-adopted the same prebuilt bundle: latestAssemblyPath Edu_CourseCatalog/v2516-s770d75c-2a34a2c6efc5.dll, mvid 40ff945b… unchanged.

So while a sealed bundle exists for a NodeType's identity, that NodeType cannot be recompiled on a portal at all: the force-release path goes through the same adoption gate, the gate keys on identity + version, and the stale bytes win again. "Recompile" is not a verb operators have.

Consequence for delivery: a source change whose seal is stale (here #868's catalog fix) reaches a portal only via a fresh seal for that identity, or via the next identity after a platform roll. Every operator-facing "recompile"/"force release" affordance is currently a no-op under a sealed bundle, and reports success.

Adds to the fix: either the force-release path bypasses adoption (a forced compile must compile), or — the root-cause version — adoption compares source snapshots (fix 1–3 above), which makes the force unnecessary because a moved source is declined on its own.

The mechanism, confirmed in the code — and why the two halves need different fixes

Read PrebuiltAssemblySeeder's adopt path today. The defect is not hidden; it is stated as an assumption in the comment beside the write:

// 🚨 The source snapshot is stamped BY THE OWNER, not here (#1834). The producer's own
// ticks are meaningless on this mesh (the bake writes zeros), so adoption asserts
// "these bytes correspond to the LIVE source set" — and only the owner knows that set.
RequestedSourceStampAt = DateTimeOffset.UtcNow,

"Adoption asserts" is the whole bug. It asserts rather than checks, and the assertion holds only while nobody has changed a source since the bundle was sealed. When someone has — Plugins#868's Edu/CourseCatalog change — the adopt still fires, RequestedSourceStampAt asks the OWNER to stamp, and the owner dutifully writes CompiledSources = CurrentSourceVersions. The node then records stale bytes as compiled from the new sources, which is why nothing downstream can detect it: the node's own record says the live source set is what is running.

The adoption gate above keys on identity + node version (var version = node.Version, used for both the store key and the stamp). Neither is a function of the source text, so neither moves when a source does.

Why #1834's fix made this reachable rather than caused it

Worth stating so nobody "fixes" it by reverting that. #1834 was right: reading CompiledSources here stamped null under a non-empty CurrentSourceVersions (this write is cross-hub and diffs against the mirror's snapshot, which predates the first-activation write), so the type read IsDirty and the release request one step later recompiled what had just been adopted. Requesting the stamp removed that ordering problem. What it did not do — and could not, with the information available at that point — is establish that the bytes and the live sources agree.

The two halves, in the order they should be built

1. Root cause — adoption compares source snapshots. The bundle must carry the per-source map it was compiled from, and adoption must decline when that map differs from the node's CurrentSourceVersions. This is a bundle-format change spanning the producer (mw-plugin-test --bake-output / the batch-bake driver, which already writes source-commit.txt per set but no per-NodeType map) and this consumer. Declining correctly is the delicate part: on a mesh with Modules:RequirePrebuilt a decline must park the type with a named refusal, never silently fall back to compiling (#2193 §A).

2. The operator hole is separable and currently absolute. From the 08-29 addendum, measured: setting requestedReleaseForce=true on Edu/CourseCatalog moved lastCompileSucceededAt and re-adopted the same bundle — mvid 40ff945b… unchanged. So while a sealed bundle exists for a NodeType's identity, that NodeType cannot be recompiled on a portal at all, and every operator-facing "recompile" / "force release" affordance reports success while doing nothing.

I am deliberately not shipping half of this. Making force-release bypass adoption on its own would give operators a working verb and leave the silent-wrong-bytes path exactly as it is — and per this repo's own rule that is the band-aid, not the fix. The two want landing together, or (1) first: once adoption declines on a moved source, the force becomes unnecessary because the stale bundle is refused on its own.

Nothing measured here contradicts the issue body; this comment exists so the next reader starts from the confirmed mechanism and the fix's real shape rather than re-deriving both.

Retraction of my earlier comment here: I cited core's NodeTypeCompileParkTest.ParkedNodeType_CompileToolRetry_AfterFix_SettlesOkAndUnparks and the Reinsurance Ifrs17/AocConfiguration red as further faces of this adoption defect. Both attributions were wrong.

Those five failures are #890 — one Roslyn Emit poisoning the process. Measured: after 16:42:42.375, seven compiles reached the metadata writer and all seven failed identically over 6m15s with zero successes, while compiles needing only diagnostics stayed correct. Emit was dead, permanently. The CS1040 I read as "the retry compiled the stale source" came from the first compile, before the test wrote the fix; the retry ran a second later and returned NullReferenceException with canary=BELOW-ROSLYN.

The defect this issue describes is unaffected and still real — but the consequence for the fix is concrete: NodeTypeCompileParkTest never adopts anything, so it cannot be the regression test. ApplyAdoptedSourceStamp being a single pure function shared by all three writers makes it the right home instead, and unit-testable directly.

Correction: I called the operator hole "currently absolute". The code does not support that.

In my comment above I wrote that force-release is a no-op under a sealed bundle and that "every operator-facing recompile / force-release affordance is currently a no-op", sourced from the 2026-08-29 addendum. "Absolute" is an over-claim, and it has now been checked against the code by another session and by me:

src/MeshWeaver.Graph/Configuration/NodeTypeCompilationHelpers.cs:1530-1532 — "RequestedReleaseForce stays the user's escape hatch: an explicit force …", followed by the if (!def.RequestedReleaseForce … guard. Both absorb branches guard on it, so a forced request falls through to the commit path and dispatches a real compile.
GitSync adopts a stale prebuilt assembly over the source it just synced — silently running last week's code, and it destroyed client data #2813's own body contains a counter-observation on a different node: a forced compile moved the MVID (ed21acce… → 07d5ca73…).

So the recovery verb exists and has been seen to work. My paragraph generalised one reported observation into a property of the system, which it is not.

What the 08-29 observation still might be

Not dismissing it — it is specific (node Edu/CourseCatalog, memex-cloud, 21:21Z, latestAssemblyPath and mvid 40ff945b… both unchanged after requestedReleaseForce=true). It should be treated as unreproduced rather than as either fact or error.

One hypothesis worth testing, from the session that caught my over-claim: on a mesh with Modules:RequirePrebuilt, a forced compile is refused by design and the type parks — which from outside is indistinguishable from "the force did nothing". If that is what was measured, it is a real and separate gap deserving its own issue, not a property of adoption.

🚨 The obvious check does not settle it. git grep -c RequirePrebuilt origin/main -- deploy memex → 0 hits, matching #2194 item 3 ("the flag is enabled on no mesh"). But the repo is not authoritative about that instance: Memex#148 exists because configuration lives only on AKS, and Chart Drift measured 81 cluster-only settings, 64 of them inline env:. The check that would settle it is the running pod's environment, not a grep.

The part of my comment that stands

Everything about the adoption mechanism is unchanged and was read from the code rather than relayed: the adopt path asserts rather than checks ("adoption asserts 'these bytes correspond to the LIVE source set'"), the gate keys on identity + node version — neither a function of the source text — and RequestedSourceStampAt then has the owner stamp CompiledSources = CurrentSourceVersions, which is why nothing downstream can detect it.

Provenance note

I should also correct how I framed it: the force-release observation was not my measurement. It is the 2026-08-29 addendum's, and I repeated it without that qualifier when passing it to another session, where it was amplified into a stronger claim on a data-loss issue before a code read stopped it. Recording that here because the distinction between measured and read is exactly what made the difference.

Settled by measurement: the 08-29 addendum's node has moved on — "adoption wins over a forced recompile, every time" is refuted

The addendum of 2026-08-29 records Edu/CourseCatalog on memex-cloud stuck at latestAssemblyPath = Edu_CourseCatalog/v2516-s770d75c-2a34a2c6efc5.dll with mvid 40ff945b… unchanged after requestedReleaseForce=true, and concludes the operator hole is "currently absolute". That claim was then relayed (by me, among others) as "force-release is a no-op; operators have no recovery verb".

Read the node now:

latestAssemblyPath      Edu_CourseCatalog/v2594-scf0fcf9-20b44abb934b.dll     (was v2516-s770d75c-…)
latestAssemblyMvid      512453347e724c9c9c7a90a69c20d0f6                     (was 40ff945b…)
lastCompiledVersion     2594                                                  (was 2516)
compiledFrameworkVersion scf0fcf989c0d340d54c75b00bb14e1af                    (was s770d75c…)
lastCompileSucceededAt  2026-08-30T17:08:48Z
requestedReleaseForce   true      (requestedReleaseAt 2026-08-30T01:47:22Z, handled at the same instant)
compiledSources         ≡ currentSourceVersions

Both the assembly path and the MVID changed. Whatever the state was on 08-29, the node did not stay there, so "every time" and "absolute" are both false as written.

What this does NOT prove

I cannot attribute the movement. The framework moved too (s770d75c → scf0fcf9), and a framework change is itself an invalidation trigger — so this is consistent with the force worked, and equally with a later framework bump recompiled it and the force never did. It refutes the absolute claim; it does not establish that force is the thing that recovered it.

And the `RequirePrebuilt` hypothesis is dead for this instance

I proposed that Modules:RequirePrebuilt would explain a force that looks inert (the forced compile refused by design, the type parks — indistinguishable from "force did nothing" from outside). Measured on the live pods, not the repo — which matters, because Memex#148 records 81 cluster-only settings, 64 of them inline env:, so a git grep is not authoritative about that instance:

memex-cloud  pod env                → Modules__RequirePrebuilt  ABSENT
             deployment inline env  → ABSENT
             ConfigMap              → ABSENT
memex        (all three)            → ABSENT

Only Modules__AutoRecycleOnStaleBuild, Modules__Required__* and Modules__Root are set. So that explanation does not apply here.

Where that leaves it

The 08-29 observation has no explanation, and should be treated as unreproduced rather than as an established property of the system — which is what InstallReleaseRequestWatcher's code already says: both absorb branches guard !def.RequestedReleaseForce, so a forced request falls through to COMMIT and dispatches a real compile (NodeTypeCompilationHelpers.cs:1530-1532, "RequestedReleaseForce stays the user's escape hatch"), and #2813's own body records a forced compile moving the MVID.

The adoption defect itself is untouched by any of this. It is real, it is confirmed in code, and it caused the data loss in #2813. What is retracted is only the corollary that operators have no way out.

Third correction, and this one reverses my second: the 08-29 addendum was RIGHT in substance

I closed this thread earlier with "the node has moved on, so 'every time' and 'currently absolute' are false — the observation stands as unreproduced." A code-level investigation has now found the mechanism, and it reconciles both observations. Filed as #2818.

RequestedReleaseForce is honoured by InstallReleaseRequestWatcher — both absorb branches guard !def.RequestedReleaseForce, exactly as I read them — and is then silently discarded one layer down: InstallCompileWatcher never consults the flag, re-adopts the prebuilt bundle, and settles without a Roslyn pass. The node reports success, the bytes are unchanged, the MVID is unchanged.

So the honest reading of every measurement in this thread:

observation	who	verdict
force → same MVID, same assembly path (`Edu/CourseCatalog`, 08-29)	addendum	correct — the bundle still resolved, so the compile watcher re-adopted
the code lets a forced request through	me, and a peer	correct, and irrelevant — it is discarded further down
the node later moved (`v2516-s770d75c` → `v2594-scf0fcf9`)	me, measured today	correct — the framework moved, which is a real invalidation; the force is not what recovered it

All three are true. Force works exactly when SeedForTypes misses, and looks inert whenever the bundle still resolves — which is most of the time, and precisely when you need it.

What I got wrong, and the pattern

I read two guard clauses, found them correct, and concluded the feature was correct — without following the flag to the layer that decides. Then I measured a node that had moved and treated "the absolute claim is false" as though it settled the question, when it only bounded it. I was careful enough to say the measurement did not discriminate, and not careful enough to notice that "unreproduced" was itself a claim I had not earned: I had shown the state changed, not that the mechanism was absent.

The addendum's author measured the system; I measured the guard. Where those disagreed, the system was right.

Consequence for #2813

This is the missing recovery verb. Until #2818 is fixed, an operator whose type is serving an assembly built from older source has no way to force a build of the live source on any mesh where the bundle still resolves — which is the situation that destroyed client documents. That belongs in the same change set as the provenance work, not after it.

Un-retraction: the original observation was CORRECT, and my correction of it was wrong

I withdrew the force-release finding above. That withdrawal was a mistake and I am reversing it. The mechanism is #2818, and I verified it in the code rather than accepting it:

InstallCompileWatcher        spans lines  87 – 995
RequestedReleaseForce occurs at 1322, 1451, 1530, 1532, 1833, 1844   ← every one OUTSIDE that span
NodeTypeCompilationHelpers.cs:339 (inside the compile watcher):
        prebuiltConsumer.SeedForTypes([hubPath])                      ← adopts, unconditionally

So the flag is honoured by InstallReleaseRequestWatcher — the guard at :1530-1532 that I and another session both read, and read correctly — and then silently discarded one layer down. InstallCompileWatcher never consults it, re-adopts the prebuilt bundle, and settles without a Roslyn pass. Success reported, bytes unchanged, MVID unchanged.

All three observations in this thread are true simultaneously:

observation	verdict
force → same MVID and assembly path (`Edu/CourseCatalog`, 08-29)	correct — the bundle still resolved, so the compile watcher re-adopted
the code lets a forced request through (`!def.RequestedReleaseForce`)	correct, and irrelevant — it is discarded further down
the node later moved (`v2516-s770d75c` → `v2594-scf0fcf9`)	correct — the framework moved, a real invalidation. Force is not what recovered it

Force works exactly when SeedForTypes misses, and looks inert whenever the bundle still resolves — which is most of the time, and precisely when an operator needs it.

What went wrong in the correction, since it is the more useful lesson

My original error was real and worth fixing: I reported someone else's measurement in the first person. But I then let that procedural fault carry the substance with it, and downgraded a correct finding to "unreproduced". A second session over-corrected in the same direction, on the strength of a measurement it had already stated did not discriminate between "the force worked" and "a later framework bump recompiled it".

The correction impulse is not self-limiting. "I cannot reproduce it" is itself a claim that needs earning — showing that a state changed is not showing that a mechanism is absent.

Two people who had each read the guard agreed with each other, and were both wrong, because the guard was not the deciding layer. The addendum's author measured the system; we measured the code we expected to be responsible. Where those disagreed, the system was right.

The right correction would have been: "not my measurement — the source is the 08-29 addendum, and I have not verified it". That preserves the finding and flags the provenance. Instead I removed the finding.

Consequence

This is #2813's missing recovery verb. Until #2818 lands, an operator whose NodeType serves an assembly built from older source has no way to force a build of the live source on any mesh where the bundle still resolves — which is the exact situation in which client documents were lost. It belongs in that change set, not after it.

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-08-30)

🚨 This is now a data-loss defect — see #2813. The mechanism this issue describes (adoption stamping the live source snapshot onto bytes compiled from older source) destroyed client data on memex.systemorph.com today: a git_hub_sync update pulled new source for Crm/Migration, logged Adopted 2 prebuilt assembly(ies), reported Succeeded, ran the PREVIOUS code, stripped the bodies from four client documents and retyped two more. Four were recovered from version history; ClientA/EslLifeCycleProcess is permanently gone — one version in history, content indexing off, no chunk copy.

Two further faces of the same adoption, so the blast radius is not one portal: core's NodeTypeCompileParkTest.ParkedNodeType_CompileToolRetry_AfterFix_SettlesOkAndUnparks fails on a docs-only PR with the OLD source's CS1040 after the fix was written and awaited; and MeshWeaver.Reinsurance main went red at 16:29Z and green at 16:35Z on the same commit with bake-seed[0] Prebuilt assembly ADOPTED for Ifrs17/AocConfiguration at version 0 — no compile needed in between.

Consolidating on #2813, which carries the data-loss evidence and the proposed fix. The requirement this issue should adopt: the stale bytes are only half of it — compilationStatus: Ok and compiledSources == currentSourceVersions both reported the fix was live, so an operator cannot distinguish an adopted assembly from a compiled one. Only forcing a compile moved the MVID.

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-08-30)

Closing as a duplicate of #2813 — one defect, one home. Nothing here is being dropped; every strand has a live destination.

Both issues describe the same mechanism: adoption asserts rather than checks that the bundle's bytes correspond to the live source, and the gate keys on identity + node version, neither of which is a function of the source text. So a sync that changes a source moves neither key, the stale bundle is adopted, and the owner then stamps CompiledSources = CurrentSourceVersions — which is why nothing downstream can detect it: the node's own record says the live sources are what is running.

#2813 is the surviving issue because it carries what this one does not:

the production consequence — a git_hub_sync update ran the previous code over live client data, stripping the bodies from four documents and retyping two; ClientA/EslLifeCycleProcess is permanently gone (one version in history, content indexing off, no chunk copy);
the measurement that makes the fix's bar clear — compilationStatus: Ok and compiledSources == currentSourceVersions both read clean, so an operator cannot distinguish an adopted assembly from a compiled one; only a forced compile moved the MVID;
the fix, fix(#2813): an adopted build records whether it was ever checked against the source #2821 (BuildProvenance three-way: Verified / refuse on mismatch / AdoptedUnverified for legacy), in the merge queue.

The strands that started here and now live elsewhere:

strand	home
force-release appearing inert — traced to the compile watcher discarding `RequestedReleaseForce` and re-adopting without a Roslyn pass	#2818
refusing to execute unverified bytes matters more than refusing to load them	#2820
the MergeGuard face (`IsDirty` never converging on an adopted stamp)	#2463, still open

And the correction recorded here stays worth reading before anyone builds a staleness check: release-vs-NodeType pointer disagreement is noise — latestReleasePath lags routinely, and all fifteen types on the affected mesh had stale pointers while fourteen were healthy. It is the cheaper-looking neighbouring signal, and it cries wolf.

🚨 One thing #2821 does not do, restated so this closure is not read as more than it is: it changes what an adopted build says, not what it does. Legacy bundles still read IsDirty == false and still will not rebuild. Anyone who needs those types to actually recompile is blocked on #2818, not on this.

Navigation Menu

Repository navigation

Prebuilt adoption stamps the LIVE source snapshot onto bytes compiled from an OLDER one — a stale bundle reports compiledSources == currentSourceVersions and runs old code as current #2702

Measured (memex-cloud and memex.systemorph.com, 2026-08-29 ~21:00, plugins session)

Mechanism (`src/MeshWeaver.Graph/Configuration/PrebuiltAssemblySeeder.cs`, the adopt path ~l.300–400)

Fix (root cause)

Addendum: the stopgap does not exist — adoption wins over a forced recompile, every time

The mechanism, confirmed in the code — and why the two halves need different fixes

Why #1834's fix made this reachable rather than caused it

The two halves, in the order they should be built

Correction: I called the operator hole "currently absolute". The code does not support that.

What the 08-29 observation still might be

The part of my comment that stands

Provenance note

Settled by measurement: the 08-29 addendum's node has moved on — "adoption wins over a forced recompile, every time" is refuted

What this does NOT prove

And the `RequirePrebuilt` hypothesis is dead for this instance

Where that leaves it

Third correction, and this one reverses my second: the 08-29 addendum was RIGHT in substance

What I got wrong, and the pattern

Consequence for #2813

Un-retraction: the original observation was CORRECT, and my correction of it was wrong

What went wrong in the correction, since it is the more useful lesson

Consequence

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Prebuilt adoption stamps the LIVE source snapshot onto bytes compiled from an OLDER one — a stale bundle reports compiledSources == currentSourceVersions and runs old code as current #2702

Description

Measured (memex-cloud and memex.systemorph.com, 2026-08-29 ~21:00, plugins session)

Mechanism (src/MeshWeaver.Graph/Configuration/PrebuiltAssemblySeeder.cs, the adopt path ~l.300–400)

Fix (root cause)

Activity

rbuergi commented on Aug 29, 2026

Addendum: the stopgap does not exist — adoption wins over a forced recompile, every time

rbuergi commented on Aug 30, 2026

The mechanism, confirmed in the code — and why the two halves need different fixes

Why #1834's fix made this reachable rather than caused it

The two halves, in the order they should be built

rbuergi commented on Aug 30, 2026

rbuergi commented on Aug 30, 2026

Correction: I called the operator hole "currently absolute". The code does not support that.

What the 08-29 observation still might be

The part of my comment that stands

Provenance note

rbuergi commented on Aug 30, 2026

Settled by measurement: the 08-29 addendum's node has moved on — "adoption wins over a forced recompile, every time" is refuted

What this does NOT prove

And the RequirePrebuilt hypothesis is dead for this instance

Where that leaves it

rbuergi commented on Aug 30, 2026

Third correction, and this one reverses my second: the 08-29 addendum was RIGHT in substance

What I got wrong, and the pattern

Consequence for #2813

rbuergi commented on Aug 30, 2026

Un-retraction: the original observation was CORRECT, and my correction of it was wrong

What went wrong in the correction, since it is the more useful lesson

Consequence

rbuergi commented on Oct 11, 2026

rbuergi commented on Oct 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Mechanism (`src/MeshWeaver.Graph/Configuration/PrebuiltAssemblySeeder.cs`, the adopt path ~l.300–400)

And the `RequirePrebuilt` hypothesis is dead for this instance