A source-built bundle's frameworkMvid is per-MODULE and unmatchable by any consumer

What is fixed, and what is not

#3293 landed and CD's no identity anchor for MeshWeaver.Maps is gone — the source arm now builds MeshWeaver.Compiler from the platform tree instead of scavenging it out of the module's publish output. That was the outage and it is over.

What it does not fix is what the recorded identity means. The build passes -p:Version="$VERSION" — the module package's version, different for every matrix entry — so MeshWeaver.Compiler.dll is compiled afresh per module and its MVID differs per module. The field stays a property of the MODULE, which is the thing #3211 exists to prevent.

The measurement

CD run 33867996265, one run, one platform commit (7d644de95), from the copies publish happened to drop:

module	recorded `frameworkMvid`
`MeshWeaver.AI`	`fc70b6e19be34b65983eb5fa15ce5243`
`MeshWeaver.Markdown.Collaboration`	`0a96ab4c9eb744558431b2a018dc4699`

Two identities for one platform build. Nothing downstream can use that: a consumer comparing "what I have landed" against a per-module value gets a different answer per module.

And neither value can match a consumer at all

A portal image is built with -p:CIRun=true on GitHub Actions, so AddCommitHashMetadata stamps AssemblyMetadata("MeshWeaverFrameworkIdentity", "g<sha>") on its MeshWeaver.Compiler, and FrameworkIdentity.ReadIdentity prefers that stamp over the MVID (#1660 WS3). So a consumer reads g<sha> and a source-built bundle states an MVID: never equal, by construction.

ModuleUpdateDecision (#3154) compares (version, framework identity) before answering SkipUpToDate. With an unmatchable identity that comparison cannot do its job, and the reconcile answers "up to date, identity could not be checked" — the exact failure #3211's own error text describes.

The fix that was measured to work

State it instead of deriving it. In the source arm the platform IS the checkout, so its identity is the commit:

sha="$(git -C "$GITHUB_WORKSPACE/meshweaver" rev-parse HEAD)"
identity=(--framework-mvid "g$sha")

--framework-mvid is the packer's own flag for "the platform's own reading", and g<sha> is one of the three tokens it documents (s<hash>, g<sha>, 32-hex MVID). One identity per platform commit, identical for every module in the run, and the same token the image stamps — so the comparison starts working. It also needs no extra compiler build per matrix entry.

That change is written and was green (28/28) as #3296, which I am closing in favour of #3293 rather than reverting a landed fix. The diff is three lines of shell plus the guard and RECIPE_VERSION that must move with it (module-build-key.py — bundles packed under the old recipe attest an identity no consumer can match, so they must be repacked rather than reused).

Suggested acceptance

A bundle packed by core's own CD and a portal image built from the same commit state the same framework identity — assert it once, rather than leaving it to be re-measured.

Fixed on main by #3306 (merged today). Verified against origin/main:

$ git show origin/main:.github/workflows/node-repo-module-pack.yml \
    | grep -A4 -F 'dotnet build "$anchordir/MeshWeaver.Compiler.csproj"' | grep -c -e '-p:Version='
0

The anchor build no longer carries the module's version, so one platform states one identity.

Provenance, since it reflects on the fix that caused this: #3293 (mine) fixed the RED half — the
anchor was read from the module's own publish output, absent for any module that does not
transitively reference MeshWeaver.Compiler — and introduced this silent half by passing
-p:Version="$VERSION" to the replacement build "to match the module build's flags". MSBuild writes
Version into the assembly, so the MVID moved per module:

-p:Version=1.3.18 (AI)         → 34337c31960d47f5b5251d32ef923fc6
-p:Version=1.0.24 (Essentials) → 7c1c4f70de084da78659e6bda495e1c5
no override, run A             → 71cc81badb364c5d8558ac5e7db6a44e
no override, run B             → 71cc81badb364c5d8558ac5e7db6a44e   ← identical

Four bundles, four identities, one platform, all non-blank, all green, none matchable by any
portal. Green-and-wrong replaced red — a strictly worse failure mode, because red stopped the pack
and this stopped nothing.

Still open and deliberately not closed by this: #3310 — both guards here are config-level and
cannot see the outcome fail. Nothing yet asserts that a run's bundles agree on one identity, and
"absent" must not read as "agrees".

Reopening — I closed this on the wrong evidence, and it is not fixed

I closed this earlier tonight on a source-level check: -p:Version= no longer appears in the
anchor build's command line after #3306. That is true and it is not the question. The question is
whether the bundles a run produces state one identity, and they do not.

Measured on CD run 33921229755 (2026-09-04T21:28Z, 42f0cbc4e) — every bundle's own
built against framework line, after #3306 merged:

MeshWeaver.Payments.Stripe          dc3a469042634045aacae69551a37288
MeshWeaver.Maps                     dc3a469042634045aacae69551a37288
MeshWeaver.AI                       86c646e96b1544d4ac38619bf4f49eb9
MeshWeaver.Markdown.Collaboration   740b3ec32fcf458da71c03827ae37600

Three identities for one platform, in one run. Confirmed on run 33921072103 as well — same four
bundles, same split.

The mechanism, and why the split is the evidence

The two that AGREE (Maps, Payments.Stripe) are exactly the two that do not reach
MeshWeaver.Compiler through their reference graph — the same pair whose missing anchor caused the
original #3293 outage. For them the anchor build genuinely compiles the compiler, unstamped, and they
land on a common MVID.

The two that DIFFER (AI, Markdown.Collaboration) both reach MeshWeaver.Compiler transitively via
MeshWeaver.Graph. Their module build — which carries -p:Version="$VERSION" — has already built
MeshWeaver.Compiler.dll into meshweaver/src/MeshWeaver.Compiler/bin/Release/net10.0/. The
anchor build that follows finds it up to date and does nothing, so the anchor is the
version-stamped artifact from the module build, and each module stamps its own version.

So #3306 removed the flag from the command but not from the artifact already on disk. An
up-to-date check silently defeated it.

Why this was invisible

Every bundle states a non-blank, plausible-looking MVID. Nothing is red. #3211's requirement — "a
bundle must state what it was built against" — is satisfied syntactically by four bundles that
disagree. This is the same green-and-wrong that #3293 introduced and that I have now failed to close
twice: first by shipping it, then by declaring it fixed from the diff rather than the output.

What should catch it

#3310's guard (#3315, merged 22:35Z) asserts precisely this — one run, one platform identity — and
no CD run has executed with it yet. The next one should go red on this and name it. If it does not,
that guard needs the same scrutiny.

Suggested fix direction: build the anchor into its own output path (or force it), so it cannot
adopt an artifact a module build left behind. --no-incremental or a distinct
-p:BaseOutputPath would both do it; the property to hold is that the anchor is produced by a build
whose inputs no module can influence.

My proposed mechanism is REFUTED — recording it so nobody else chases it

I hypothesised that the module build (which carries -p:Version="$VERSION") leaves a version-stamped
MeshWeaver.Compiler.dll on disk, and the anchor build that follows finds it up to date, does
nothing, and adopts it — which would explain why the two modules that reach MeshWeaver.Compiler
transitively (AI, Markdown.Collaboration) each state their own identity while the two that do not
(Maps, Payments.Stripe) agree.

Measured locally on MeshWeaver.Compiler, and it does not hold:

1) build WITH -p:Version=1.3.18            2cb72edceccb48b2a940251d64578181
2) rebuild WITHOUT the flag (incremental)  42cc369606104c0287a34bc9f957549a   ← DID recompile
3) rebuild WITHOUT the flag, --no-incremental
                                           42cc369606104c0287a34bc9f957549a   ← identical to (2)
4) FRESH build, clean bin+obj, no flag     42cc369606104c0287a34bc9f957549a
5) FRESH build again (determinism control) 42cc369606104c0287a34bc9f957549a

Three findings, each of which kills part of the story:

Changing the Version property invalidates the incremental build — (1)→(2) recompiles. The
anchor never adopts the stamped artifact.
--no-incremental changes nothing — (2)==(3). I wrote that fix and have discarded it rather
than ship a plausible non-fix.
The build is deterministic and leaves no residue — (4)==(5)==(2). A rebuild after a stamped
build yields the fresh MVID.

So the anchor build is not the cause, and I do not currently have the mechanism.

What is still true, and what to try next

The CI observation stands and is reproduced twice (runs 33921229755, 33921072103): three
identities across four bundles, with Maps/Payments.Stripe agreeing. The split is still the best
clue — it correlates exactly with whether a module reaches MeshWeaver.Compiler transitively — but
the local model of why is wrong.

Hypotheses I have NOT tested, in the order I would test them:

The bundles are packed by different lanes. There are at least two callers (Plugins: pack the module bundles the bake composes and Module bundles (floor)), and a different caller can mean a
different platform input entirely. Compare the two jobs' resolved REFS/platform-image-digest
before assuming they took the same arm.
The REFS arm vs the from-source arm. If a platform image is pinned for some entries and not
others, the anchor comes from the image's /app for one and from source for the other — two
legitimately different identities, and then the defect is in expecting them to match at all.
Path/inputs differing per job. Determinism is a function of inputs including paths; identical
here locally, but worth confirming in the runner environment rather than assumed.

#3310's guard (#3315, merged 22:35Z) asserts one-run-one-identity and has not yet executed on a CD
run. The next run should go red and print the disagreeing set, which is a better starting point than
any of the above. I would wait for that rather than guess again — I have now been wrong about this
defect twice, once by shipping it and once by declaring it fixed.

Fix merged in #3339. Leaving this open until a real CD run demonstrates it — I closed this issue
once before on evidence that could not have distinguished the fix from the defect, and had to reopen
it against myself. Not repeating that.

Verified on origin/main (the code shipped):

identity=(--framework-mvid "g$sha")     present
MeshWeaver.Compiler.csproj              0 occurrences — the per-job anchor build is gone
RECIPE_VERSION                          "3"  (recipe-2 bundles repack rather than reuse)

What is NOT yet verified, and is the actual acceptance: that a run's bundles now state ONE
identity. That is not something the diff can show — it is what #3310's All selected bundles built
gate asserts, and the next core CD run whose module-pack lane actually executes is the measurement.
Note the trap that already bit once here: the CD run that went green at 07:27Z skipped the
module-pack job entirely, so a green run is not evidence either. The signal is that job running and
passing.

Recorded so the next reader does not re-derive it: the three earlier cures all failed the same way,
by deriving the platform's identity from a DLL in a per-MODULE job —

A module bundle that states no framework identity is still publishable, so the #3154 comparison has nothing to compare — part 3 of Plugins#931 #3211 read it out of $PACKDIR (present only when the module transitively references the
compiler — a property of two reference graphs, written down as a property of the arm);
fix(ci): the bundle identity anchor is a platform property, not a module's reference graph #3293 built it from the platform tree (right location, still per job);
fix(#3176): the identity anchor must not carry the module's version — #3293's cure leaves one identity per module #3306 removed the -p:Version polluting that build (right property, still per job).

Run 33950389008 still stated three identities, and the split was exactly the reference graph: the
two modules that reach MeshWeaver.Compiler transitively had it compiled first by their own
module build, under their own properties, into the same output path. No property removable from the
anchor build can reach pollution that arrives from the module build upstream.

And determinism alone would not have been enough: a portal image stamps g<sha> and
FrameworkIdentity.ReadIdentity prefers that over the MVID, so a bundle stating 32-hex could
never have matched a consumer — not "usually differs", never, by construction.

Acceptance MET — measured on a real CD run, not inferred

I held this open after #3339 merged because the code shipping is not the acceptance and a green run
is not either (the run that went green at 07:27Z skipped module-pack entirely). Here is the
measurement.

Core CD run 7807, whose module-pack lane actually ran, all four bundles built:

module	stated framework identity
`MeshWeaver.AI`	`gb0d58eccebf4eae60b500195c83b40639b43606f`
`MeshWeaver.Markdown.Collaboration`	`gb0d58eccebf4eae60b500195c83b40639b43606f`
`MeshWeaver.Maps`	`gb0d58eccebf4eae60b500195c83b40639b43606f`
`MeshWeaver.Payments.Stripe`	`gb0d58eccebf4eae60b500195c83b40639b43606f`

One identity, every module — and All selected bundles built (the #3310 gate that refuses a lane
whose bundles disagree) completed success. Compare the state this issue was reopened on, run
33950389008 earlier today:

0051e7210e4144a6a18620a36e629d84  ← Maps, Payments.Stripe   (1 compiler build)
6553ea371eab49e4afdd0af20ea7ac59  ← Markdown.Collaboration  (2 compiler builds)
395909c22d3042d4b23fb055337db63b  ← AI                      (2 compiler builds)

Two properties, not one:

They agree. Three identities → one, across a lane whose split was exactly the reference graph
(the two modules reaching MeshWeaver.Compiler transitively had it compiled first by their own
module build, under their own properties, into the same output path).
It is the MATCHABLE form. gb0d58ecce… is g<sha> of the platform commit — the same token
AddCommitHashMetadata stamps into a CI-built portal image, which FrameworkIdentity.ReadIdentity
prefers over the MVID. A 32-hex MVID could never have equalled what a consumer reports; that
half was unfixable by determinism and is why "state it, don't derive it" was the fix rather than
"make the derivation stable". #3154's (version, identity) comparison can now actually answer.

b0d58ecce is core's merge of #3339 itself, so the identity names the commit the bundles were built
against — which is the whole point of the field.

Closing. The three earlier cures (#3211 → $PACKDIR, #3293 → build it from the platform tree,
#3306 → drop the -p:Version) all failed the same way: deriving the platform's identity from a DLL
sitting in a per-MODULE job. No property removable from the anchor build could reach pollution that
arrives from the module build upstream. The guard now forbids MeshWeaver.Compiler.csproj in the
pack step at all, so a fourth attempt down that road fails immediately.

review(#3554): a source-build lane is measured, never exempted

Navigation Menu

Repository navigation

A source-built bundle's frameworkMvid is per-MODULE and unmatchable by any consumer #3308

What is fixed, and what is not

The measurement

And neither value can match a consumer at all

The fix that was measured to work

Suggested acceptance

Reopening — I closed this on the wrong evidence, and it is not fixed

The mechanism, and why the split is the evidence

Why this was invisible

What should catch it

My proposed mechanism is REFUTED — recording it so nobody else chases it

What is still true, and what to try next

Acceptance MET — measured on a real CD run, not inferred

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

A source-built bundle's frameworkMvid is per-MODULE and unmatchable by any consumer #3308

Description

What is fixed, and what is not

The measurement

And neither value can match a consumer at all

The fix that was measured to work

Suggested acceptance

Activity

rbuergi commented on Sep 4, 2026

rbuergi commented on Sep 4, 2026

Reopening — I closed this on the wrong evidence, and it is not fixed

The mechanism, and why the split is the evidence

Why this was invisible

What should catch it

rbuergi commented on Sep 4, 2026

My proposed mechanism is REFUTED — recording it so nobody else chases it

What is still true, and what to try next

rbuergi commented on Sep 5, 2026

rbuergi commented on Sep 5, 2026

Acceptance MET — measured on a real CD run, not inferred

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions