A portal can serve STALE compiled code while reporting Ok/compiled — recycles, forced compile and the stale-build banner are all inert #2471

Activity

1. rbuergi commented on Aug 27, 2026

   

   rbuergi

   on Aug 27, 2026

   ContributorAuthor

   More actions

   Evidence for "which replica answered": memex is currently serving from replicas on TWO different framework identities

   Found while root-causing the prebuilt-adoption reports (Education#216 / SocialMedia#75 / Plugins#723). This bears directly on your settling question 2 ("whether the served assembly's MVID matches the one the compile produced").

   memex, 2026-08-27 09:00Z:

   memex-portal-deployment-54bcd4f496-r2tr7   READY=false  RESTARTS=68   ci.6011   (created 03:38Z)
   memex-portal-deployment-54cc8d6b4d-cjnqn   READY=true   RESTARTS=1    ci.5670
   memex-portal-deployment-54cc8d6b4d-qsxvp   READY=true   RESTARTS=0    ci.5670
   
   kubectl rollout status → error: deployment "memex-portal-deployment" exceeded its progress deadline
   lastState.terminated  → Error, exitCode 139
   

   The serving ci.5670 pods resolve framework identity sceaefc9…. memex-cloud, which did roll, resolves s66ba35f…. So the fleet has been straddling a framework-identity boundary for hours because the new image will not stay up (that crash-loop is Plugins#723's binary break; note exitCode 139 is not proof of SIGSEGV — it fires on managed exceptions too, so the exception type still has to be read from the dump).

   Two replicas that disagree about the framework identity each consider the other's build stale, recompile, and re-stamp the node — which is a candidate generator for both symptoms recorded here and in SocialMedia#75 (the compiles arriving in near-simultaneous pairs 65 ms apart; "seven compiles for one merge"; a node whose record says Ok/lastCompiledAt seconds ago while the hub that answers your render is a different pod serving its own older assembly).

   🚨 I have not established this as the cause of the stale serve — it is a mechanism consistent with the evidence, not a measurement of it. What would settle it is exactly what you proposed: surface the served assembly's MVID alongside the record's, and have $Banner fire on a mismatch. With mixed-identity replicas live right now, that comparison is also cheap to take: the same node read through two pods should disagree.

   Separately, #2492 fixes a reporting defect that has been obscuring this whole family: NodeTypeBakeStatus.Classify checked the dependency record before the framework comparison, and the two are coupled (!toolchain hashes the FullMvidAssemblies closure MVIDs; FrameworkVersion folds the same ones), so a framework roll was reported as DependencyStale — "a bound module/toolchain dependency changed" — and FrameworkStale was unreachable. memex-cloud showed 273 of 273 uncovered types DependencyStale, zero FrameworkStale, with nothing changed but the image.
2. rbuergi commented on Aug 29, 2026

   

   rbuergi

   on Aug 29, 2026

   ContributorAuthor

   More actions

   PR for the detection half: #2655

   Your settling question 2 is the one it answers — "whether the served assembly's MVID matches the one the compile produced. That comparison, surfaced, is also the fix: $Banner must fire on a mismatch."

   While wiring it, the reason the banner was silent turned out to be structural rather than incidental, and it explains the inert recycles too: every staleness check compared LatestAssemblyPath, and a path is a store key. (nodeTypePath, LastCompiledVersion), resolved per pod through its own local cache, with no version advance on a recompile of an already-Ok type. So the key is identical on both sides while the bytes behind it differ — and a recycle re-resolves the same key from the same local copy. The adornment was not broken; it was blind, and so was every remedy built on the same comparison.

   What landed:

   • ServedBuildIdentity — MVID from a file or bytes, metadata only (nothing loaded), null on any unknown so a legacy node never fires.
   • NodeTypeDefinition.LatestAssemblyMvid, stamped by all three success writers.
   • The bind-time comparison, taken once per activation: the mismatch is true at bind time, which is why a publication-driven watcher could never catch it. It seeds the offer with the verdict and logs at Error, not Warning.
   • StaleBuildKind splits the two states: a same-key mismatch gets no recycle link and is never auto-recycled, because the recycle demonstrably does not help and spending the watcher's Take(1) on it would leave the instance unchanged with nothing left to fire.
   • get_diagnostics' Ok reply drops "…and is loaded" — a claim the formatter has no evidence for and which was false for 30 minutes — names the mvid as a field, and points at the check that is about the served bytes.

   🚨 This is detection, not convergence, and three things remain — which is why I am not proposing to close this:

   1. Getting a wedged instance onto the published bytes. The honest banner says the recycle will not help, and it is right. The cure is upstream, in how the local copy is keyed and invalidated: the store key is not unique per build. That is a change to the resolution path, not the reporting path.
   2. Your two-framework-identity mechanism is still unproven — you were explicit that it is consistent with the evidence rather than measured. This gives that measurement its instrument (the same node read through two pods should now disagree, loudly) but does not take it.
   3. No fleet-wide sweep. ServedBuildIdentity.Unverifiable exists so one can report what it could not check — an all-green count over an all-unknown fleet must not read as proof — but nothing calls it yet.

   For the 4 September talk the operational advice in your issue still stands unchanged: a green compile status is not evidence that the portal serves the fix. What changes is that the portal will now say so out loud when it is not.
3. rbuergi

   added 3 commits that reference this issue on Aug 29, 2026

   fix(graph): make "compiled Ok" checkable against the bytes actually s…

   ...

   eda0af5

   docs: what's new — a page now says when it is running old code (#2471)

   ...

   1693a7a

   Merge pull request #2655 from Systemorph/fix/served-bytes-are-verifiable

   ...

   695438a
4. rbuergi commented on Aug 30, 2026

   

   rbuergi

   on Aug 30, 2026

   ContributorAuthor

   More actions

   Fixed by #2655 (merged 2026-08-29): 'compiled Ok' is now checked against the bytes actually served — ServedBuildIdentity + NodeTypeDefinition.LatestAssemblyMvid, compared once per activation at bind time, logged at Error and surfaced on the banner; a same-key mismatch is never auto-recycled because the recycle demonstrably did not help. The adjacent defect the investigation uncovered — adoption stamping the live source snapshot onto bytes compiled from an older one — is its own issue, #2702.
5. rbuergi

   closed this as completedon Aug 30, 2026
6. rbuergi

   reopened this on Aug 31, 2026
7. rbuergi commented on Aug 31, 2026

   

   rbuergi

   on Aug 31, 2026

   ContributorAuthor

   More actions

   Reopening with a live occurrence — the detection this issue shipped fired tonight, which means the STALE-SERVE itself is back (a regression of the fix, or a path it never covered).

   Reproduction (Plugins e2e, runs 33348228704 + rerun, 2-of-2 on 1dc5fc41; also seen on the #1006 run without that diff):

   02:06:55.864 fail: MeshWeaver.Graph.Configuration.MeshNodeHubFactory
   Served build is NOT the published build for instance 'PaidPack6f550bc…':
   'Store/Plugin' reports a compiled build with MVID 7f5df496b0254ad09a2c9cd249029da4,
   but the bytes this instance bound are MVID 810f4775156648ebae1fc0e698678e64.
   A recycle re-binds the same local copy and will not change it (#2471).
   

   Blast radius in the same run — every symptom downstream of the one stale bind:

   • Family-wide content degradation on the affected hubs: StoreContent (Store root), all five Admin/Tiers/* TierContent, and the order's OrderContent all "stayed an untyped JsonElement" — the same-named-type-from-another-collectible-assembly identity split (ObjectAsExtensions case 3), which no WithType registration can cure.
   • The paid-fulfilment watcher never fires (reads/writes straddle two builds) → order stuck PendingPayment → Plugins Render timestamps in each viewer's local time zone (store UTC, display per-user) #455's "completed without an outcome", 3 occurrences tonight, blocking the Hosting: RotateRegistryKey — instance keys rotated through the Hosting plugin, never by hand (first verb of the record-driven deployment process) #2802 → Plugins#987 → Memex#151 rotation chain.
   • ObjectDisposedException observer drops at the mismatched instance's teardown (MeshDataSource.SubscribeToOwnDeletion).

   Suspected uncovered path: an INSTANCE hub minted mid-recompile binds the pre-publish bytes and is never rebound — the recycle wave that follows a publish reaches hubs that existed before the publish, not one whose creation raced it; and per this issue's own banner text, a recycle of that instance re-binds the same local copy.

   Fix direction: the factory already computes both MVIDs at the banner site — turn detection into a bounded self-heal at the bind seam (bind-time: refuse the stale copy and resolve the published bytes; or detection-time: schedule ONE rebind against the published MVID, guarded so a flapping publish cannot loop it). Taking this; design + PR to follow.
8. rbuergi commented on Sep 1, 2026

   

   rbuergi

   on Sep 1, 2026

   ContributorAuthor

   More actions

   Closing — both halves are fixed and merged.

   Detection half — PR #2655 (merged 2026-08-29) made the same-path MVID mismatch detectable: the stale-build banner and LogError fire on schedule. It was deliberately report-only, on the reasoning that "a recycle re-binds the same local copy and will not change it".

   Cure half — PR #2866 (merged 2026-08-31T02:39Z) closes exactly the gap that reasoning left open. Once bound, nothing downstream can fix it — but at activation, before the bind, both MVIDs are already in hand, and there a refusal is cheap. ShouldRefuseStaleBind now refuses on evidence (both MVIDs present and different) within a retry budget and routes into the existing TriggerRecompileAndRetry machinery; on budget exhaustion it falls through to bind-and-banner, because degraded-but-functional beats parked. Verified live at origin/main: ServedBuildIdentity/LatestAssemblyMvid are wired in NodeTypeCompilationHelpers.cs and NodeTypeContractHandler.cs.

   The 'forced compile is inert' third strand was split out as #2818 and fixed by PR #2824 (force release compiles live source).

   The PR bodies never used a GitHub auto-close keyword, which is the only reason this stayed open.

   🚨 One caveat worth recording for anyone reading this later, found while verifying: a sibling guard from the same family is armed but inert in production. ApplyAdoptedSourceStamp's AdoptionRefused branch can only fire when AdoptedSourceFingerprint is non-empty, and both production callers of PrebuiltAssemblySeeder.Seed (PluginBundleClient.cs:526, ShippedPrebuiltBundles.cs:624) use the 7-arg overload that hard-codes sourceFingerprint: null. That is #2813/#2820's territory, not this issue's, and it is being tracked there — noting it here only so this close is not read as covering it.
9. rbuergi

   closed this as completedon Sep 1, 2026