Repository navigation
Design of record: modules ship as baked DLLs — no compile fallback, no source in the mesh DB #2193
Description
Mandate
Never get uncompiled/stale state to prod. Error early when a pre-built DLL is missing — never fall back to compiling. Remove all module source from the mesh DB. If Store updates, all dependent packages update as well. — Roland, 2026-08-25
This issue is the design of record for that program, motivated by today's Store outage on memex.
The incident (2026-08-25, memex)
Four merges to MeshWeaver.Plugins main (#637–#640, Store + Edu + Northwind) were picked up by the catalog auto-update (PluginUpdateWatcher → PackageUpdateReconciler → CatalogLayoutAreas.InstallOrUpdate) at 08:53 and 09:01 UTC. Every one of the 17 Store NodeTypes recompiled green (compilationStatus: Ok). The store page still went down:
- Serving never converged. Instances keep executing the previously bound assembly behind the "a newer build of this type is available — Recycle" banner (
StaleBuildBanner— the deliberate offer-not-restart design). On prod nobody clicks:/Storeserved a mixture of old and new assemblies. Tell in the logs:Received '$type':'StoreContent' which is NOT registered in this (receiving) hub— two hubs on different compiles of the same sources disagree about the type registry. - The Store/Catalog type hub wedged. Five portal replicas raced its recompile:
MonotonicWriteGuardconflicts onStore/CatalogandStore/Catalog/_Activity/compile-state,[UpdateRemote] LATE_NACK_TERMINAL … TimeoutExceptionon compile-state writes.@Store/area/Catalogtimed out until the type + instance hubs were manually recycled. - Dependents were never updated. The install path releases only the installed package's own types (
PackageInstaller→RequestReleasesover that package'snodeTypePaths). Packages that compileshared=@Store/...INTO their assemblies (Edu, courses, satellite-repo modules) were left on assemblies built against the old Store sources — the mixed-assembly$typemismatch above is exactly that.
"no dlls found" (as observed by Roland) is the consumer bundle lane resolving nothing and silently falling back to Roslyn:
PluginBundleClient— "No prebuilt bundle for {Plugin}@{Version} at {Registry} — will compile", "Bundle for {Plugin} carried no assemblies — compiling instead", "the registry could resolve NO artifact for {N} NodeType(s) on framework {Identity}/{Architecture} — they will be compiled here".InstanceAutoRegistrationService— "[DefaultInstall] {Id}: no prebuilt assemblies adopted — compiling instead".ShippedPrebuiltBundles— "bundle {Bundle} carries no assemblies".
Every one of these turns a distribution failure into a silent, slow, wedge-prone compile. That silence is the design flaw.
Target architecture
The runtime artifact of a module is a baked DLL per framework identity. Live Roslyn compilation stops being a fallback lane on production meshes. Module source does not live in the mesh DB.
A. Fail early, never compile (Modules:RequirePrebuilt)
A mode flag for production meshes. When on:
- A bundle miss (
NotServed,NoAssemblies,BundleDeclined, per-type misses) FAILS the install/update early, naming the package, bundle, framework identity and architecture — never "will compile". - Boot: a dynamic NodeType not covered by adopted bundles is an error (named list), and first-access does not compile — the type parks with a clear diagnostic naming the missing bundle/identity instead of paying a Roslyn compile.
- Bake/CI/dev meshes leave the flag off: the disposable-mesh gates, the bake itself, and local authoring still compile.
Seams to flip (all located): PluginBundleClient.Download/SeedAll miss branches, InstanceAutoRegistrationService DefaultInstall fallback, DynamicTypePreWarmer/NodeTypeEnrichmentHelpers first-access compile path.
B. Source leaves the mesh DB — strictly in this order
a. Import stops persisting Source//Test/ as nodes (same mode flag): the GitSync/package import routes them to compile input only where compiling is legal (bake/CI meshes) and SKIPS persistence on prod. Without this first, any purge is undone by the next sync.
b. Adopt-only proven: every module type on the prod meshes resolves its assembly from the bundle store for the live identity; the early-error from (A) is deployed for the miss case.
c. Purge existing source nodes — destructive, human-gated, NOT before (a)+(b) are live. Sketch:
- enumerate per partition:
path:{pkg} scope:subtreenodes whose path matches*/Source/*or*/Test/*under a NodeType (the same resolutioncompile-check.pyuses); - verify each owning type's
latestAssemblyPathresolves in the assembly store for the live identity BEFORE deleting its inputs; - delete in a maintenance window, per package, with a dry-run count first; keep a
pg_dumpof affected schemas; - after purge: recycle type hubs, verify
CompilationStatusstays Ok (no recompile must ever be triggered — with (A) on, it would error loudly, which is the point).
C. GUI source browsing moves to GitHub
Viewing a type's source fetches the file from the module's repo (the {Space}/_GitSync config already knows repo/branch/subdirectory). Private repos: the REGISTRY re-serves the file through its GitHub App credential — the same credential encapsulation as plugin serving — so consumer meshes need no GitHub access. The DB source nodes stop being the browsing surface before they are purged.
D. Dependent cascade at BUNDLE level
- A Store bundle rebuild must rebuild every dependent bundle in the same publication (in-repo:
affected-modules.pyalready closes over dependents dependents-first and the bake narrows by it — keep). - Cross-repo is the open gap: a Store change in MeshWeaver.Plugins does not trigger the education/SocialMedia/Reinsurance bakes until their next own push or release poll — stale dependent bundles for up to a poll cycle. Arm the platform's dormant
notify-dependentsdispatch (BAKE_SUBSCRIBER_REPOS/DEPENDENT_DISPATCH_TOKEN— needs a human-minted PAT with repo scope on the four satellites). - Until adopt-only lands, the in-mesh equivalent is missing too:
InstallOrUpdateshould close release requests over cross-package dependents (owners ofshared=@{pkg}/…references —LocalNodeRepo.CollectDependencieshas the edge semantics,BuildOrder/PackageGraphthe ordering). Under adopt-only this becomes "adopt the dependent bundles of the same publication", which the sealed-complete publication already guarantees exists.
E. Convergence after adopt/recompile — landed
#2192: Modules:AutoRecycleOnStaleBuild — instances self-recycle onto a newly published build (assembly-path gated, settle-window throttled, once per binding). Prod portals must set the key (config/Helm change, tracked below). This closes incident leg 1 regardless of how far A–D have progressed.
F. The compile-state replica race
Incident leg 2 (five replicas racing one type's recompile; LATE_NACK_TERMINAL on compile-state) disappears entirely under adopt-only — nothing on prod compiles. If any compiling surface survives on multi-replica meshes in the interim, the release watcher needs single-flight (the compile-state node as the lease, idempotent settle instead of racing writes). Documented here so it is not re-discovered; not worth building if A lands promptly.
Already in place to build on
PluginBundleClient+ the registry's per-identity/per-arch artifact resolution (A module must always carry a resolvable link to its shipped DLL (per framework identity, per arch) — routable via the registry, so nothing compiles #1751) — the consumer lane exists; the misses are what must become errors.PublishedBundleCatalogue+ReleaseAvailability(Self-update must refuse to roll the platform until every DEPLOYED plugin has a build available for the target release #1754/A build must not trigger until every upstream dependency is available for the target framework — otherwise exit, don't build #1755) — the availability predicate and release gates.- The satellites'
publish-bake(sealed_completepublications per identity) andShippedPrebuiltBundlesadoption. affected-modules.pydependents-first closure;BuildOrder/PackageGraph.
Rollout order
- Convergence opt-in: auto-recycle instances onto a newly published build #2192 merged + prod portals set
Modules:AutoRecycleOnStaleBuild(immediate outage-class fix). - A (
Modules:RequirePrebuilt) with the named-error seams + tests. - B(a) import mode; C browsing seam (can proceed in parallel with A).
- D cross-repo dispatch armed (PAT — human action).
- B(b) verification on prod; then B(c) purge, human-gated.
Cross-references: today's plugin-side wave Systemorph/MeshWeaver.Plugins#637 #638 #639 #640 (the update that surfaced this), and the plugins-repo tracking issue.
Addendum: the measured memory cost of non-convergence (2026-08-25)
Peer-session measurement (container_memory_working_set_bytes, memex-cloud) of the same incident — the retention half of the root cause:
- The two type-hosting silo pods (born 01:26 from a portal roll) sat flat at 2.7–4.5 GB for five hours.
- Inflection 06:10–06:30, immediately after the 05:55 scheduled bake tick — the first after MeshWeaver.Plugins#634 bumped the platform pin: new framework identity → full-catalog rebake → adoption → recompile of every type on the type-hosting silos.
- ~2.5 GB/h climb through the Patch ack by write identity, not emission counting (second acked-write-loss mode) #635 and Auth: storage/read timeouts surface as 401 Unauthorized #637–StaticRepoImporter: incremental skip fires on an EMPTY partition (phantom manifest checkpoint) #640 auto-synced recompiles. Peak 20.4 GB / 17.2 GB, ~4 cores spent in GC. Later-born pods stayed ≤3.6 GB — only the silos hosting non-converged instances fattened.
- No OOMKill (28Gi never reached); liveness/readiness probe
/alive(trivial), so the degraded pods stayed in rotation ~3.5 h.
Mechanism: superseded AssemblyLoadContexts stay rooted by serving instances that never converge onto the new build. Every recompile mints a new ALC; every instance still bound to an old one keeps it — and its statics, type registries and JIT'd code — alive. The mixed-assembly $type wedge (§incident leg 1) was the user-visible half; this is the memory half, and it compounds per update until a recycle or pod death. #2192's convergence opt-in releases the last roots of each superseded ALC as instances converge — and under the target adopt-only architecture (§A–B) prod never mints per-update ALC churn from Roslyn at all: one adoption per bundle publication.
Owned elsewhere (do not duplicate here): prod chart/values enablement of Modules:AutoRecycleOnStaleBuild for both portals, the RCA issue + runbook (superseded-build retention cost model, ready-but-degraded triage), and the progress-aware /alive probe (maintainer's open item).
Activity
Merge pull request #2198 from Systemorph/feat/require-prebuilt
Merge pull request #2200 from Systemorph/feat/import-source-nodes-mode
Merge pull request #2215 from Systemorph/feat/source-browsing-via-reg…
rbuergi commentedon Aug 25, 2026
§D addendum — a bundle is COVERAGE only if it carries the pack's static web assets
Fail-early is only an improvement if "prebuilt exists" means "prebuilt is complete." Today it
does not: a landed store pack carries assemblies only, so
RequirePrebuiltwould happily accepta bundle that renders the pack's pages unstyled and 404s its collocated JS. That is strictly worse
than the compile fallback it replaces, because the mesh is green while the product is broken —
the same silent-failure shape as the rest of this issue.
Evidence (peer's adopt-lane canary, re-verified from outside 2026-08-25)
Every landed Blazor pack has
wwwroot= 0 files.MeshWeaver.Blazor.Viewslands 11 filesagainst the image copy's 58 — the missing 47 are
_content/,<Name>.styles.cssand<Name>.modules.json.It has already shipped a prod regression.
MeshWeaver.Blazor.EntityViewsis store-only sinceEntityViews source moves out — the pack lives in MeshWeaver.Plugins (#2169) #2188, and its scoped CSS is gone on memex:
/_content/MeshWeaver.Blazor.EntityViews/MeshWeaver.Blazor.EntityViews.styles.css/_content/MeshWeaver.Blazor.Views/MeshWeaver.Blazor.Views.styles.css/_content/MeshWeaver.Blazor.Graph/MeshWeaver.Blazor.Graph.styles.cssThe two 200s are not evidence the lane works — those packs still ship in the image, so the
image copy answers. EntityViews is the first pack with no image copy to hide behind, and it is the
one that 404s. Every pack that leaves the image next inherits this.
Where it is NOT
The consumer landing side is already correct and should not be touched:
PluginBundleClientreadsBundleReader.ReadModuleAssets(bundleBytes)and passes them toModuleLandingService.LandModule, which writes each asset at its relative path under the modulefolder for the host's asset provider (#1724). It lands nothing because the bundle contains
nothing. Suspected fix site is the packer: a plain RCL
dotnet publishdoes not compose_content/ scoped CSS — that composition happens in the consuming app's publish. (The peer isrunning the exact publish to confirm and owns that fix.)
The condition to add to §D
What makes that checkable — and why the check cannot be a heuristic
Consumer-side there is currently no honest signal: most modules legitimately have no
wwwroot,so "0 assets" cannot be distinguished from "0 assets expected". Guessing by name
(
MeshWeaver.Blazor.*) would put a guess on a fail-early path — the one place a false positivetakes the mesh down.
So the packer fix should also declare what it composed, and the coverage check follows for free:
declared > 0 && carried == 0→Miss(...)(and underRequirePrebuilt,PrebuiltRequiredException) — same funnel as the partial-assembly miss already atPluginBundleClient.cs:488, which fails a require-prebuilt mesh on partial coverage rather thanseeding half a module;
condition takes out every bundle baked before the packer fix.
That keeps the rule declarative (the producer states what complete means for this pack) instead of
the consumer inferring it, which is the same reason
requiresis declared rather than remembered.Not built here — recorded as the design condition; the packer fix is the peer's lane.
rbuergi commentedon Aug 25, 2026
§D correction — the assets are dropped at the REGISTRY's publish seam, not by the packer
My previous comment named the packer as the suspected fix site. That is wrong and is superseded
here. The peer root-caused it (#2216): the publish path is healthy end to end except for two
arguments that are never passed.
publish/wwwrootcarries<Name>.styles.css,<Name>.modules.jsonand_content/, and thepacked bundle declares 254
staticAssets.PluginBundleEndpoints.MapPublish: it callsModulePublish.Validate(plugin, manifest, files, version, packagePath)andShelveModule(...)passing neither the assets to
Validatenoraccepted.StaticAssetstoShelveModule—while every other half was already built for it.
Validatetakes astaticAssetsparameter,AcceptedcarriesStaticAssets,ShelveModuletakesstaticAssets,LandCorematerializesthem, and the consumer's
PluginBundleClient.LandFromBundlepasses them correctly.So: publish healthy, packer healthy, consumer healthy — two missing arguments at the warehouse
door meant every module ever published landed assemblies-only.
The mechanism is worth naming because it is this issue's own theme.
ShelveModule's parameter isoptional with a
nulldefault:A caller that forgets it does not fail to compile, does not warn, and shelves a bundle that is
accepted, recorded, downloadable and loads fine. The only visible symptom is CSS that never
arrives — which is why it survived from the first publish until a pack (
EntityViews) finally hadno image copy to hide behind.
What changes in §D
The completeness condition is enforceable AT THE SEAM, which is better than the
consumer-side declaration check I proposed in the previous comment — the producer is where
"complete" is known. The peer's fix adds a test that publishes through the endpoint and asserts
the asset exists on disk under the landed generation. Without such a test the failure is
entirely silent: 200 response, entry recorded, module loads, pages render unstyled.
Migration consequence — every already-published pack must be REPUBLISHED. The fix does not
retro-fill: the shelf holds asset-less bytes for everything published before it. Therefore:
A version/timestamp floor on the shelf entry is the cheap discriminator; whatever the mechanism,
"published before the fix" has to be expressible, or the republish cannot be verified — only
assumed.
Evidence for the regression itself stands as measured (re-verified from outside, 2026-08-25):
/_content/MeshWeaver.Blazor.EntityViews/MeshWeaver.Blazor.EntityViews.styles.css→ 404, whileViewsandGraphreturn 200 only because their image copies still answer.Fix lane and the seam test are the peer's (#2216); recorded here as the design condition.
Merge pull request #2209 from Systemorph/feat/require-prebuilt-park
rbuergi commentedon Aug 25, 2026
§D's stated action is obsolete — flagging rather than editing the body.
Verified against
origin/main@9494b1219: the cross-repo fan-out was redesigned.main-cd.yml:1400-1425moved it to memex (src/MeshWeaver.GitSync/FrameworkReleaseBroadcaster.cs), and the design deliberately holds no satellite write credential. So §D's instruction to mint a PAT and armBAKE_SUBSCRIBER_REPOS/DEPENDENT_DISPATCH_TOKENshould be struck — #2235 says the same explicitly.Suggest also deleting the vestigial
BAKE_SUBSCRIBER_REPOSrepo variable, so it cannot be mistaken for the live list.RequirePrebuiltParkTestshows the gate is mesh-wide — it parksRequirePrebuiltTest/NeverBaked, a type in a partition that is not a package. TurningModules:RequirePrebuilton today would park every user-authored NodeType on memex/memex-cloud, including everything created through the MCP tools. That contradicts #1707 requirement 4 ("there will always be code without any CI — bake-your-own stays as the universal fallback"), which this mandate scopes to module source and does not overturn. A provenance predicate is needed atNodeTypeCompilationHelpers.cs:283and inPrebuiltAssemblySeeder.RequiredParkReasonbefore prod enablement.Status of the rest, verified: §A code seams shipped but
RequirePrebuiltis enabled nowhere (git grep RequirePrebuilt origin/main -- deploy/returns nothing). §E shipped and enabled fleet-wide (values.aks.yaml:295 Modules__AutoRecycleOnStaleBuild: "true").rbuergi commentedon Aug 25, 2026
Maintainer decision (2026-08-25): MeshWeaver.Manufacturing is EXCLUDED from the adopt-only estate for now.
Consequence for this program: Manufacturing does not need a bake/publish lane before
Modules:RequirePrebuiltis enabled, and it is not a blocker on that stage. It currently has no bake or publish lane at all (ci.yml:preflight, validate, tag-modules, compile-check, test-repos) and noBAKE_PUBLISH_TARGETS— that stays as-is.The satellite adoption order is therefore: Plugins → Education → Reinsurance → SocialMedia, with Manufacturing out of scope.
🚨 Still outstanding before the flag can flip, and unaffected by this decision:
RequirePrebuiltParkTestshows the gate is mesh-wide — it parks a type in a partition that is not a package. Enabling it today would park every user-authored NodeType on memex/memex-cloud, including everything created through the MCP tools, which contradicts #1707 requirement 4 ("bake-your-own stays as the universal fallback"). A provenance predicate atNodeTypeCompilationHelpers.cs:283and inPrebuiltAssemblySeeder.RequiredParkReasonis the remaining prerequisite.rbuergi commentedon Aug 25, 2026
Maintainer decision (2026-08-25) on §A scoping:
RequirePrebuiltapplies to everything WE MARK as such — and the marking should cover almost all of it.This resolves the blocker raised earlier in this thread. The gate must key on an explicit mark, not on the mesh:
RequirePrebuiltenforces. No adopted assembly ⇒ park with the named refusal, never compile. This is the mandate's "error early when a pre-built DLL is missing — never fall back to compiling".Why this matters concretely:
RequirePrebuiltParkTestcurrently proves the gate is mesh-wide — it parksRequirePrebuiltTest/NeverBaked, a type in a partition that is not a package. Enabling the flag as it stands today would park every user-authored NodeType on memex and memex-cloud, including everything created through the MCP tools, and render the compilation-error overlay to anonymous visitors. Echoes #683, where nine plugin partitions went to that overlay in production.So the remaining prerequisite is now well-defined, and it is the marking + predicate rather than a policy debate:
NodeTypeCompilationHelpers.cs:283and inPrebuiltAssemblySeeder.RequiredParkReasonreads that mark rather than the mesh-wide flag.GetCompileAttemptCount == 0; an unmarked type in the same mesh compiles normally.Only after (1)–(3) does
Modules:RequirePrebuiltbecome safe to set indeploy/aks/values.aks.yaml(it is currently enabled nowhere —git grep RequirePrebuilt origin/main -- deploy/returns nothing).Related decision recorded above: MeshWeaver.Manufacturing is excluded from the adopt-only estate for now, so it is not a blocker on this stage.
rbuergi commentedon Aug 25, 2026
Two prerequisites for the marked-types decision are now root-caused, and one is a new follow-up (#2264) that matters specifically for enabling this broadly.
#2260 — a parked type could be silently un-parked (fix in PR #2262, in CI). This mattered because your directive rests on a refusal that parks the type so no later trigger can storm it — and that containment guarantee was false: the failed-verdict re-drive lifted the park, then re-parked, and anything reading inside that window saw an unparked type.
The mechanism was one step further than #2260's own hypothesis, established by instrumenting the registry rather than inferring from the double-log:
So there is exactly one un-park, and the fix had to close the window rather than shorten it: the re-drive now asks to be let through (a one-shot admission the parked short-circuit consumes) and never clears the failure. Un-parking stays reserved for the two events that genuinely end one — a human-requested build, and a successful compile.
#2264 —
SettleAsErrornever stampsFailedBuildInputs, so the adopt-only gate's refusal reads as "never attempted". Every refused type therefore burns one automatic re-drive plus a needless Pending round-trip on the owning hub's action block.🚨 Why that matters here specifically: under your decision, "almost all" types become marked. If each refusal costs a wasted re-drive and a Pending round-trip, that cost is multiplied by the number of marked-but-unbaked types at any moment — exactly the condition during a rollout, when the bake lane is catching up. It is not the race (#2260 fixes that); it is the trigger that made the race reachable, and it is a scaling concern for enabling the flag fleet-wide.
#2264 is deliberately not folded into the #2260 fix: the two
SettleAsErrorcall sites need different answers (stamping on the parked short-circuit would mask a genuine input change and defeat #1793's recovery), so it is a design question rather than a one-liner.Merge pull request #2200 from Systemorph/feat/import-source-nodes-mode
Merge pull request #2215 from Systemorph/feat/source-browsing-via-reg…
Merge pull request #3015 from Systemorph/docs/program-state-of-record
rbuergi commentedon Sep 1, 2026
Closed by #3015 (merged as
8a31eb40), which converts this program to a committed state-of-record page.The PR body said "Closes #1707, #2193, #2550 and #2802" — GitHub only parses the keyword before the first number, so only #1707 auto-closed. Closing the rest by hand; the work is landed either way.
Where this now lives:
Both ship with the platform under
Doc/Architecture/, so the next session finds them without reading an issue thread.🚨 The remaining work is named in those pages, not lost. The sharpest items:
Modules:RequirePrebuiltis real, load-bearing, and set in ZERO config files — so "never fall back to compiling" is provided for and enforced nowhere.Carried out in the same PR: deleted the dead
CompilationLock(no caller, andasync/Task.Delay-based, which the house rules forbid); taught the operator gate the direction that rots (it could only fail when a script was deleted, never when one shipped unclaimed); and gavehosting-kv-rotatebehaviour tests including the one property whose failure is unrecoverable — it must never print the minted key, mutation-proven.