Skip to content

Design of record: modules ship as baked DLLs — no compile fallback, no source in the mesh DB #2193

Closed

Description

@rbuergi

Mandate

Never get uncompiled/stale state to prod. Error early when a pre-built DLL is missing — never fall back to compiling. Remove all module source from the mesh DB. If Store updates, all dependent packages update as well. — Roland, 2026-08-25

This issue is the design of record for that program, motivated by today's Store outage on memex.

The incident (2026-08-25, memex)

Four merges to MeshWeaver.Plugins main (#637–#640, Store + Edu + Northwind) were picked up by the catalog auto-update (PluginUpdateWatcher → PackageUpdateReconciler → CatalogLayoutAreas.InstallOrUpdate) at 08:53 and 09:01 UTC. Every one of the 17 Store NodeTypes recompiled green (compilationStatus: Ok). The store page still went down:

  1. Serving never converged. Instances keep executing the previously bound assembly behind the "a newer build of this type is available — Recycle" banner (StaleBuildBanner — the deliberate offer-not-restart design). On prod nobody clicks: /Store served a mixture of old and new assemblies. Tell in the logs: Received '$type':'StoreContent' which is NOT registered in this (receiving) hub — two hubs on different compiles of the same sources disagree about the type registry.
  2. The Store/Catalog type hub wedged. Five portal replicas raced its recompile: MonotonicWriteGuard conflicts on Store/Catalog and Store/Catalog/_Activity/compile-state, [UpdateRemote] LATE_NACK_TERMINAL … TimeoutException on compile-state writes. @Store/area/Catalog timed out until the type + instance hubs were manually recycled.
  3. Dependents were never updated. The install path releases only the installed package's own types (PackageInstaller → RequestReleases over that package's nodeTypePaths). Packages that compile shared=@Store/... INTO their assemblies (Edu, courses, satellite-repo modules) were left on assemblies built against the old Store sources — the mixed-assembly $type mismatch above is exactly that.

"no dlls found" (as observed by Roland) is the consumer bundle lane resolving nothing and silently falling back to Roslyn:

  • PluginBundleClient — "No prebuilt bundle for {Plugin}@{Version} at {Registry} — will compile", "Bundle for {Plugin} carried no assemblies — compiling instead", "the registry could resolve NO artifact for {N} NodeType(s) on framework {Identity}/{Architecture} — they will be compiled here".
  • InstanceAutoRegistrationService — "[DefaultInstall] {Id}: no prebuilt assemblies adopted — compiling instead".
  • ShippedPrebuiltBundles — "bundle {Bundle} carries no assemblies".

Every one of these turns a distribution failure into a silent, slow, wedge-prone compile. That silence is the design flaw.

Target architecture

The runtime artifact of a module is a baked DLL per framework identity. Live Roslyn compilation stops being a fallback lane on production meshes. Module source does not live in the mesh DB.

A. Fail early, never compile (Modules:RequirePrebuilt)

A mode flag for production meshes. When on:

  • A bundle miss (NotServed, NoAssemblies, BundleDeclined, per-type misses) FAILS the install/update early, naming the package, bundle, framework identity and architecture — never "will compile".
  • Boot: a dynamic NodeType not covered by adopted bundles is an error (named list), and first-access does not compile — the type parks with a clear diagnostic naming the missing bundle/identity instead of paying a Roslyn compile.
  • Bake/CI/dev meshes leave the flag off: the disposable-mesh gates, the bake itself, and local authoring still compile.

Seams to flip (all located): PluginBundleClient.Download/SeedAll miss branches, InstanceAutoRegistrationService DefaultInstall fallback, DynamicTypePreWarmer/NodeTypeEnrichmentHelpers first-access compile path.

B. Source leaves the mesh DB — strictly in this order

a. Import stops persisting Source//Test/ as nodes (same mode flag): the GitSync/package import routes them to compile input only where compiling is legal (bake/CI meshes) and SKIPS persistence on prod. Without this first, any purge is undone by the next sync.
b. Adopt-only proven: every module type on the prod meshes resolves its assembly from the bundle store for the live identity; the early-error from (A) is deployed for the miss case.
c. Purge existing source nodes — destructive, human-gated, NOT before (a)+(b) are live. Sketch:

  • enumerate per partition: path:{pkg} scope:subtree nodes whose path matches */Source/* or */Test/* under a NodeType (the same resolution compile-check.py uses);
  • verify each owning type's latestAssemblyPath resolves in the assembly store for the live identity BEFORE deleting its inputs;
  • delete in a maintenance window, per package, with a dry-run count first; keep a pg_dump of affected schemas;
  • after purge: recycle type hubs, verify CompilationStatus stays Ok (no recompile must ever be triggered — with (A) on, it would error loudly, which is the point).

C. GUI source browsing moves to GitHub

Viewing a type's source fetches the file from the module's repo (the {Space}/_GitSync config already knows repo/branch/subdirectory). Private repos: the REGISTRY re-serves the file through its GitHub App credential — the same credential encapsulation as plugin serving — so consumer meshes need no GitHub access. The DB source nodes stop being the browsing surface before they are purged.

D. Dependent cascade at BUNDLE level

  • A Store bundle rebuild must rebuild every dependent bundle in the same publication (in-repo: affected-modules.py already closes over dependents dependents-first and the bake narrows by it — keep).
  • Cross-repo is the open gap: a Store change in MeshWeaver.Plugins does not trigger the education/SocialMedia/Reinsurance bakes until their next own push or release poll — stale dependent bundles for up to a poll cycle. Arm the platform's dormant notify-dependents dispatch (BAKE_SUBSCRIBER_REPOS / DEPENDENT_DISPATCH_TOKEN — needs a human-minted PAT with repo scope on the four satellites).
  • Until adopt-only lands, the in-mesh equivalent is missing too: InstallOrUpdate should close release requests over cross-package dependents (owners of shared=@{pkg}/… references — LocalNodeRepo.CollectDependencies has the edge semantics, BuildOrder/PackageGraph the ordering). Under adopt-only this becomes "adopt the dependent bundles of the same publication", which the sealed-complete publication already guarantees exists.

E. Convergence after adopt/recompile — landed

#2192: Modules:AutoRecycleOnStaleBuild — instances self-recycle onto a newly published build (assembly-path gated, settle-window throttled, once per binding). Prod portals must set the key (config/Helm change, tracked below). This closes incident leg 1 regardless of how far A–D have progressed.

F. The compile-state replica race

Incident leg 2 (five replicas racing one type's recompile; LATE_NACK_TERMINAL on compile-state) disappears entirely under adopt-only — nothing on prod compiles. If any compiling surface survives on multi-replica meshes in the interim, the release watcher needs single-flight (the compile-state node as the lease, idempotent settle instead of racing writes). Documented here so it is not re-discovered; not worth building if A lands promptly.

Already in place to build on

Rollout order

  1. Convergence opt-in: auto-recycle instances onto a newly published build #2192 merged + prod portals set Modules:AutoRecycleOnStaleBuild (immediate outage-class fix).
  2. A (Modules:RequirePrebuilt) with the named-error seams + tests.
  3. B(a) import mode; C browsing seam (can proceed in parallel with A).
  4. D cross-repo dispatch armed (PAT — human action).
  5. B(b) verification on prod; then B(c) purge, human-gated.

Cross-references: today's plugin-side wave Systemorph/MeshWeaver.Plugins#637 #638 #639 #640 (the update that surfaced this), and the plugins-repo tracking issue.

Addendum: the measured memory cost of non-convergence (2026-08-25)

Peer-session measurement (container_memory_working_set_bytes, memex-cloud) of the same incident — the retention half of the root cause:

Mechanism: superseded AssemblyLoadContexts stay rooted by serving instances that never converge onto the new build. Every recompile mints a new ALC; every instance still bound to an old one keeps it — and its statics, type registries and JIT'd code — alive. The mixed-assembly $type wedge (§incident leg 1) was the user-visible half; this is the memory half, and it compounds per update until a recycle or pod death. #2192's convergence opt-in releases the last roots of each superseded ALC as instances converge — and under the target adopt-only architecture (§A–B) prod never mints per-update ALC churn from Roslyn at all: one adoption per bundle publication.

Owned elsewhere (do not duplicate here): prod chart/values enablement of Modules:AutoRecycleOnStaleBuild for both portals, the RCA issue + runbook (superseded-build retention cost model, ready-but-degraded triage), and the progress-aware /alive probe (maintainer's open item).

Activity

  1. added 3 commits that reference this issue on Aug 25, 2026
  2. rbuergi commented on Aug 25, 2026

    @rbuergi
    ContributorAuthor

    §D addendum — a bundle is COVERAGE only if it carries the pack's static web assets

    Fail-early is only an improvement if "prebuilt exists" means "prebuilt is complete." Today it
    does not: a landed store pack carries assemblies only, so RequirePrebuilt would happily accept
    a bundle that renders the pack's pages unstyled and 404s its collocated JS. That is strictly worse
    than the compile fallback it replaces, because the mesh is green while the product is broken —
    the same silent-failure shape as the rest of this issue.

    Evidence (peer's adopt-lane canary, re-verified from outside 2026-08-25)

    • Every landed Blazor pack has wwwroot = 0 files. MeshWeaver.Blazor.Views lands 11 files
      against the image copy's 58 — the missing 47 are _content/, <Name>.styles.css and
      <Name>.modules.json.

    • It has already shipped a prod regression. MeshWeaver.Blazor.EntityViews is store-only since
      EntityViews source moves out — the pack lives in MeshWeaver.Plugins (#2169) #2188, and its scoped CSS is gone on memex:

      URL Status
      /_content/MeshWeaver.Blazor.EntityViews/MeshWeaver.Blazor.EntityViews.styles.css 404
      /_content/MeshWeaver.Blazor.Views/MeshWeaver.Blazor.Views.styles.css 200
      /_content/MeshWeaver.Blazor.Graph/MeshWeaver.Blazor.Graph.styles.css 200

      The two 200s are not evidence the lane works — those packs still ship in the image, so the
      image copy answers. EntityViews is the first pack with no image copy to hide behind, and it is the
      one that 404s. Every pack that leaves the image next inherits this.

    Where it is NOT

    The consumer landing side is already correct and should not be touched:
    PluginBundleClient reads BundleReader.ReadModuleAssets(bundleBytes) and passes them to
    ModuleLandingService.LandModule, which writes each asset at its relative path under the module
    folder for the host's asset provider (#1724). It lands nothing because the bundle contains
    nothing. Suspected fix site is the packer: a plain RCL dotnet publish does not compose
    _content / scoped CSS — that composition happens in the consuming app's publish. (The peer is
    running the exact publish to confirm and owns that fix.)

    The condition to add to §D

    A bundle covers a pack only when it carries the pack's static web assets as well as its
    assemblies. An asset-less bundle for a pack that has assets is a miss, not coverage, and
    Modules:RequirePrebuilt must fail early on it exactly as it does on an absent bundle.

    What makes that checkable — and why the check cannot be a heuristic

    Consumer-side there is currently no honest signal: most modules legitimately have no wwwroot,
    so "0 assets" cannot be distinguished from "0 assets expected". Guessing by name
    (MeshWeaver.Blazor.*) would put a guess on a fail-early path — the one place a false positive
    takes the mesh down.

    So the packer fix should also declare what it composed, and the coverage check follows for free:

    • packer: record the composed asset list (or at minimum the count) in the bundle manifest;
    • consumer: declared > 0 && carried == 0 → Miss(...) (and under RequirePrebuilt,
      PrebuiltRequiredException) — same funnel as the partial-assembly miss already at
      PluginBundleClient.cs:488, which fails a require-prebuilt mesh on partial coverage rather than
      seeding half a module;
    • a bundle with no declaration is treated as legacy, warned once, not failed — otherwise the
      condition takes out every bundle baked before the packer fix.

    That keeps the rule declarative (the producer states what complete means for this pack) instead of
    the consumer inferring it, which is the same reason requires is declared rather than remembered.

    Not built here — recorded as the design condition; the packer fix is the peer's lane.

  3. rbuergi commented on Aug 25, 2026

    @rbuergi
    ContributorAuthor

    §D correction — the assets are dropped at the REGISTRY's publish seam, not by the packer

    My previous comment named the packer as the suspected fix site. That is wrong and is superseded
    here.
    The peer root-caused it (#2216): the publish path is healthy end to end except for two
    arguments that are never passed.

    • A plain RCL publish does emit the assets — the peer ran the workflow's exact sequence:
      publish/wwwroot carries <Name>.styles.css, <Name>.modules.json and _content/, and the
      packed bundle declares 254 staticAssets.
    • The drop is at PluginBundleEndpoints.MapPublish: it calls
      ModulePublish.Validate(plugin, manifest, files, version, packagePath) and ShelveModule(...)
      passing neither the assets to Validate nor accepted.StaticAssets to ShelveModule —
      while every other half was already built for it. Validate takes a staticAssets parameter,
      Accepted carries StaticAssets, ShelveModule takes staticAssets, LandCore materializes
      them, and the consumer's PluginBundleClient.LandFromBundle passes them correctly.

    So: publish healthy, packer healthy, consumer healthy — two missing arguments at the warehouse
    door meant every module ever published landed assemblies-only.

    The mechanism is worth naming because it is this issue's own theme. ShelveModule's parameter is
    optional with a null default:

    public IObservable<ModuleLandingOutcome> ShelveModule(
        string name,
        IReadOnlyList<(string FileName, byte[] Bytes)> assemblies,
        …,
        IReadOnlyList<(string RelativePath, byte[] Bytes)>? staticAssets = null)

    A caller that forgets it does not fail to compile, does not warn, and shelves a bundle that is
    accepted, recorded, downloadable and loads fine. The only visible symptom is CSS that never
    arrives — which is why it survived from the first publish until a pack (EntityViews) finally had
    no image copy to hide behind.

    What changes in §D

    1. The completeness condition is enforceable AT THE SEAM, which is better than the
      consumer-side declaration check I proposed in the previous comment — the producer is where
      "complete" is known. The peer's fix adds a test that publishes through the endpoint and asserts
      the asset exists on disk under the landed generation. Without such a test the failure is
      entirely silent: 200 response, entry recorded, module loads, pages render unstyled.

    2. Migration consequence — every already-published pack must be REPUBLISHED. The fix does not
      retro-fill: the shelf holds asset-less bytes for everything published before it. Therefore:

      Adopt-only / Modules:RequirePrebuilt must treat a bundle shelved before the seam fix as
      NOT covered. Otherwise fail-early adopts an asset-less bundle with full confidence and we get
      a green mesh with unstyled pages — the failure mode this whole issue exists to remove, wearing
      a passing gate.

      A version/timestamp floor on the shelf entry is the cheap discriminator; whatever the mechanism,
      "published before the fix" has to be expressible, or the republish cannot be verified — only
      assumed.

    Evidence for the regression itself stands as measured (re-verified from outside, 2026-08-25):
    /_content/MeshWeaver.Blazor.EntityViews/MeshWeaver.Blazor.EntityViews.styles.css → 404, while
    Views and Graph return 200 only because their image copies still answer.

    Fix lane and the seam test are the peer's (#2216); recorded here as the design condition.

  4. added a commit that references this issue on Aug 25, 2026
  5. rbuergi commented on Aug 25, 2026

    @rbuergi
    ContributorAuthor

    §D's stated action is obsolete — flagging rather than editing the body.

    Verified against origin/main @ 9494b1219: the cross-repo fan-out was redesigned. main-cd.yml:1400-1425 moved it to memex (src/MeshWeaver.GitSync/FrameworkReleaseBroadcaster.cs), and the design deliberately holds no satellite write credential. So §D's instruction to mint a PAT and arm BAKE_SUBSCRIBER_REPOS / DEPENDENT_DISPATCH_TOKEN should be struck — #2235 says the same explicitly.

    Suggest also deleting the vestigial BAKE_SUBSCRIBER_REPOS repo variable, so it cannot be mistaken for the live list.

    ⚠️ And a caveat on §A that affects when it can be enabled: RequirePrebuiltParkTest shows the gate is mesh-wide — it parks RequirePrebuiltTest/NeverBaked, a type in a partition that is not a package. Turning Modules:RequirePrebuilt on today would park every user-authored NodeType on memex/memex-cloud, including everything created through the MCP tools. That contradicts #1707 requirement 4 ("there will always be code without any CI — bake-your-own stays as the universal fallback"), which this mandate scopes to module source and does not overturn. A provenance predicate is needed at NodeTypeCompilationHelpers.cs:283 and in PrebuiltAssemblySeeder.RequiredParkReason before prod enablement.

    Status of the rest, verified: §A code seams shipped but RequirePrebuilt is enabled nowhere (git grep RequirePrebuilt origin/main -- deploy/ returns nothing). §E shipped and enabled fleet-wide (values.aks.yaml:295 Modules__AutoRecycleOnStaleBuild: "true").

  6. rbuergi commented on Aug 25, 2026

    @rbuergi
    ContributorAuthor

    Maintainer decision (2026-08-25): MeshWeaver.Manufacturing is EXCLUDED from the adopt-only estate for now.

    Consequence for this program: Manufacturing does not need a bake/publish lane before Modules:RequirePrebuilt is enabled, and it is not a blocker on that stage. It currently has no bake or publish lane at all (ci.yml: preflight, validate, tag-modules, compile-check, test-repos) and no BAKE_PUBLISH_TARGETS — that stays as-is.

    The satellite adoption order is therefore: Plugins → Education → Reinsurance → SocialMedia, with Manufacturing out of scope.

    🚨 Still outstanding before the flag can flip, and unaffected by this decision: RequirePrebuiltParkTest shows the gate is mesh-wide — it parks a type in a partition that is not a package. Enabling it today would park every user-authored NodeType on memex/memex-cloud, including everything created through the MCP tools, which contradicts #1707 requirement 4 ("bake-your-own stays as the universal fallback"). A provenance predicate at NodeTypeCompilationHelpers.cs:283 and in PrebuiltAssemblySeeder.RequiredParkReason is the remaining prerequisite.

  7. rbuergi commented on Aug 25, 2026

    @rbuergi
    ContributorAuthor

    Maintainer decision (2026-08-25) on §A scoping: RequirePrebuilt applies to everything WE MARK as such — and the marking should cover almost all of it.

    "require prebuilt for all which we mark as such. should be almost all."

    This resolves the blocker raised earlier in this thread. The gate must key on an explicit mark, not on the mesh:

    Why this matters concretely: RequirePrebuiltParkTest currently proves the gate is mesh-wide — it parks RequirePrebuiltTest/NeverBaked, a type in a partition that is not a package. Enabling the flag as it stands today would park every user-authored NodeType on memex and memex-cloud, including everything created through the MCP tools, and render the compilation-error overlay to anonymous visitors. Echoes #683, where nine plugin partitions went to that overlay in production.

    So the remaining prerequisite is now well-defined, and it is the marking + predicate rather than a policy debate:

    1. A durable mark on the type/package (provenance: package-owned vs user-authored) — the thing the operator sets and that "almost all" refers to.
    2. The predicate at NodeTypeCompilationHelpers.cs:283 and in PrebuiltAssemblySeeder.RequiredParkReason reads that mark rather than the mesh-wide flag.
    3. Acceptance must pin the BOUNDARY, not either side alone — one test class asserting both: a marked type with no bundle parks with the named refusal and GetCompileAttemptCount == 0; an unmarked type in the same mesh compiles normally.

    Only after (1)–(3) does Modules:RequirePrebuilt become safe to set in deploy/aks/values.aks.yaml (it is currently enabled nowhere — git grep RequirePrebuilt origin/main -- deploy/ returns nothing).

    Related decision recorded above: MeshWeaver.Manufacturing is excluded from the adopt-only estate for now, so it is not a blocker on this stage.

  8. rbuergi commented on Aug 25, 2026

    @rbuergi
    ContributorAuthor

    Two prerequisites for the marked-types decision are now root-caused, and one is a new follow-up (#2264) that matters specifically for enabling this broadly.

    #2260 — a parked type could be silently un-parked (fix in PR #2262, in CI). This mattered because your directive rests on a refusal that parks the type so no later trigger can storm it — and that containment guarantee was false: the failed-verdict re-drive lifted the park, then re-parked, and anything reading inside that window saw an unparked type.

    The mechanism was one step further than #2260's own hypothesis, established by instrumenting the registry rather than inferring from the double-log:

    ISPARKED  => False        watcher checks — not parked yet
    PARK-ADDED   sources=null adopt-only gate refuses -> parks
    ISPARKED  => False   <--- read lands HERE, inside the window
    UNPARK-REMOVED            the failed-verdict re-drive un-parks (the ONLY removal in the run)
    PARK-ADDED   sources=0    gate refuses again -> re-parks
    

    So there is exactly one un-park, and the fix had to close the window rather than shorten it: the re-drive now asks to be let through (a one-shot admission the parked short-circuit consumes) and never clears the failure. Un-parking stays reserved for the two events that genuinely end one — a human-requested build, and a successful compile.

    #2264 — SettleAsError never stamps FailedBuildInputs, so the adopt-only gate's refusal reads as "never attempted". Every refused type therefore burns one automatic re-drive plus a needless Pending round-trip on the owning hub's action block.

    🚨 Why that matters here specifically: under your decision, "almost all" types become marked. If each refusal costs a wasted re-drive and a Pending round-trip, that cost is multiplied by the number of marked-but-unbaked types at any moment — exactly the condition during a rollout, when the bake lane is catching up. It is not the race (#2260 fixes that); it is the trigger that made the race reachable, and it is a scaling concern for enabling the flag fleet-wide.

    #2264 is deliberately not folded into the #2260 fix: the two SettleAsError call sites need different answers (stamping on the parked short-circuit would mask a genuine input change and defeat #1793's recovery), so it is a design question rather than a one-liner.

  9. added 3 commits that reference this issue on Aug 28, 2026
  10. rbuergi commented on Sep 1, 2026

    @rbuergi
    ContributorAuthor

    Closed by #3015 (merged as 8a31eb40), which converts this program to a committed state-of-record page.

    The PR body said "Closes #1707, #2193, #2550 and #2802" — GitHub only parses the keyword before the first number, so only #1707 auto-closed. Closing the rest by hand; the work is landed either way.

    Where this now lives:

    Both ship with the platform under Doc/Architecture/, so the next session finds them without reading an issue thread.

    🚨 The remaining work is named in those pages, not lost. The sharpest items:

    • Modules:RequirePrebuilt is real, load-bearing, and set in ZERO config files — so "never fall back to compiling" is provided for and enforced nowhere.
    • A Store update does not rebuild its dependents. The mesh-wide closure exists and has two callers; the package installer is not one of them. That is the mechanism behind the 2026-08-25 Store outage, and it is a live bug rather than a program — written up with the fix shape.
    • The awaiting-setup flag is set and read by nothing, so an empty image now boots successfully into a portal that can do nothing and says nothing about why — strictly worse than the exception it replaced.
    • One ordering hazard called out so it is not discovered the hard way: verification compares a bundle's fingerprint against the live mesh source nodes, so removing source from the mesh demotes every adoption from verified to unverified, which the execution gate permits. Doing that before the adopt-only gate is enforced loses verification silently, with nothing red.

    Carried out in the same PR: deleted the dead CompilationLock (no caller, and async/Task.Delay-based, which the house rules forbid); taught the operator gate the direction that rots (it could only fail when a script was deleted, never when one shipped unclaimed); and gave hosting-kv-rotate behaviour tests including the one property whose failure is unrecoverable — it must never print the minted key, mutation-proven.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions