Skip to content

13 NodeTypes are parked at CompileError on memex-cloud (2 on memex) — configuration lambdas reference types their source query never includes #1786

Closed

Description

@rbuergi

Found while stabilising prod on 2026-08-17. Not a new regression — a standing condition that nothing has been ticketing (see the log-watcher issue filed alongside this).

What is broken

portal NodeTypes parked at CompileError
memex-cloud (memex.meshweaver.cloud) 13
memex (memex.systemorph.com) 2 — BinaryClickerV2/BinaryToggle, Edu/Course
client-i (customer prod) 0 ✅

The 13, with the first error of each:

node error
Northwind/Product CS0246 Supplier, Category not found
Northwind/AnalyticsCatalog CS0103 OrderViews, SalesViews do not exist
FutuRe/{AmericasIns,EuropeRe,AsiaRe}/LineOfBusiness CS0246 LineOfBusiness; CS1061 AddLineOfBusinessLayoutAreas
FutuRe/{AmericasIns,EuropeRe,AsiaRe}/TransactionMapping CS0246 TransactionMapping
Doc/DataMesh/SocialMedia/{Post,Profile} CS0246 SocialMediaPost/SocialMediaProfile, Platform
SocialMedia/{Post,Profile} same
Doc/Architecture/BusinessRules/Cession CS0246 CessionData, Cashflow

All are under the MeshWeaver/samples/Graph/Data/** import.

One shape, three causes — and they need different fixes

Every one of these is "the configuration lambda references a type its source query never included". The compiler says so explicitly for the worst case:

Matched Code nodes (0):
  (none) — the configuration lambda cannot reference types because no source files were
  included. Check that your Source Code nodes exist and that the NodeType's `sources` list
  points at them.

But the reason the query comes up short differs, and lumping them would produce one wrong fix:

  1. Stale orphans. samples/Graph/Data/SocialMedia/** does not exist in the MeshWeaver repo at all — SocialMedia moved to its own repo. The mesh still holds the NodeTypes, so the import adds and updates but never retires what the source tree dropped. These should be deleted, not fixed.
  2. Sources that are not under the node's own Source/. FutuRe/* and Doc/Architecture/BusinessRules/Cession have zero .cs files under them in the repo — their types live elsewhere (shared/sibling), and the default query (namespace:<node>/Source scope:subtree nodeType:Code) cannot reach them. Needs a sources list, or the content moved.
  3. Sources that exist and still do not resolve. Northwind/Product and AnalyticsCatalog do ship Source/*.cs with correct // <meshweaver> headers and no NodeType: override, so they should import as Code — yet Supplier, Category, OrderViews, SalesViews are missing while other types in the same lambda resolve. This is the one to investigate first: a partial source set means the import dropped specific files, which is the same silent-skip family as A UTF-8 BOM makes the node-repo parser skip a file — PensionFund loses 62 of 72 files and gates ZERO NodeTypes, silently #1767.

🚨 Ruled out: it is not the #1767 BOM defect — 0 of 28 .cs under the failing Northwind nodes carry a BOM.

Why it matters, stated accurately

The failures are contained: CompileWatcher parks a failed NodeType and serves the cached error without recompiling, so there is no retry storm (the log says so in as many words). What it costs is real anyway — every page backed by these types is broken for users, and the parked state is invisible unless someone reads pod logs.

Verification

kubectl -n memex-cloud logs <portal-pod> | grep "PARKED after compile failure", on a pod started fresh at 19:26 UTC, so this is the current state and not history.

Activity

  1. rbuergi commented on Aug 17, 2026

    @rbuergi
    ContributorAuthor

    Root cause of "case 3" — and it is not a dropped file

    No file was dropped by the importer. The mesh was never asked.

    The 32 NodeType files under samples/Graph/Data were produced by exporting a running mesh, so each carries that machine's runtime compile record next to the authored definition: compilationStatus, compilationError, Roslyn compilationDiagnostics, compiledSources/currentSourceVersions, assembly coordinates, release-request timestamps.

    Exactly eight carry "compilationStatus": "Error" — and they are exactly the eight non-SocialMedia rows in the table above:

    samples/Graph/Data/Northwind/Product.json
    samples/Graph/Data/Northwind/AnalyticsCatalog.json
    samples/Graph/Data/FutuRe/{AmericasIns,EuropeRe,AsiaRe}/LineOfBusiness.json
    samples/Graph/Data/FutuRe/{AmericasIns,EuropeRe,AsiaRe}/TransactionMapping.json
    

    The compilationError and Matched Code nodes (0) in each is the transcript of a compile that ran on 2026-06-26, committed into the file. Two proofs it cannot describe the sources it ships beside:

    • AnalyticsCatalog's error names OrderViews, SalesViews, SupplierViews, ProductViews. Those identifiers occur nowhere in the configuration in the same file, which calls AddOrderLayoutAreas() / AddSalesLayoutAreas() / AddSupplierLayoutAreas() / AddProductLayoutAreas() — all of which exist under AnalyticsCatalog/Source/. The configuration had already been corrected; the correction could never take effect.
    • Product's error points at lines 200 and 204 of a source set that is 181 lines long.

    Why the state is unrecoverable, not merely stale

    Every automatic path that could re-drive a compile is closed to such a node:

    path requires a baked-Error node
    first-build kickoff CompilationStatus is null ✗ it is Error — reads as "already attempted here"
    recovery kickoff CompilationStatus == Compiling ✗
    framework-stale kickoff LatestAssemblyCollection + LatestAssemblyPath + CompiledFrameworkVersion ✗ ApplyCompileFailure stamps none of them
    release-request watcher RequestedReleaseAt > LastReleaseRequestHandledAt ✗ the export baked the two equal

    The 24 files that baked Ok escaped only by accident: they also carry assembly coordinates plus a compiledFrameworkVersion MVID, which never matches another deployment, so the framework-stale kickoff rebuilds them. The Ok bake is the same defect; it is merely self-healing.

    PR #1792 strips the runtime-owned members from all 32 files (byte-exact elsewhere) and adds ShippedNodeTypeStateTest, whose banned set is derived from NodeTypeDefinition by reflection over the control plane's naming convention so a member added later is banned the day it is added. Verified on memex-cloud that omitting the state is the supported shape: Doc/Architecture/BusinessRules/Cession ships no compile state in its file and its live node holds a full, current compile record.


    The measurements correct three rows of the table

    Read live off memex-cloud (get on each node, so this is the node's own record, not a log line):

    node measured now
    Doc/DataMesh/SocialMedia/Post Ok, v804, compiled 19:24:44, 3 sources present
    Doc/DataMesh/SocialMedia/Profile Ok, v664, compiled 19:24:44, 2 sources present
    SocialMedia/Post Ok, v1177, compiled 15:43:50, 9 sources present
    SocialMedia/Profile Ok, v471, 10 sources present (uses sources: ["namespace:Source scope:subtree", "shared=@SocialMedia/Post/Source"])
    Doc/Architecture/BusinessRules/Cession Ok, v671, lsp_diagnostics_for_node returns ok:true — warnings only

    So "case 1 — stale orphans that want deleting" does not hold. SocialMedia/{Post,Profile} are the live plugin's types with their sources in the mesh, and Doc/DataMesh/SocialMedia/{Post,Profile} compile from Doc/DataMesh/SocialMedia/*/Source/*, which exist. Cession is likewise fine — its four Source/* Code nodes are present (CessionData declares Cashflow, ExcessOfLossLayer, CededCashflow). Deleting any of these would have destroyed working content.

    Those five rows were true of some pod, not of the mesh at the time they were written down — which is the same shape as the bug itself: a compile verdict outliving the thing it described.

    Northwind/{Product,AnalyticsCatalog} are not on memex-cloud at all — Memex.Portal.Distributed syncs only Doc, Agent, Model, Harness, Skill, and search nodeType:NodeType namespace:MeshWeaver scope:descendants returns 19 types under ACME / FutuRe / Systemorph and no Northwind. Their baked Error is still a real defect for every deployment that does load them (the dev Monolith, the test fixtures) — fixed by #1792.


    What is genuinely broken, and why I stopped short of fixing it

    MeshWeaver/samples/Graph/Data/FutuRe/{AmericasIns,EuropeRe,AsiaRe}/{LineOfBusiness,TransactionMapping} — six nodes, and this one is live, not stale:

    • compilationStatus: Error, currentSourceVersions: {} — the live sources watcher measured zero sources.
    • lastCompileStartedAt 2026-08-17T19:40:20, one second after requestedReleaseAt 19:40:19. A real compile ran today and really failed.
    • search namespace:.../AmericasIns/LineOfBusiness/Source scope:subtree → 0; the group-level .../FutuRe/LineOfBusiness/Source → 2 Code nodes.

    Each declares WithContentType<LineOfBusiness>() (+ .AddLineOfBusinessLayoutAreas()) while the types live in the group-level node's Source/.

    I did not "just delete them", and the reason matters. They are not dead weight — they are the per-business-unit browse pages, and that is tested: FutuReAnalysisTest.EuropeRe_LineOfBusiness_Search_ShouldReturn8LoBs asserts FutuRe/EuropeRe/LineOfBusiness renders the NodeType search shell with HiddenQuery scoped to namespace:FutuRe/EuropeRe/LineOfBusiness, and explicitly asserts it is not the instance fallback (path:). Deleting the node removes a tested behaviour.

    Nor is the obvious repair safe: pointing their sources at the group-level Source would compile the same LineOfBusiness record into four separate collectible assemblies, giving four CLR types with the same $type discriminator — the silent-empty-render trap.

    So the fix is a design call I don't want to guess at:

    1. drop WithContentType<…> from the regional nodes and let them be pure browse pages over instances of the group-level type (needs confirming that the Search shell doesn't require a compiled content type — it renders correctly today while the node is parked, which suggests it doesn't); or
    2. keep one type and have the regional pages be instances/views of it, as every other container in the FutuRe tree already is (FutuRe/AsiaRe.json is nodeType: FutuRe/BusinessUnit, FutuRe/AmericasIns/Analysis.json is nodeType: FutuRe/LocalAnalysis — the six are the only nodes in that tree that declare a type instead of being one); or
    3. a first-class "same types, one assembly, many browse scopes" mechanism, if that is what the shape actually wants.

    Which of those do you want?


    Platform gap worth its own issue

    A NodeType that has never compiled successfully and settles at Error is unreachable by every automatic re-drive — so a fix to the cause can never reach the node. That is not only how the baked state persists; it is why the AnchorIncludePath fix (which was written for "memex-cloud 2026-08-12 parked 15 NodeTypes (FutuRe, Cession, SocialMedia, Northwind/AnalyticsCatalog)" — near enough the same list) left them parked after it shipped.

    The framework-stale kickoff's own Where already reads CompilationStatus is Ok or Error, so covering Error is clearly intended; it cannot, because HasStaleFrameworkBuild requires assembly coordinates that ApplyCompileFailure never writes. Stamping CompiledFrameworkVersion on failure (or an equivalent "which framework produced this verdict" marker) would let a framework bump retry a failed type exactly once — which is already the treatment a stale successful build gets.

    I have not written that: it changes compile-control-plane behaviour and wants its own review.

  2. added a commit that references this issue on Aug 17, 2026
  3. rbuergi commented on Aug 21, 2026

    @rbuergi
    ContributorAuthor

    Re-measured on prod today — 13 → 6, and PR #2030 takes the rest

    Verified against memex.meshweaver.cloud (serving 3.0.0-rc6.ci.4780) with GetDiagnostics per type, 2026-08-21. Seven of the thirteen are gone; six are still Error.

    the 13 today which cause (from this issue)
    FutuRe/{AmericasIns,EuropeRe,AsiaRe}/LineOfBusiness Error ❌ 2 — sources not under the node's own Source/
    FutuRe/{AmericasIns,EuropeRe,AsiaRe}/TransactionMapping Error ❌ 2
    Doc/DataMesh/SocialMedia/{Post,Profile} Ok ✅ (compiled 20:11:24Z) 1
    SocialMedia/{Post,Profile} Ok ✅ (20:19:40Z / 20:17:26Z) 1
    Doc/Architecture/BusinessRules/Cession Ok ✅ (20:07:50Z) 2
    Northwind/{Product,AnalyticsCatalog} node no longer exists ✅ (namespace:Northwind scope:subtree → 0) 1

    Cause 3 was the interesting one, and it is answered

    This issue flagged Northwind/Product / AnalyticsCatalog as "sources exist and still do not resolve — investigate first". They are gone now, and ShippedNodeTypeStateTest records what they actually were: the tree shipped 32 NodeType files exported from a running mesh, carrying that machine's compile record. Eight had "compilationStatus": "Error" baked in from 2026-06-26, and every deployment adopted the verdict verbatim — unreachable by every automatic re-drive path. That is why AnalyticsCatalog's committed error named OrderViews/SalesViews, identifiers that occur nowhere in the configuration it shipped alongside: the configuration had been corrected and the correction could never take effect.

    So cause 3 was not a silent-skip sibling of #1767 at all — it was a stale imported verdict, and stripping it is what let the other seven compile for real.

    What is left, and why it survived the cleanup

    The six FutuRe types now fail for a real reason — the live compiler's own report, not a baked-in one:

    CS0246 The type or namespace name 'LineOfBusiness' could not be found
    --- Source discovery ---
      - namespace:…/AmericasIns/LineOfBusiness/Source scope:subtree nodeType:Code
    Matched Code nodes (0):
      (none) — the configuration lambda cannot reference types because no source files were included.
    

    They are copy-pastes of their parents: identical WithContentType<LineOfBusiness>() lambda, none of the parent's Source/*.cs. The parent FutuRe/LineOfBusiness compiles Ok on the same mesh because it owns its Source/.

    And nothing is typed by any of the six. Every instance beneath them names the parent — 8+8 LoB instances as FutuRe/LineOfBusiness, 14+13 mappings as FutuRe/TransactionMapping, AsiaRe has none at all — and a repo-wide scan for "nodeType": "<any of the six>" returns 0 files. They were dead weight that could only ever park.

    PR #2030

    Retires the six as Markdown nodes (keeping the governance prose and their children), removes three source-less duplicates under samples/Graph/Data/Doc/ that can only clobber the documentation tree's good copies, and adds ShippedNodeTypeSourceResolutionTest — a shipped NodeType must not name a content type that neither its own Source/ nor the framework declares. Proven non-vacuous: reverting only the data half fails the compiled test naming all nine.

    One thing this issue is right about that the guard cannot fix: the mesh keeps what the source tree drops. Merging #2030 will not clear the six parked nodes on memex-cloud — they will still be there after the sync until they are retyped or removed on the instance. That is a live-mesh write, so I have not made it; flagging it for whoever deploys.

  4. added 2 commits that reference this issue on Aug 24, 2026
  5. rbuergi commented on Aug 25, 2026

    @rbuergi
    ContributorAuthor

    ~85% done — narrowing scope. Verified against origin/main @ 9494b1219:

    Fixed: b63fbf057 + 091a57a6c retired the six FutuRe types and the three Doc/ duplicates; Cession now ships Source/*.cs; .github/doc-gate.allow is empty.

    Remaining, and all that this issue should now cover:

    1. Northwind/AnalyticsCatalog — its Source/*.cs do exist (Supplier.cs, Category.cs under AnalyticsCatalog/Source/), so this is a query/config problem, not a missing-source one.
    2. Northwind/Product — has no Source/ folder at all and no sources list, so the default namespace:Northwind/Product/Source query resolves nothing.
    3. Six remaining FutuRe/* entries in .github/samples-gate.allow.
    4. Mesh-side residue not visible in the repo: stale SocialMedia/Post, SocialMedia/Profile and Doc/DataMesh/SocialMedia/* orphans are gone from the repo but still in the mesh DB — they need deleting there, proven by a get returning nothing.

    Done-condition: .github/samples-gate.allow is empty (like doc-gate.allow), and an MCP sweep where every nodeType:NodeType reads Compiled. Per AGENTS.md, ok:false with Absent/NotCompilable/Unavailable is a sweep failure, not a pass.

  6. rbuergi commented on Aug 26, 2026

    @rbuergi
    ContributorAuthor

    These are not just CI debt — 7 of the 8 compile entries in .github/samples-gate.allow are exactly the NodeTypes this issue reports parked at CompileError in prod. Nobody had connected the two, and the framing differs sharply depending on which you read.

    Current allow file on main:

    FutuRe/AmericasIns/LineOfBusiness compile      ← #1786
    FutuRe/AmericasIns/TransactionMapping compile  ← #1786
    FutuRe/AsiaRe/LineOfBusiness compile           ← #1786
    FutuRe/AsiaRe/TransactionMapping compile       ← #1786
    FutuRe/EuropeRe/LineOfBusiness compile         ← #1786
    FutuRe/EuropeRe/TransactionMapping compile     ← #1786
    Northwind/Product compile                      ← #1786
    PensionFund/BalanceSheet compile               ← separate (a legacy `#r` that cannot resolve on a gate host)
    PensionFund idempotence intermittent           ← separate (#2324's new token)
    

    Read from the allow file these look like accepted samples-gate debt. Read from here they are a live production condition on memex-cloud — and per AGENTS.md a NodeType left at CompileError refuses portal readiness and parks every instance hub for the full 60 s activation budget. That is a materially different cost than "the gate is noisy".

    Two of the thirteen were fixed today, which shows the rest are tractable

    #2308 (merged 2026-08-26 06:45) fixed Northwind/AnalyticsCatalog — DashboardLayoutAreas.cs named four classes that do not exist (SalesViews.SalesByCategory, SupplierViews.SupplierSummary, …) where the real declarations live in SalesLayoutAreas / SupplierLayoutAreas. It removed that entry from the allow file.

    It also documents the cascade this issue's table hides: Northwind/Product was only red because its ROOT was. Supplier and Category are declared in AnalyticsCatalog's own Source/, so a red root makes every dependent red. That matches AGENTS.md's sweep guidance — fix roots first, a red upstream makes dependents UpstreamFailed — and means the 13 are probably fewer than 13 distinct defects.

    Prediction worth checking rather than assuming: if the root fix landed, Northwind/Product should now compile, making its allow entry genuinely stale — which #2324's ratchet will now report (and, since it is not marked intermittent, fail on). I did not see a stale-entry flag on the last five main runs, so either the gate has not re-evaluated it yet or it still fails for an independent reason. Worth confirming before removing the line.

    What I could NOT verify

    I tried to check current prod state directly and could not: lsp_diagnostics_for_node Northwind/Product returned Absent — which per AGENTS.md is a sweep FAILURE, not a pass ("no node resolved at this path, so nothing was checked"). That replica does not hold the path, so this issue's per-portal counts are as-of 2026-08-17 and unconfirmed today. Re-running the documented sweep (Search('nodeType:NodeType') → LspDiagnosticsForNode per type, on the memex-cloud replica specifically) is the first step for anyone picking this up — and the Absent/NotCompilable/Unavailable statuses must be treated as unchecked, not clean.

    Suggested order

    Roots first, then dependents, then delete each allow line as its subject goes green — the ratchet will now enforce that last part rather than letting the line linger.

  7. added 2 commits that reference this issue on Aug 26, 2026
  8. rbuergi commented on Aug 27, 2026

    @rbuergi
    ContributorAuthor

    Re-swept both portals today (2026-08-27) — 12 of the 13 are gone; 1 remains, and it is a NEW node, not one of the 13

    Measured two independent ways, because AGENTS.md says neither diagnostics surface is trustworthy alone:

    1. The mesh's own record — search 'nodeType:NodeType -compilationStatus:Ok' on memex.meshweaver.cloud (a complete result set, truncated:false), then get_diagnostics per hit.
    2. The portal's own log — kubectl -n <ns> logs <portal-pod> -c memex-portal | grep 'PARKED after compile failure', the verification this issue was opened with.

    Both answer the same thing.

    portal image NodeTypes at CompileError
    memex-cloud 3.0.0-rc8.ci.6011 1 — LinkedIn/Skill
    memex serving ci.5670 (see the incident at the bottom) 0

    The 13, one by one

    # as named in #1786 today cause closed by
    1 Northwind/Product absent from memex-cloud (namespace:Northwind scope:subtree nodeType:NodeType → 0); in-repo it is green in the samples gate (ok Northwind/Product [compile:Passed render:Passed tests:Skipped], main run 33044271801) cascade — red only because its ROOT AnalyticsCatalog was red; plus a baked foreign compile verdict #2308 (root), #1792 (baked state)
    2 Northwind/AnalyticsCatalog same — absent from memex-cloud, green in the gate DashboardLayoutAreas.cs named SalesViews.…/SupplierViews.… classes that do not exist #2308
    3–5 FutuRe/{AmericasIns,EuropeRe,AsiaRe}/LineOfBusiness absent (namespace:FutuRe scope:subtree → 0 on memex-cloud); retired to Markdown in-repo copy-pastes of the parent's WithContentType<LineOfBusiness>() lambda carrying none of the parent's Source/*.cs; nothing was typed by any of them #2030
    6–8 FutuRe/{AmericasIns,EuropeRe,AsiaRe}/TransactionMapping same same #2030
    9–10 Doc/DataMesh/SocialMedia/{Post,Profile} Ok — compiled 2026-08-27T03:34:45Z a foreign machine's "compilationStatus":"Error" baked into the shipped file, unreachable by every automatic re-drive #1792
    11–12 SocialMedia/{Post,Profile} Ok — 03:38:16Z / 05:56:51Z same #1792
    13 Doc/Architecture/BusinessRules/Cession Ok — 03:34:44Z baked Error + no Source/; it now ships Source/*.cs #1792 + the Cession sources

    The two on memex (BinaryClickerV2/BinaryToggle, Edu/Course) are gone too — zero PARKED after compile failure lines on either serving pod.

    So the 13 were not 13 defects. Three causes accounted for all of them, and the largest — a shipped node file carrying a foreign machine's compile verdict — was one edit to 32 files.


    The one that is left: LinkedIn/Skill — the SAME cause-1 shape, a week old

    CS0246 The type or namespace name 'Skill' could not be found
    --- Source discovery ---
      - namespace:LinkedIn/Skill/Source scope:subtree nodeType:Code
    Matched Code nodes (0):
      (none) — the configuration lambda cannot reference types because no source files were included.
    

    It was retired on purpose and the mesh kept it. MeshWeaver.SocialMedia@75f3f3d (2026-08-20, "LinkedIn stops redefining Skill") deleted LinkedIn/Skill.json and LinkedIn/Skill/Source/Skill.cs, because the platform owns the name Skill and this type was dead — the profile Overview's nodeType:LinkedIn/Skill query had never returned a row. The real type is LinkedIn/LinkedInSkill, which has its 5 source nodes and compiles green, and the Overview was moved onto it in that same commit.

    Why it only broke now — a framework-identity flip, exactly the mechanism this issue's brief predicted. The node's own record:

    lastCompileSucceededAt   2026-08-21T13:38:24Z    (assembly LinkedIn_Skill/v848-sbf2f93e-…dll)
    compiledFrameworkVersion sbf2f93e099b84829478731f0c64835b9
    lastCompileStartedAt     2026-08-27T06:16:17Z
    failedBuildInputs        fw=s990240a13187861c03d0a2db1cb67775; …; src=0
    currentSourceVersions    {}
    

    It served its already-built assembly happily until the framework identity moved sbf2f93e… → s990240a1…. The stale-framework kickoff then re-drove the compile, found src=0, and parked it. A bump recompiles every dynamic NodeType at once, which is why this surfaced as a single event rather than on the day the sources went away.

    It is safe to delete, and that is the fix — not a content edit:

    • search 'nodeType:LinkedIn/Skill' → 0 instances;
    • the source of truth dropped it deliberately, with its one reader migrated in the same commit;
    • across all node repos only comments/AGENTS.md prose mention it (as the cautionary example).

    🚨 I could not land the delete. MCP refuses it for my identity and says not to retry:

    Refused deleting LinkedIn/Skill: Access denied: user 'rbuergi' lacks Delete permission on 'LinkedIn/Skill/_Activity/compile-20260827061616…'.

    Someone with Delete on the LinkedIn partition needs to run it in the GUI: https://memex.meshweaver.cloud/LinkedIn/Skill/Delete — that is the last step to a clean sweep. (NodeTypeUnparkPostDeletionHandler clears the park on delete, so nothing else is needed.)


    The platform gap that produced it — and will produce the next one

    Only the DELTA install prunes. PackageInstaller.InstallNodeRepoDelta deletes removedNodePaths derived from a manifest diff; the FULL install has no prune at all, deliberately (PackageInstaller.cs:25 — full-replace semantics would wipe the rest of a shared partition). And the delta is unavailable whenever there is no install record with InstalledFiles, or when the diff touches the package's shared Source/-Test/ (CatalogLayoutAreas.cs:598 throws "full install required"). Even on the delta path, a prune delete that fails is .Catch-ed to a LogWarning and counted as 0, so the {Pruned} pruned line under-reports silently.

    Net: the mesh keeps what the source tree drops, and a retired NodeType is inert until the next framework bump turns it into a parked type that refuses portal readiness. That is the same sentence the 2026-08-21 comment on this issue ended with; it has now cost a second incident. Worth its own issue with #723 (platform-identity rebuilds) — the two compound: an identity flip is exactly what converts a harmless orphan into a park.

    Also unchanged and still worth doing: a NodeType that has never compiled and settles at Error is unreachable by every automatic re-drive, because ApplyCompileFailure writes no CompiledFrameworkVersion and HasStaleFrameworkBuild requires one. LinkedIn/Skill escaped that only because it had previously succeeded.


    Repo-side residue closed: PR #2472

    .github/samples-gate.allow still carried six compile entries for the FutuRe types that #2030 retired to Markdown — dead lines asserting, to anyone reading the file, that six NodeTypes are still failing to compile in production.

    The gate's ratchet could not retire them: stale-detection asks did this subject pass?, and a subject that no longer exists never passes and never fails. It printed unverifiable allow entry (check skipped or scope absent this run) six times on every main run since #2030 and stayed green. That default is right for the shared tester (satellites stage a subset of their tree), so #2472 leaves it alone and adds a repo-local guard instead — this repo stages its whole content tree unconditionally, so here "absent" means retired and is decidable statically. Proven non-vacuous: with the six lines restored it fails naming all six and what each became.

    After #2472, this issue's share of that file is zero. The two PensionFund lines are separately diagnosed debt that #1786 does not own.


    🚨 Unrelated live incident found while sweeping — memex cannot roll to ci.6011

    Not #1786, but it is in front of the same portals and nobody has filed it:

    memex (memex.systemorph.com) has been stuck mid-rollout for ~3 h — memex-portal-deployment-54bcd4f496 is CrashLoopBackOff at 38 restarts, exit 139 / signo 6 (SIGABRT, not SIGSEGV — a managed unhandled exception). The cause is at host start:

    Autofac.Core.DependencyResolutionException: … activating IHostedService[] -> MeshWeaver.AI.OpenAI.OpenAICompatibleModelSync
    None of the constructors found on type 'OpenAICompatibleModelSync' can be invoked …
    Cannot resolve parameter 'MeshWeaver.AI.ProviderModelLister lister'
    

    Both namespaces target the same image memex-portal-ai:3.0.0-rc8.ci.6011 and memex-cloud runs it fine, so the break is namespace-local: memex's installed OpenAI module does not register ProviderModelLister. That is the subject of MeshWeaver.Plugins #730 (fix/openai-module-carries-providermodellister, merged at plugins main tip). Two ci.5670 pods still serve, so the site answers 200 — but the roll cannot complete and the fix cannot land. This is also why the memex.systemorph.com MCP endpoint errors on every call for me right now.

  9. added a commit that references this issue on Aug 27, 2026
  10. rbuergi commented on Aug 28, 2026

    @rbuergi
    ContributorAuthor

    Re-measured against the live memex mesh today (2026-08-28). The memex half of this issue is almost cleared, but not quite, and the remaining one is a different defect from what this issue describes.

    Then: "13 NodeTypes are parked at CompileError on memex-cloud (2 on memex)".
    Now, on memex: 1.

    nodeType:NodeType -compilationStatus:Ok   →  10 results
       9 × built-in static types (ApiCredential, Release, Build, User, VUser,
           Partition, OAuthCode, Space, WebhookEvent) — no compilationStatus at
           all, because they are not dynamically compiled. Expected, not failures.
       1 × LinkedIn/Skill  — compilationStatus: "Error"
    

    🚨 A warning for anyone re-running this check. I first queried compilationStatus:CompileError and got 0, which reads as "all clear". It is a false negative: the stored enum value is Error, not CompileError. The filter itself works (compilationStatus:Ok returns matches), so the zero looked entirely credible. Use the negation -compilationStatus:Ok and classify the results, rather than guessing the failure term.

    The one that remains is not a configuration-lambda break

    This issue's premise is "configuration lambdas referencing deleted framework symbols". LinkedIn/Skill is a different failure: its source is gone.

    CS0246 (line 58): The type or namespace name 'Skill' could not be found
    --- Source discovery ---
      namespace:LinkedIn/Skill/Source scope:subtree nodeType:Code
      namespace:LinkedIn/Skill/Test   scope:subtree nodeType:Code
    Matched Code nodes (0):
      (none) — the configuration lambda cannot reference types because no source
      files were included.
    failedBuildInputs: fw=s990240a13187861c03d0a2db1cb67775;mod=…;src=0
    

    Its configuration is config => config.WithContentType<Skill>().AddDefaultLayoutAreas(), and src=0 says plainly that zero source files were found. It last compiled successfully at v848 on 2026-08-21 13:38; it is now at v877. So it was healthy and lost its Source subtree afterwards.

    And nothing in any repo defines it. git ls-files | grep -i 'LinkedIn/Skill' returns 0 in MeshWeaver, MeshWeaver.Plugins and MeshWeaver.SocialMedia. It is an orphan: a live NodeType whose definition no longer exists anywhere, so no sync or redeploy can ever repair it, and per AGENTS.md a NodeType parked in error refuses portal readiness and parks instance hubs for the full 60 s activation budget.

    I have not touched it — deleting a live NodeType on your portal is your call, not mine. The two options are to delete the orphan, or to restore its Source if LinkedIn/Skill is still wanted. Given no repo carries it, deletion looks right, but say the word.

    What I could not measure: memex-cloud. I have no MCP connection to that mesh, so the "13 on memex-cloud" figure is unverified and this issue should stay open for it.

  11. rbuergi commented on Aug 28, 2026

    @rbuergi
    ContributorAuthor

    Correction to my measurement above: I labelled the wrong instance.

    I wrote "Re-measured against the live memex mesh" and reported "on memex: 1". The mesh my MCP session is attached to is memex-cloud, not memex.

    Evidence, from data I read in that same session: the webhook events under Hosting/PlatformBuilds/_Inbox — nodes stored in the mesh I queried — carry

    "Host": "memex.meshweaver.cloud",  "X-Original-Host": "memex.meshweaver.cloud"
    

    and the cluster's ingress map assigns memex.meshweaver.cloud → namespace memex-cloud, while memex.systemorph.com → namespace memex. Confirmed independently just now: a node I located from the memex namespace's own logs (Admin/_LogIncident/75d8e54a0f563fa1) returns Not found through this MCP session, which it would not if the two were the same mesh.

    So the numbers should read:

    this issue said measured 2026-08-28
    memex-cloud 13 at CompileError 1 (LinkedIn/Skill)
    memex 2 at CompileError not measured

    That is a better result than I reported, not a worse one — 13 → 1 on the instance this issue leads with. But the memex half is now the unmeasured one, the exact opposite of what I claimed, and the LinkedIn/Skill orphan (no repo defines it; src=0) is a memex-cloud node.

    Everything else in that comment stands unchanged: the count, the orphan diagnosis, and in particular the query warning — compilationStatus:CompileError returns a false 0 because the stored enum value is Error; use -compilationStatus:Ok and classify.

  12. rbuergi commented on Aug 28, 2026

    @rbuergi
    ContributorAuthor

    Re-measured on memex-cloud today (2026-08-28): 13 → 1. Twelve of the thirteen are gone; one remains, and it is an instance of a different issue.

    Method, because the obvious query lies

    compilationStatus:CompileError returns a false 0 — the stored enum value is Error, not CompileError. And the negation -compilationStatus:Ok over-reports: it returned 13, but that set includes types with no compilation status at all (User, VUser, Partition, Space, OAuthCode, WebhookEvent, ApiCredential, Release, Build — compiled-in types with no source) and types merely Pending. Publish/Slide was in that 13 and had lastCompileSucceededAt: 2026-08-28T09:45:39 — it compiled successfully an hour before I looked.

    The query that answers the question is nodeType:NodeType compilationStatus:Error, and it returns exactly one row.

    The one that is still red: LinkedIn/Skill

    CS0246 Error (line 58): The type or namespace name 'Skill' could not be found
    --- Source discovery ---
    Executed source queries (2):
      - namespace:LinkedIn/Skill/Source scope:subtree nodeType:Code
      - namespace:LinkedIn/Skill/Test  scope:subtree nodeType:Code
    Matched Code nodes (0):
      (none) — the configuration lambda cannot reference types because no source
      files were included.
    

    failedBuildInputs carries src=0, and I confirmed it directly: namespace:LinkedIn/Skill/Source scope:subtree returns 0 nodes. The source tree is gone.

    What survives is the NodeType itself (version 881, last compiled OK at version 848 on 2026-08-21) and its entire Release history — the descendants listing is wall-to-wall Release nodes.

    This is #2473, not a compile bug

    The configuration lambda is fine; there is simply nothing left for it to reference. A node the source repo retired survived in the mesh, and the NodeType it fed is now parked at Error — which is precisely the failure #2473 describes ("a retired NodeType parks the portal on the next framework flip"). Per AGENTS.md a NodeType at CompileError refuses portal readiness and parks every instance hub for the full 60 s activation budget, so this is not cosmetic.

    Fixing #2473's retirement path should either remove this NodeType with its source or stop the source from being retired out from under it. Suggest narrowing this issue to LinkedIn/Skill and letting #2473 carry the mechanism — or closing it in favour of #2473 once that lands, since re-fixing the same node by hand would leave the cause live.

    Measured on memex-cloud only (that is the mesh this MCP connects to). The "2 on memex" half of the original report is unverified here.

  13. rbuergi commented on Aug 28, 2026

    @rbuergi
    ContributorAuthor

    Closing per the re-measurement above and the thread's own recommendation.

    The comment above already proposed exactly this: "closing it in favour of #2473 once that lands, since re-fixing the same node by hand would leave the cause live." #2473's fix has landed, so closing.

    Unverified residue, for the record: the original report's "2 on memex" half was never re-measured (the MCP session connects to memex-cloud). If a sweep on memex still shows compilationStatus:Error rows that are not the retirement class, that is a fresh finding — file it against the compile pipeline, not here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions