Skip to content

Module build ledger: content-addressed module builds coordinated through memex (Plugins#889, #931) - #3135

Merged
meshweaver-cloud[bot] merged 4 commits into
mainfrom
feat/889-content-addressed-module-builds
Sep 2, 2026
Merged

meshweaver-cloud[bot] merged 4 commits into
mainfrom
feat/889-content-addressed-module-builds

Conversation

@rbuergi

@rbuergi rbuergi commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Closes Systemorph/MeshWeaver.Plugins#889. Producer half of Systemorph/MeshWeaver.Plugins#931 (the consumer half — comparing the recorded platform identity in ModuleUpdateDecision — is a separate PR).

Maintainer directive (2026-09-02): "we should not start the same build multiple times ⇒ coordinate which packages are in progress; track progress through memex. Build and test only when we have to: if Plugin X was built against Platform version Y, we don't have to rebuild this."

What lands

A content address per module build — .github/scripts/module-build-key.py (self-tested, 36 cases: determinism, every input flips the key, noise does not, two refusals). K = sha256(package, manifest.lock moduleVersion, tree hashes of the in-repo compiled+tested closure, reached packages' moduleVersions, repo-level build inputs, tester digest, platform digest, platform-ref, recipe version).

A ledger on the registry portal — .github/scripts/module-build-ledger.py (self-tested against an in-process fake MCP server, 35 cases). One node per key at Admin/ModuleBuilds/<K> (nodeType ModuleBuild, content ModuleBuildRecord — shipped in the framework, src/MeshWeaver.Graph.Contract/ModuleBuildRecord.cs + src/MeshWeaver.Graph/Configuration/ModuleBuildNodeType.cs), written over MCP JSON-RPC at /mcp with a mw_ ApiToken of a CI user holding a partition-admin grant on that root. Claim = create-if-absent (the mutex; "already exists" is the follower's success), heartbeat staleness = the 45-min job cap, reuse = download the recorded run's bundle artifact (sha256-verified) and run only the missing phases, wait = poll 30 s on a live holder, tolerance = compile failures block, test failures block from the 2nd attempt, nothing else blocks. The registry being unavailable never reds a build: the lane builds without coordination and says so in yellow (core #3119).

Lane (node-repo-module-pack.yml): new input ledger: off|required (default off = today's behaviour, printed every run; required asserts the ledger-token secret RED), select keys + consults, build-workspace compiles the ledger's build-modules subset and feeds its postcondition the same list, prepare resolves the PORTAL's framework identity (tester's framework-identity /app under the platform image's dotnet), pack gains a reuse leg and records Built (after the artifact upload) / Tested (after the suite, from the trx) / Published (after the hand-over) / Failed (phase from step outcomes) / released (cancelled) / finished. Bundle artifact retention 3 → 7 days (the reuse window). Every job ≤ 45 min (gate green).

Guard: test/MeshWeaver.Documentation.Test/ModuleBuildLedgerLaneGuard.cs pins the flag + token, the self-tests, the single enumerator, the three transitions in order gated on step outcomes, the two verdict steps, the reuse verification.

Docs: ModuleBuildArchitecture.md "Content-addressed outputs" rewritten from roadmap to as-built (key, ledger, protocol, why Admin/ModuleBuilds not Admin/Build, why run artifacts not the branch-scoped actions cache, what a satellite passes); ModuleVersioning.md push row; What's New Category: Feature.

Deviations from the brief, and why

  • Run artifacts, not actions/cache module-bundle-<K>: the actions cache is branch-scoped, so a bundle cached by a PR run is invisible to the push-to-main run — the one flow that matters most. Artifacts are repo-wide (7-day retention); the caller needs permissions: actions: read for reuse, otherwise select sees the 403 and builds, loudly.
  • Admin/ModuleBuilds, not Build/Modules: the bake's Admin/Build root has an arbiter over its children; the ledger is a different protocol. Admin partition kept so the subject cannot write the decision; the CI user is a partition admin on exactly that root, never a global admin.
  • ledger defaults to off: the token cannot be provisioned from a PR (a mw_ token must be minted on memex.meshweaver.cloud by an admin), and a required-by-default flag would red every satellite and the platform CD (main-cd.yml plugins-modules) at pin move. The summary says OFF on every run so the state cannot be mistaken for a consulted ledger.
  • platformRef is in the key on purpose (dotnet test builds core from source at it). Satellites tracking MW_PLATFORM_REF: main get a new key per core commit; pinning the ref to the promoted set is the lever, and is theirs.
  • Type in C#, not as content: like Build, so every registry portal has it with no content package and no cross-repo half.

Verification

  • module-build-key.py --self-test 36/36, module-build-ledger.py --self-test 35/35, node-repo-scope.py --self-test 44/44
  • check-workflow-timeouts.py --root . 0 violations; check-workflow-shell.py 0 live findings
  • dotnet build src/MeshWeaver.Graph -c Release -warnaserror and test/MeshWeaver.Documentation.Test -c Release -warnaserror: 0 warnings, 0 errors
  • dotnet test test/MeshWeaver.Documentation.Test (whole project): see the CI run

Adoption (per satellite; not in this PR)

  1. Once, on the registry portal (as a global admin): create Admin/ModuleBuilds, grant the CI user the Admin role there with MainNode = "Admin/ModuleBuilds", mint that user a mw_ ApiToken → repo secret REGISTRY_LEDGER_TOKEN.
  2. Move the lane pin to this PR's merge sha; add ledger: required, secrets.ledger-token, and permissions: actions: read on the calling job.

🤖 Generated with Claude Code
Claude-Session: https://claude.ai/code/session_01G4xPnGYEbdu8AVvR5jytyj

rbuergi and others added 4 commits September 2, 2026 18:07
… Plugins#889)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G4xPnGYEbdu8AVvR5jytyj
…lable; lane: select + build-workspace wiring (WIP)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G4xPnGYEbdu8AVvR5jytyj
…se leg, guard, docs, What's New (Plugins#889, #931)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G4xPnGYEbdu8AVvR5jytyj
Copilot AI lite review requested due to automatic review settings September 2, 2026 16:23
@meshweaver-cloud
meshweaver-cloud Bot enabled auto-merge September 2, 2026 16:23
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

340 tests   338 ✅  42s ⏱️
  3 suites    2 💤
  3 files      0 ❌

Results for commit aba6447.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

234 tests   234 ✅  2m 2s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit aba6447.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It introduces a new cross-run CI coordination protocol (ledger + artifact reuse) and significant reusable-workflow logic changes that warrant careful human validation despite no single blocking defect found.

Pull request overview

Introduces a content-addressed “module build ledger” to coordinate reusable module builds across runs (and avoid rebuilding identical module bytes), with the ledger persisted as mesh nodes on the registry portal and integrated into the node-repo-module-pack reusable workflow.

Changes:

  • Adds ModuleBuildRecord (plus supporting types) and registers a new built-in ModuleBuild NodeType rooted at Admin/ModuleBuilds.
  • Extends the reusable module-pack workflow to (optionally) key selections, consult the ledger, reuse prior run artifacts, and record build/test/publish transitions.
  • Adds lane-guard tests and documentation/What’s New updates describing the as-built protocol and adoption.
File summaries
File Description
test/MeshWeaver.Documentation.Test/ModuleBuildLedgerLaneGuard.cs Guard test ensuring the reusable workflow wiring and transition ordering can’t silently drift.
src/MeshWeaver.Graph/Configuration/ModuleBuildNodeType.cs Adds the built-in ModuleBuild NodeType definition and hub type registrations for ledger payloads.
src/MeshWeaver.Graph/Configuration/GraphConfigurationExtensions.cs Registers the new ModuleBuild NodeType in graph setup.
src/MeshWeaver.Graph.Contract/ModuleBuildRecord.cs Defines the ledger record schema (ModuleBuildRecord, status enum, run/artifact/tests records).
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-02-a-module-built-once-is-not-built-again.md What’s New entry announcing the user-visible behavior change.
src/MeshWeaver.Documentation/Data/Architecture/ModuleVersioning.md Updates versioning docs to describe ledger-backed reuse on push runs.
src/MeshWeaver.Documentation/Data/Architecture/ModuleBuildArchitecture.md Rewrites the “content-addressed outputs” section into the as-built ledger/key/protocol contract.
.github/workflows/node-repo-module-pack.yml Adds ledger input + secret, consult/reuse/record logic, and feeds build-modules into the workspace build.
.github/scripts/node-repo-scope.py Updates push-scope rationale to reflect ledger-based reuse of already-published keys.
.github/scripts/module-build-ledger.py Implements ledger coordination + MCP read/write protocol and self-tests.
.github/scripts/module-build-key.py Implements deterministic build key computation over the compiled+tested closure + global inputs + platform/tester identity.
Review details
  • Files reviewed: 11/11 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

    4 files      4 suites   1m 32s ⏱️
1 271 tests 1 271 ✅ 0 💤 0 ❌
1 272 runs  1 272 ✅ 0 💤 0 ❌

Results for commit aba6447.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

437 tests   437 ✅  1m 2s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit aba6447.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

576 tests   384 ✅  2m 36s ⏱️
  3 suites  192 💤
  3 files      0 ❌

Results for commit aba6447.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

2 133 tests   2 133 ✅  2m 20s ⏱️
    3 suites      0 💤
    3 files        0 ❌

Results for commit aba6447.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Test Results

   15 files     15 suites   10m 16s ⏱️
4 991 tests 4 797 ✅ 194 💤 0 ❌
4 992 runs  4 798 ✅ 194 💤 0 ❌

Results for commit aba6447.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants