Skip to content

cd: satellite-compat — every core build compile-checks every satellite's NodeTypes against the new set (phase 2a) - #4088

Merged
rbuergi merged 5 commits into
mainfrom
cd/satellite-compat-gate
Sep 12, 2026
Merged

rbuergi merged 5 commits into
mainfrom
cd/satellite-compat-gate

Conversation

@rbuergi

@rbuergi rbuergi commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Phase 2(a) of the fleet CI refactor — the per-build counterweight to the daily rebuild

Decision (maintainer, 2026-09-12): the platform is assumed backwards compatible within a MAJOR; the satellites (MeshWeaver.SocialMedia, .Crm, .Reinsurance, .Education, .Manufacturing — Plugins is built inside CD already) rebuild once a day, not per platform build. That removed the only per-build evidence that a core merge did not break them. This PR is the counterweight: every core build still MEASURES compatibility, cheaply, so a break is known within the hour — not at 03:00 by the daily run, not on a portal at self-update.

Phase 2(b), the instance link probe (#4083), is another session's; ModulePlatformLink.cs / Mesh.Contract are untouched.

Design

A new matrix job in main-cd.yml, satellite-compat, runs after promote: one leg per satellite, in parallel (fail-fast: false), each a uses: of the existing reusable node-repo-compile-check.yml — the very gate the satellites run on their own PRs — pointed at the satellite's main, at this run's set, and — the control — at the set the fleet is currently on (maintainer pushback via the reviewing session: "why fail compatibility?" — a single red compile cannot tell which side moved).

The baseline control. Every leg compiles the same content twice and compat-verdict.py (new, self-tested on every row) classifies per NodeType:

previous image new image verdict leg colour
compiled fails BROKE — this core build broke it 🔴 RED — the only red
failed fails already broken — satellite side; its own daily run / ci-main-red issue own it 🟢 green, advisory line naming satellite + first failing types
absent from the previous report (type newer than the set the fleet is on) fails unjudged (no per-type baseline) — its own row: only previous-compiled-ok → new-fails licenses "broke", and absence licenses neither "broke" nor "already broken"; never folded into either 🟢 green, advisory
compiled compiled compatible 🟢
unresolvable (any) refusal to judge — "could not establish a baseline: "; new-image failures listed unjudged, never attributed 🟢 green, advisory

Where the baseline comes from — one derivation, the one the run already has. gate (pre-promote, already Azure-logged-in for its completeness probe) reads the mw-plugin-test manifest tagged main from ACR (az acr manifest list-metadata) and records its digest and its 3.0.0-ci.<run> tag (baseline_tester_digest / _version / _run_number) — the set the fleet is on, read before this run's promote moves the pointer. satellite-compat-image then (1) asserts GHCR holds that digest, (2) locates the run that built it by run_number (= the version suffix; verified live: 3.0.0-ci.8414 ↔ run 34688449431 run_number=8414), (3) downloads that run's four module-bundle-* artifacts — the modules that shipped with the baseline — and re-uploads them as satellite-compat-baseline-bundles for the legs. Any step of that failing → no baseline, with the reason (::warning::, never fatal, never a skip painted grey: the leg still runs and reports the refusal). Both digests are recorded in every verdict JSON (images.new / images.baseline / images.baseline_reason).

Why the baseline compiles against the baseline run's bundles and not this run's (stated in the workflow comment and the contract paragraph so a later "we already have this run's bundles, why download the old ones" optimisation is told what it would break): the four modules are Plugins content packed per build. Compiling the previous image with this run's bundles would make a Plugins-side move between the two sets (a type renamed in MeshWeaver.AI, say) fail the baseline reading too and be filed as "already broken" — attributing a Plugins move to the satellite — or fail it spuriously where the new bundle needs new core surface, hiding a core break. The whole control is attribution; that optimisation would remove it.

Every classification is per NodeType (the type's own name in the baseline report's ok list), never inferred from the satellite's overall colour — compat-verdict.py --self-test holds that with an all-red-baseline case that still yields BROKE for the one type that compiled there.

Inputs per leg (all five identical except content-repository):

Input Value Where it comes from
content-repository Systemorph/<satellite> (matrix, literal per entry) so the guards that inventory cross-repo reaches SEE each one
content-ref main the satellite's default branch — what its daily run would bake; the leg resolves it ONCE and records the sha (content.sha), shown as <satellite> (main @ <sha>)
content-app-id / -private-key secrets.FLEET_READER_APP_ID / _PRIVATE_KEY the READ-ONLY fleet App; permission-contents: read, repositories: = that one satellite (publish-bake's exact shape). Measured 2026-09-12: meshweaver-reader (installation 161095771) is installed org-wide, repository_selection: all — Crm included
test-image / image-digest ghcr.io/systemorph/mw-plugin-test:<version> @ this run's tester digest satellite-compat-image resolves the GHCR mirror promote phase B wrote and asserts its digest equals plugins-bake-image.outputs.digest (ACR)
acr-username / acr-password github.actor / GITHUB_TOKEN GHCR pull. Core holds no ACR password (it pushes by OIDC), and OIDC inside the lane would demand id-token: write of every satellite caller — the #3933 startup-failure shape
module-artifacts module-bundle-MeshWeaver.{AI,Markdown.Collaboration,Maps,Payments.Stripe} this run's plugins-modules artifacts — the same pattern and bytes plugins-bake seals; every satellite declares exactly these four (registry-modules: AI Essentials Stripe Maps, upstream-seed: plugins)
platform-ref needs.gate.outputs.sha compile-check.py and compat-verdict.py at this run's core sha
verdict-artifact satellite-compat-<satellite> the verdict as JSON, uploaded on red as well as green
judge-against-baseline true the classified verdict decides the leg's colour (a node repo's own PR gate leaves it false: plain red stays red)
baseline-image-digest / baseline-unavailable-reason satellite-compat-image.outputs.baseline_digest / .baseline_reason the previous set as verified above, or empty + why
baseline-module-artifacts satellite-compat-baseline-bundles the previous run's four bundles, re-uploaded under this run

Education — measured, green

Education has no compile-check job and no scripts/compile-check.allow on its main, so its leg judges the whole tree with nothing grandfathered — therefore it was measured before being listed, on the lane's exact script path (the lane's steps replayed: docker cp /app + shared frameworks from the promoted tester mw-plugin-test@sha256:1ad82ff0cbac…, the four module-bundle-* artifacts of publishing run 34684158910 unpacked into the reference set, compile-check.py --refs … --report with MW_REPO_ROOT/MW_ALLOW_FILE as the lane sets them) against Education main @ 7c106210ba6f0b8118dbe70b3cd101ce19d12020:

scope: full — every NodeType in all 9 package(s)
compile-check: 13 NodeType(s), 13 distinct source set(s) (13 with a configuration lambda), 378 ref assemblies
── summary ──  13 clean · 0 known-debt · 0 NEW break(s) · 0 fingerprint-drift · 0 stale-allow · 0 unverifiable   (7s)
✓ compile gate GREEN (full tree): 13 clean, 0 known-debt (shrinking), no new breaks.
rc=0

Positive control on the same harness — a ThisSymbolDoesNotExist reference appended to AgenticBusiness/Menu/Source/MenuAreas.cs in the scratch clone: ✗ 1 NON-allowlisted NodeType(s) FAIL to compile (regression): - AgenticBusiness/Menu, gate_fail: true, rc=1; clone restored. Education stays in the matrix, the reading in the workflow comment beside its entry. No Education issue was needed.

Reusable lane changes (node-repo-compile-check.yml, extended, not forked): optional content-repository, content-ref, verdict-artifact, judge-against-baseline, baseline-image-digest, baseline-module-artifacts, baseline-unavailable-reason inputs; optional content-app-id / content-app-private-key secrets (asserted RED when content-repository is set — never a fallback to github.token); job permissions gain packages: read — inside GitHub's RESTRICTED default, so the four satellites calling with inherit stay valid (check-workflow-permission-pairing.py --fleet: 44 pairs, 0 violations). With judge-against-baseline the new-image compile's exit code becomes a datum, a second compile runs against refs-baseline/ (previous image + its bundles), and the judge (compat-verdict.py, fetched at platform-ref like compile-check.py, self-tested in-run) sets the leg's colour. A node repo calling with no content-repository behaves byte-for-byte as before.

Scripts. compile-check.py: --report PATH writes the verdict as JSON (gate_fail, new_breaks[{type,error}], fp_drift, stale_allow, known_debt, the clean types by name (ok), counts, content.{repository,sha}); self-tested. compat-verdict.py (new): the classifier above; --self-test covers all rows in both directions — including the explicit fifth row "type absent from baseline, fails on new ⇒ unjudged, not BROKE", alone, beside an already-broken type, and beside a BROKE type — plus per-type granularity, fingerprint-drift, stale-allow, known-debt-on-new and the CLI exit codes (1 = broke, 0 otherwise, 2 = no new report).

Semantics — RED, but NOT blocking, and both halves are deliberate

Stated plainly: a red satellite leg ANNOTATES delivery — the Compatibility table on the run summary and a line on the ci-failure issue — and never stops promote, notify-platform-update, plugins-bake or publish-bake. Nothing downstream consumes the verdict.

And the contradiction that follows, stated: a red satellite leg turns the run RED while delivery has completed; the run's colour then says compatibility, not shipping. (No continue-on-error, by rule, so satellite-compat = failure makes the run's conclusion failure even though promote, notify, bake and seal all completed.) And since the baseline control, red means exactly one thing. So the verdict's summary opens — and its final headline repeats — 🚨 DELIVERY COMPLETED — this run is red because this core build BROKE <n> satellite(s): <list> (previous image compiled, new image does not). Nothing failed to ship., and the ci-failure alert body carries the same line above its Compatibility section (which lists only BROKE lines — an already-broken satellite or an unestablished baseline is an advisory on the run page, never an alert about this build). The same sentence is in ContinuousDeliveryContract.md.

  • Not blocking. Nothing waits for it: promote is already done, notify-platform-update and plugins-bake do not need it. The set is published; each instance gates itself. Blocking the publish on a satellite's compile would hand every satellite a veto over the platform's delivery.
  • Red. A failing leg fails the job → the run is red →
    • alert-on-failure (satellite-compat + satellite-compat-image in its needs) downloads the verdict artifacts and appends the banner plus one line per BROKE satellite — satellite, content sha, both digests, the set, the first three failing types with their first error.
    • delivery-verdict (both jobs in needs) renders a per-satellite table (content sha, verdict 🔴/🟡/✅/⚪, NodeType count, types, baseline → new digests) under Compatibility, runs under always() on a publishing run, and goes red itself only if the gate was skipped/cancelled on a publishing run or reports success with a satellite lacking a verdict. Executed locally against fixture verdicts for all rows (broke, already-broken, unjudged, already-broken with absent types, compatible, no-baseline) plus a not-measured leg: table, advisory line, banner and outputs as designed; skipped → rc 1; success with a missing verdict → rc 1.
    • preflight asserts FLEET_READER_APP_ID / FLEET_READER_APP_PRIVATE_KEY RED naming both secret stores. No continue-on-error anywhere; no if: on a secret; a leg that cannot mint, pull or compile is a red leg. Timeouts: lane job 20 (existing), satellite-compat-image 10, all literal.

What a red means and who acts: "this core build broke that satellite's NodeTypes — they compiled against the set the fleet is on and do not against this one." The ci-failure issue names satellite, both digests, set and types → the control instance's triage. The fix is on core's side by measurement: restore the surface, or — if the move was intended — land the satellite's adaptation. A satellite that was already broken is not a red here (advisory; its own daily run / ci-main-red issue own it), and a stale allow entry is an advisory. The daily rebuild will not repair a core break on its own.

Budget

  • Runner: ubuntu-latest in a public repo — zero billed minutes.
  • Measured compile duration: 1 min 50 s — the same lane, same digest-pinned image + four sealed modules, on MeshWeaver.Crm's own CI (compile-check / Compile every NodeType (vs core), run 34686027657, 2026-09-12 09:31:58Z → 09:33:48Z). Not a trial of this branch (below). Two compiles per leg now (new + baseline): expect ~4–5 min a leg; the extra compile is the control the red rests on.
  • Wall time: the legs need plugins-modules and run beside plugins-bake (which needs the same thing and takes longer): expected addition to CD's critical path ~0; worst case ≈ +5 min. satellite-compat-image (~1 min) runs beside plugins-modules.
  • Concurrency: 5 jobs per build against the org's 60-job ceiling; at ~43 merges/day ≈ 215 legs ≈ ~900 unbilled runner-minutes/day.
  • Not done, and why: narrowing the matrix to the satellites whose NodeTypes reference the changed surface. Nothing in the run knows which satellite binds which surface (that map lives in the satellites' node JSON), and a matrix that guessed would skip exactly the leg it should have run — a skipped leg reads as satisfied. If the concurrency cost ever matters, that narrowing is the honest lever; a shorter timeout is not.

The guards — original rationale, and why this job is outside each hazard

Each guard's comment quotes the commit that wrote the rule it extends (found by git log -S); each was proven able to fail by mutating the workflow (then restored, md5-verified).

Guard Original (commit, date) — the hazard, quoted Why satellite-compat is outside it Does the hazard apply?
PlatformBakeLaneGuard.PlatformBake_CompilesOnlyWhatTheImageEmbeds 4105384, 2026-08-26 (PR #2445): "the set of checked-out repositories in main-cd is exactly ["Systemorph/MeshWeaver.Plugins"]. Present, and the only one. A third would be a new decision rather than an extension of this one." Hazard: the ADOPT model — "plugins arriving pre-built from a bundle their own lane published", the #1814 bake-identity class. A checkout is dangerous there because what it feeds becomes the set. Read-only content at content-ref: main, resolved once to a sha the leg records; compiled into a throwaway check.csproj against the set ALREADY promoted; nothing enters an image, bundle or seal; no version derived; after promote, nothing waits, nothing consumes the verdict. No. Rule not relaxed: the register is asserted positively (exactly five, only in that job; outside it still only Plugins). Dropping Crm → red.
CdImageVersionsShareOneSourceGuard.EveryPluginCheckoutUsesTheCommitTheGateResolved ce16aa6, 2026-09-01: "A plugins merge landing mid-run therefore produces an image built from a commit the pair tag does not name … a portal image and the module bundles sealed beside it from DIFFERENT trees." Both halves are about set members. The satellites are not set members: no image or seal holds their bytes, no tag names them; each leg records its sha beside both digests it judged against. No. Detector still SEES the register; new fact TheSatelliteCompatRegister_IsAMeasurementNotAnImageInput excuses it only with content-ref: main and never Plugins. content-ref → sha: red.
PlatformNeverDependsOnPluginsGuard.OnlyTheImageLanes_ReachIntoAPluginRepository c88cd5d, 2026-09-01: an edge by which "core's own build or release can be broken by a repository it should not know about". On no pull request; builds nothing into an image or seal; runs after promote; reads content only. And the red is classified: a satellite's own broken commit fails at the baseline too → advisory, never red. No longer, by construction — a red means core moved a surface, measured against the set the fleet is on. main-cd was already ledgered; the reason names the gate and the control; MeshWeaver.Crm joins SatelliteRepos. Narrower shape (API-read-only) rejected: it cannot compile NodeTypes.
PlatformBakeLaneGuard.PlatformBake_NeverAdoptsAnotherBuildsBundles #1725: "a bundle from another run is a DIFFERENT compilation of the same source and resolves a different framework identity, so no pod can adopt what it publishes" — gh run download in a publishing workflow is the defect. satellite-compat-image downloads the previous run's bundles as a compile reference for the baseline reading; re-uploaded under a name only the compat legs read, unpacked into refs-baseline/, never into a bake directory, seal or publication. No. The detector skips exactly that job's block; new fact SatelliteCompatBaselineDownload_IsAReferenceNeverAPublication asserts the block still contains the download + re-upload and that neither it nor the consuming lane carries any publishing form (publish-bake-bundles.sh, push-bundle-publication.sh, --bake-output, --seed, az storage, BAKE_PUBLISH_TARGETS). Planting publish-bake-bundles.sh in the block → red; a gh run download outside it → the original fact red.

Guards (all with --root . and --self-test; verbatim summary lines, after the last push)

check-workflow-timeouts: 85 job(s) checked, 3 reusable-call job(s) exempt, 0 violation(s), cap=45 min   (--self-test rc=0, --root . rc=0)
check-workflow-shell: 1 finding(s): 0 live, 1 allow-listed, 0 stale entr(ies), 0 allow-file error(s)   (rc=0/0)
check-workflow-yaml-keys: 34 workflow file(s) and 0 composite action(s) checked, 0 violation(s)   (rc=0/0)
check-pr-secret-preflight: 2 secret(s) reachable on a Dependabot pull request, 2 asserted by a preflight, 0 violation(s)   (rc=0/0)
check-workflow-permission-pairing: 3 caller/lane permission pair(s) resolved, 0 violation(s)   (rc=0/0)
check-workflow-permission-pairing (fleet): 44 roster caller/lane pair(s) resolved, 0 violation(s)   rc=0
actionlint -shellcheck= -pyflakes= (CI's flags) rc=0; with embedded shellcheck on: identical to main's 6 pre-existing info/style notes
compile-check.py --self-test rc=0
compat-verdict.py --self-test rc=0: broke (red) · already-broken (green, advisory) · compatible · no-baseline (green, unjudged) · type-absent-from-baseline ⇒ unjudged, never BROKE (row 5) — all rows both directions, per-type granularity, drift, stale-allow and known-debt edges, CLI exit codes
test-cd-steps.py: all cases passed against main-cd.yml steps `release` + `decide` + `verdict` + `heal` (extracted, not copied)   rc=0
dotnet test test/MeshWeaver.Documentation.Test -c Release -warnaserror:
Passed!  - Failed:     0, Passed:   434, Skipped:     0, Total:   434, Duration: 16 s - MeshWeaver.Documentation.Test.dll (net10.0)   rc=0

Docs

One dated paragraph in Data/Architecture/ContinuousDeliveryContract.md ("After the promote" section), now covering the baseline control and the banner. No new page. No What's New entry: CI-only, no user-visible effect.

First-run plan

  • Which run: the first Continuous Delivery (main) run after merge whose gate decides publish=true. Reconcile ticks that publish nothing skip the compat lane by design, and delivery-verdict does not judge it on those.
  • What to check: (1) gate's line baseline (the set the fleet is on): mw-plugin-test:main = sha256:… = 3.0.0-ci.<n>, built by run #<n> and satellite-compat-image green with satellite-compat tester: … (equal to ACR's) and baseline set: ghcr.io/…@sha256:… (<version>, run <id> #<n>) + 4 module bundles — a baseline NOT established: <reason> warning instead means every leg refuses to judge (⚪) and the reason is the thing to fix; (2) five Compatibility: <satellite>'s NodeTypes compile against this set legs, none skipped or cancelled (the verdict's Compatibility step goes RED if the job did not run); (3) five satellite-compat-* artifacts, each a compile-check-verdict.json with verdict, content.sha, images.new and images.baseline; (4) the Compatibility table listing all five as ✅ (or 🟡 with a named satellite-side cause; expected: Education 13 clean); (5) delivery-verdict green; no ci-failure issue opened by this run.
  • If a leg is red: it means this build broke that satellite. The run is red, alert-on-failure files/comments the ci-failure issue with the banner and * **<satellite>** (main @ <sha>): compiled against the previous image (<digest>) and does not against <version>: <type> — <error>; … → control-instance triage: read the leg's log (both compiles are there); the fix is core's — restore the surface, or land the satellite's adaptation if the move was intended. The set is already published and every instance gates itself — nothing to roll back in delivery.
  • Rollback of this PR: git revert of the merge commit. Nothing else needs undoing — no instance, publication, tag, registry entry or downstream job consumes the verdict artifacts or the new jobs' outputs; the lane's new inputs are optional and the satellites' callers do not pass them.

Not done / to watch

  • No trial run. main-cd.yml's workflow_dispatch resolves main's HEAD server-side and, with rebuild=true, publishes a real image set, seals the Plugins publication and POSTs the release event — not a branch trial. The reusable lane has no dispatch trigger. The compile duration is the same lane on Crm's CI; the verdict/alert shell was executed locally from the workflow file against fixtures; the first real execution is the first main build after merge.
  • Cross-registry digest: promote phase B copies the index to GHCR by digest; satellite-compat-image asserts equality and fails RED naming both digests if that ever stops holding.
  • Baseline availability: the previous run's bundle artifacts live 7 days and the run lookup scans the last 100 runs of main-cd.yml (~1.5 days at current cadence); a baseline older than that yields a refusal-to-judge with that reason, not a red. main carries a 3.0.0-ci.<n> tag today (measured: 3.0.0-ci.8414).
  • Education is measured green today; it runs no compile-check on its own PRs, so a future Education merge can turn its leg 🟡 on its own change — the table's content.sha and the baseline column tell that apart from a core break.
  • plugins-modules comment says "this repo holds no registry credential" while MW_REGISTRY_KEY exists in core's secrets (no MW_REGISTRY_URL var). Not touched; noting it.

🤖 Generated with Claude Code

…e's NodeTypes against the new set (phase 2a)

The fleet rebuilds its satellites once a day (maintainer, 2026-09-12), which
removed the only per-build evidence that a core merge did not break them.
This is the counterweight: after `promote`, one reusable compile-check leg
per satellite (SocialMedia, Crm, Reinsurance, Education, Manufacturing),
in parallel, against the tester THIS run promoted and the module bundles
THIS run packed. RED but not blocking: nothing waits for it, a red leg
fails the job, `alert-on-failure` names the satellite/set/types on the
ci-failure issue, `delivery-verdict` renders a per-satellite table and
goes red only if the gate was SKIPPED on a publishing run.

* main-cd.yml: `preflight` asserts FLEET_READER_APP_ID/_PRIVATE_KEY red;
  `satellite-compat-image` resolves the tester's GHCR mirror and asserts
  its digest equals ACR's (core holds no ACR password; OIDC inside the
  lane would be the #3933 startup-failure shape); `satellite-compat`
  matrix `uses:` node-repo-compile-check.yml; verdict + alert extended.
* node-repo-compile-check.yml: optional `content-repository`/`content-ref`
  + `content-app-*` (publish-bake's shape), `verdict-artifact`, and
  `packages: read` (inside GitHub's restricted default, so the four
  `inherit` callers stay valid — pairing --fleet green).
* compile-check.py: `--report PATH` writes the verdict as JSON; self-tested.
* Guards: PlatformBakeLaneGuard pins the register (exactly five, only in
  that job); CdImageVersionsShareOneSourceGuard exempts the register
  POSITIVELY (ref is `main`, never Plugins); PlatformNeverDependsOnPlugins
  now sees MeshWeaver.Crm and its main-cd ledger names the gate.
* ContinuousDeliveryContract.md: one dated paragraph.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 12, 2026 10:39
rbuergi and others added 2 commits September 12, 2026 12:48
…arry their original rationale; Education measured, not predicted

Review feedback on #4088:

* The three guards now QUOTE the hazard their 2026-08-26 / 2026-09-01
  originals closed (4105384: the #1814 bake-identity class — bytes
  from a tree the identity does not name; ce16aa6: a mid-run merge
  falsifying the pair tag's provenance and sealing bundles from a third
  tree; c88cd5d: a core PR verdict behind a sibling checkout) and
  state per hazard why satellite-compat is outside it: read-only
  content at `content-ref: main`, resolved to a sha the leg records in
  its verdict, compiled into a throwaway check.csproj against the set
  already promoted, never built into an image or seal, nothing waits
  on it, nothing consumes the verdict. What it CAN do — red a CD run
  on a satellite's state — is the decision.
* node-repo-compile-check.yml resolves the content commit once and the
  report carries {repository, sha}; the verdict table and the
  ci-failure line show `<satellite> (main @ <sha>)`. Self-tested.
* Education measured on the lane's exact script path (main 7c106210b,
  tester sha256:1ad82ff0…, run 34684158910's four bundles): 13
  NodeTypes, 13 clean, 0 debt, 7 s; the harness reds by name on a
  planted break. It stays in the matrix, with the reading in the
  comment.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… the RUN while delivery completed

A red satellite leg fails `satellite-compat` (no continue-on-error, by rule),
so the run's conclusion is `failure` while promote, notify, bake and seal all
completed. The run's colour then says COMPATIBILITY, not shipping — stated
now rather than left implicit:

* delivery-verdict: the Compatibility step (id: compat) opens its summary
  with `🚨 DELIVERY COMPLETED — this run is red because N satellite(s)
  failed compatibility: <list>. Nothing failed to ship.` and exports
  compat_red / compat_red_list; the run's final headline ("Say plainly")
  repeats it above the PUBLISHED line when any leg is red.
* alert-on-failure: the same line opens the Compatibility section of the
  ci-failure body.
* ContinuousDeliveryContract.md: the same sentence in the dated paragraph.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved workflow reporting, artifact handling, and architecture-documentation issues remain.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds non-blocking per-build compatibility checks for five satellite repositories against the promoted platform image and modules.

Changes:

  • Adds satellite matrix checks, digest verification, reporting, and alerts.
  • Extends reusable compile-check workflows and JSON verdict generation.
  • Updates architecture documentation and validation guards.
File summaries
File Summary
test/MeshWeaver.Documentation.Test/PlatformNeverDependsOnPluginsGuard.cs Validates the updated dependency direction.
test/MeshWeaver.Documentation.Test/PlatformBakeLaneGuard.cs Validates the satellite register.
test/MeshWeaver.Documentation.Test/CdImageVersionsShareOneSourceGuard.cs Updates image/reference guard behavior.
src/MeshWeaver.Documentation/Data/Architecture/ContinuousDeliveryContract.md Documents the compatibility policy.
.github/workflows/node-repo-compile-check.yml Adds cross-repository inputs and verdict artifacts.
.github/workflows/main-cd.yml Adds compatibility jobs, reporting, and alerts.
.github/scripts/compile-check.py Generates machine-readable compatibility reports.
Review details

Suppressed comments (10)

.github/workflows/main-cd.yml:3160

  • When satellite-compat fails before compile-check.py --report (for example during checkout, image setup, or artifact download), unmeasured is nonzero but COMPAT_RESULT is failure, so the success-only guard is skipped and red remains zero; this else then prints every satellite compiles even though the table says NOT MEASURED. That is a false-green summary for the exact fail-closed case this relay is meant to expose. Only emit the green message when the job succeeded and every satellite has a verdict; otherwise report the measurement as incomplete.
              echo "::error::the gate decided to PUBLISH, but satellite-compat=${COMPAT_RESULT} (satellite-compat-image=${COMPAT_IMAGE_RESULT}): the compatibility of this set with the satellites was NOT measured. A skipped gate is painted like a passed one; it is not one. Read satellite-compat-image / plugins-modules for why the legs never ran."
              exit 1 ;;
          esac
          if [ "$COMPAT_RESULT" = "success" ] && [ "$unmeasured" -gt 0 ]; then
            echo "::error::satellite-compat reports success but $unmeasured satellite(s) above have no verdict artifact — a leg that measured nothing must not read as a pass. Either the matrix and the SATELLITES register in this step diverged, or a leg uploaded no report."

.github/workflows/main-cd.yml:2577

  • The budget counts only the five matrix legs, but this change also adds the separate satellite-compat-image runner job, which runs on every publishing build and overlaps with the legs/modules. Thus the stated “five concurrent jobs per build” and 215-leg daily estimate undercount the new concurrency and runner load; please account for five satellite legs plus one digest-verification job.
  # Budget: ubuntu-latest in a public repository — unbilled. Five parallel legs of ~2 min add
  # ≤ ~3 min of wall time beside `plugins-bake` (which they do not wait for), and five concurrent
  # jobs per build against the org's job ceiling. Narrowing the matrix to the satellites whose

.github/workflows/main-cd.yml:3213

  • By adding satellite-compat to this alert path, a compatibility-only failure now creates/comments the ci-failure issue, but the label is created below with the description “no deployable image was published.” The new failure mode explicitly publishes the image set, so the durable issue classification is false and can mislead triage/automation; use a separate compatibility label or update the label contract to distinguish measurement failures.
            echo "### 🚨 DELIVERY COMPLETED — this run is red because ${COMPAT_RED} satellite(s) failed compatibility: ${COMPAT_RED_LIST}. Nothing failed to ship." >> "$GITHUB_STEP_SUMMARY"
            echo "✅ PUBLISHED an image set for \`${SHORT}\` (reason: ${REASON}) — the run's colour says COMPATIBILITY, not shipping; see the Compatibility table above and the \`ci-failure\` issue." >> "$GITHUB_STEP_SUMMARY"
          elif [ "$PUBLISH" = "true" ] && [ "$PROMOTE_RESULT" = "success" ]; then
            echo "### ✅ PUBLISHED an image set for \`${SHORT}\` (reason: ${REASON})" >> "$GITHUB_STEP_SUMMARY"

.github/workflows/main-cd.yml:3103

  • actions/download-artifact fails when a pattern matches no artifacts unless the no-match case is handled. That is exactly what happens when satellite-compat is skipped before any leg can upload (for example, if its image bridge or plugins-modules dependency fails), so this step aborts before the explicit NOT MEASURED check below and no compatibility summary is rendered. Make collection of an absent artifact set non-fatal while preserving the following script's explicit skipped/missing-verdict failure path; the existing shard collector uses this pattern for the same action behavior.
      # run, and `alert-on-failure` files it.
      # `always()`: the table renders even when a verdict step above went red — a run that is
      # already red for delivery still owes the reader the compatibility reading it took.
      - name: Collect the satellite verdicts

.github/workflows/main-cd.yml:3230

  • The alert path has the same no-match failure: when no compatibility leg produced a report, download-artifact fails here and the job exits before Create or comment on ci-failure issue. That can leave the CD run red for an unmeasured compatibility gate with no failure issue at all. Make this optional evidence collection tolerate an empty artifact set so the generic alert is still filed, while retaining the per-satellite enrichment when reports exist.
    # and a failure nobody is paged for is still invisible.
    # `preflight` and `notify-platform-update` are in this list ON PURPOSE (#2235): the release
    # event that starts the cross-repo rebake wave used to be unfailable, and a failure nobody is
    # paged for is still invisible. A red notify job does NOT mean the images are missing —
    # `verify-images` answers that — it means no satellite learned about them.

.github/workflows/main-cd.yml:3143

  • The report already carries .unverifiable, and compile-check.py does not set gate_fail for those NodeTypes. With any such entry, this else branch still emits ✅ compiles and the final summary says every satellite compiles even though part of its tree was never verified. Surface the unverifiable count/status here and ensure the run cannot present an incomplete measurement as a clean compatibility result.
            debt=$(jq -r '.known_debt | length' "$f")
            breaks=$(jq -r --arg bt '`' '([.new_breaks[].type] + [.fp_drift[].type] + [.stale_allow[] | . + " (compiles now — remove from allow)"]) | map($bt + . + $bt) | join(", ")' "$f")
            if [ "$fail" = "true" ]; then
              red=$((red + 1)); redlist="${redlist:+$redlist, }$sat"

.github/workflows/main-cd.yml:3254

  • This message says the satellite "does not compile" for every reported red item, but stale_allow is deliberately a red ratchet result for a type that now compiles cleanly. The generated issue would therefore tell triage the opposite of what happened (the detail appended on the same line contradicts the headline). Use wording such as "has compatibility failures" or classify stale-allow entries separately.
        env:
          GH_TOKEN: ${{ github.token }}

src/MeshWeaver.Documentation/Data/Architecture/ContinuousDeliveryContract.md:686

  • Education is explicitly documented in this change as having no compile-check caller on its main, so this sentence is false for one of the five legs. Clarify that the lane uses the reusable gate adopted by the other satellites and applies it directly to Education as an adoption gap; otherwise the contract claims a PR gate that does not exist.
in parallel runs the same reusable compile gate the satellites run on their own pull requests
(`node-repo-compile-check.yml`, pointed at the satellite's `main` through the read-only fleet-reader

src/MeshWeaver.Documentation/Data/Architecture/ContinuousDeliveryContract.md:700

  • A red compatibility report is not necessarily evidence that this set breaks a NodeType: the checker also fails on fingerprint drift and stale allow entries (and Education has no allow file, so pre-existing debt can be newly exposed). This statement makes the triage contract inaccurate. Describe the result as a new break or ratchet/measurement failure, then keep the existing guidance about where the fix belongs.
under **Compatibility** — and goes red itself if the gate was *skipped* on a publishing run, because
a skipped measurement painted grey is the one outcome worse than a red one. `fail-fast: false`, no

test/MeshWeaver.Documentation.Test/PlatformNeverDependsOnPluginsGuard.cs:109

  • Adding Crm changes the current fleet denominator to five satellites, but the repository-wide architecture inventory still says MeshWeaver.Plugins plus four satellites (RepositoryDependencyDirection.md:11) and the current bake contract still describes four satellite publishers (CiContentBake.md:695). The new contract paragraph lists five, so the shipped documentation now disagrees about which repositories the fleet contains. Update the affected current-count statements in the same change.
    [
        "MeshWeaver.Plugins",
        "MeshWeaver.Education",
        "MeshWeaver.Reinsurance",
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +66 to +71
+ "core/plugins PAIR (#2622), packs+bakes the Plugins module bundles for that identity, "
+ "and — since 2026-09-12, the counterweight to the satellites' once-a-day rebuild — "
+ "points the reusable compile gate at each satellite's main in `satellite-compat`: a "
+ "read-and-compile after `promote` that blocks nothing and builds nothing into an "
+ "image, RED so a break is known within the hour rather than at 03:00 (its register "
+ "is pinned by PlatformBakeLaneGuard)"),
rbuergi and others added 2 commits September 12, 2026 13:05
…ld broke it, never "the satellite was already broken"

Maintainer pushback (via the reviewing session): "why fail compatibility?" —
a single red compile cannot say which side moved: core changed a surface
(red is right) or the satellite merged broken content (core reddened by a
repo it should not know about — the c88cd5d hazard, exactly).

The control: every leg compiles the same content a second time against the
set the fleet is ON, and compat-verdict.py (new, self-tested on every row)
classifies per NodeType — compiled at the baseline and not now = BROKE, the
only red; failed there too or absent there = already broken, satellite
side, green + advisory; no baseline = refusal to judge, green + advisory,
new-image failures listed unjudged.

The baseline is the one derivation the run already has: `gate` records
mw-plugin-test:main's digest + 3.0.0-ci.<run> tag pre-promote (same ACR
read as its completeness probe); satellite-compat-image asserts GHCR holds
it, locates the building run by run_number and re-uploads that run's four
module bundles (the ones that SHIPPED with the baseline) for the legs. Any
gap = "no baseline: <reason>", never fatal, never silent. Both digests are
in every verdict.

Banner: "🚨 DELIVERY COMPLETED — this run is red because this core build
BROKE N satellite(s): <list> (previous image compiled, new image does
not). Nothing failed to ship." — verdict table, headline, alert (BROKE
lines only), contract paragraph, guard comments.

PlatformBakeLaneGuard.PlatformBake_NeverAdoptsAnotherBuildsBundles (#1725,
cross-run ADOPTION) now exempts exactly the satellite-compat-image block,
and a new fact asserts the block still contains the download and that
neither it nor the consuming lane carries any publishing form. Verdict
step, headline and alert composition executed locally against fixtures
for all rows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ver BROKE — and why the baseline uses the baseline run's bundles

* compat-verdict.py: a type that fails on the new image and is absent from
  the previous report gets its own row, `unjudged` (reason: no per-type
  baseline — newer than the set the fleet is on), green, advisory; never
  folded into already-broken. Only previous-compiled-ok → new-fails
  licenses "broke". Explicit self-test row 5, alone / beside already-broken
  / beside broke, plus a per-type-granularity case (all-red baseline still
  yields BROKE for the one type that compiled there).
* delivery-verdict table: 🟡 unjudged row; absent types listed distinctly on
  broke and already-broken rows.
* Workflow comment + contract paragraph: WHY the baseline compiles against
  the BASELINE run's module bundles (located by run number) and not this
  run's — a Plugins-side move between the two sets would otherwise be
  filed as "already broken" (or fail the baseline spuriously), destroying
  the attribution the control exists for.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

508 tests  ±0   508 ✅ ±0   42s ⏱️ ±0s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 425d68f. ± Comparison against base commit 46459b4.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

390 tests  ±0   390 ✅ ±0   58s ⏱️ +14s
  2 suites ±0     0 💤 ±0 
  2 files   ±0     0 ❌ ±0 

Results for commit 425d68f. ± Comparison against base commit 46459b4.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

259 tests  ±0   259 ✅ ±0   2m 13s ⏱️ +8s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 425d68f. ± Comparison against base commit 46459b4.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

1 331 tests  ±0   1 329 ✅ ±0   3m 50s ⏱️ ±0s
    3 suites ±0       2 💤 ±0 
    3 files   ±0       0 ❌ ±0 

Results for commit 425d68f. ± Comparison against base commit 46459b4.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

2 017 tests  +2   2 017 ✅ +2   4m 43s ⏱️ -22s
    4 suites ±0       0 💤 ±0 
    4 files   ±0       0 ❌ ±0 

Results for commit 425d68f. ± Comparison against base commit 46459b4.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

    6 files  ±0      6 suites  ±0   8m 36s ⏱️ +2s
2 822 tests ±0  2 630 ✅ ±0  192 💤 ±0  0 ❌ ±0 
2 826 runs  ±0  2 634 ✅ ±0  192 💤 ±0  0 ❌ ±0 

Results for commit 425d68f. ± Comparison against base commit 46459b4.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results

   17 files  ±0     17 suites  ±0   21m 4s ⏱️ ±0s
7 327 tests +2  7 133 ✅ +2  194 💤 ±0  0 ❌ ±0 
7 331 runs  +2  7 137 ✅ +2  194 💤 ±0  0 ❌ ±0 

Results for commit 425d68f. ± Comparison against base commit 46459b4.

@rbuergi
rbuergi marked this pull request as ready for review September 12, 2026 12:13
@rbuergi
rbuergi added this pull request to the merge queue Sep 12, 2026
Merged via the queue into main with commit 983c383 Sep 12, 2026
35 checks passed
@rbuergi
rbuergi deleted the cd/satellite-compat-gate branch October 10, 2026 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants