fix(ci): the arm lane merges with an App token, so main's push lanes run (#2916) by rbuergi · Pull Request #3003 · Systemorph/MeshWeaver

rbuergi · 2026-09-01T19:12:30Z

Fixes the mechanism behind #2916 — full RCA on the issue.

What was happening

Nothing had been published since 8cc1cc6e (15:13Z on 2026-09-01). Not because a job failed — because the image jobs never ran.

GitHub performs an auto-merge as the identity that armed it. auto-arm.yml (#2959, merged the same day) armed with secrets.GITHUB_TOKEN, so every merge landed as github-actions[bot] — and a push created with GITHUB_TOKEN deliberately does not trigger workflow runs.

main commit	merged by	`push`-event runs on that SHA
`8cc1cc6e` 15:13	rbuergi (human)	Build and Test ✅ · Chart Gate ✅ · Hosting Operator ✅
`19628536` 16:16	`github-actions[bot]`	none
`b3e6ae65` 16:26	`github-actions[bot]`	none
`04d25efa` 17:26	`github-actions[bot]`	none
`dc6c5a45` 18:27	`github-actions[bot]`	none

$ gh api repos/Systemorph/MeshWeaver/commits/dc6c5a45d/check-runs --jq '.check_runs|length'
0

No push run ⇒ no Consolidate test results on main's HEAD ⇒ main-cd's readiness gate reads absent/none and waits — on every tick, forever. Six hours, four commits, every self-updating install frozen on the previous image, and every dashboard green throughout.

The reconciler already distinguishes running from never ran. There is a third state it cannot see: the trigger was suppressed, so the check will never exist. That is the AGENTS.md skip-trapdoor rule one level down — a suppressed trigger renders exactly like a slow one, and absence is the one thing polling cannot age out of.

The fix — at the credential, not at the gate

Teaching CD to accept weaker evidence (the green-tree marker refs/ci-green/<tree>/<epoch>) would let it publish while Chart Gate and Hosting Operator stayed silently dead — one invisible hole traded for two. The merge simply has to be performed by an identity whose pushes behave normally.

auto-arm.yml now:

asserts MESHWEAVER_APP_ID / MESHWEAVER_APP_PRIVATE_KEY — red, naming what to provision;
mints an installation token requesting permission-contents: write (the merge writes to the branch) and permission-pull-requests: write (arming is the enablePullRequestAutoMerge mutation), so a missing grant fails at the mint rather than downstream;
arms with that token, with no fallback — falling back to GITHUB_TOKEN is the outage itself.

No if: on any of the three steps. The one exemption stays on the job and remains a property of the event (draft, fork), never a question about whether a credential is set.

🚨 Requires a grant before it can arm

The meshweaver-cloud App currently holds only contents: write + metadata: read:

$ gh api orgs/Systemorph/installations --jq '.installations[] | "\(.app_slug) \(.permissions|tojson)"'
azure-boards      {"actions":"write","checks":"write","contents":"write","pull_requests":"write",...}
meshweaver-cloud  {"contents":"write","metadata":"read"}

Grant Pull requests: Read and write in the App's settings and accept the new permission on the org installation (a requested-but-unaccepted permission is not granted). Until then the arm job fails red naming exactly that — which is the intended shape, and does not block merging: Consolidate test results is the only required context, so PRs stay mergeable by hand meanwhile.

auto-arm.yml is identical in all seven fleet repos, so the one grant fixes the fleet; the file change should follow to the others.

Guard

ArmedMergeMustTriggerMainsPushLanesGuard fails the build if any workflow merges or arms a PR with GITHUB_TOKEN, and if the arm lane stops naming a minted token (deleting the GH_TOKEN line would recreate the failure by omission).

It reads configuration, normally the weak shape — deliberately, and the file says so at length. The outcome ("did the merge start main's push lanes?") is unobservable from a pull-request run by construction: it can only be seen on main, after the merge, and what you would inspect is precisely what is missing. The credential is the only place a pre-merge check can stand, and it is causal rather than correlated — with GITHUB_TOKEN the trigger is suppressed 100% of the time, by documented design.

Verification

Mutation test: reverted auto-arm.yml to origin/main and ran the guard → 2/2 failed, naming auto-arm.yml; restored → passes.
Full project, not a filter (a --filtered run is a false pass for ratchet guards): MeshWeaver.Documentation.Test 168 passed, 0 failed, DocumentationLinkIntegrityTest present in the .trx.
dotnet build -c Release -warnaserror on the touched project: 0 Warning(s), 0 Error(s).

Also conserved

Doc/Architecture/ContinuousDeliveryContract gains "The third state the reconciler cannot see — a SUPPRESSED trigger", beside the existing running-vs-never-ran paragraph it extends.
A What's New entry (Category: Fix) — an installation that looked stuck yesterday was not stuck; there was nothing newer to take.

Still open, and it will re-fire the moment publication resumes

The last commit that actually reached the bake, af817d9a (run 33522667934), failed on a genuine and unrelated defect — the images promoted and verified fine, only the bake leg went red:

framework-identity: MISMATCH — the bake published under 's5f11d7e9a9142927c3c7c9da657a485d'
but '/portal' resolves 'sbc0cb08739867bb4cb45a8b5deab48a7'. Bundles published under an identity
this host never asks for are INERT ... (issue #1814).

Two independent stoppages are stacked here. This PR fixes the trigger; the identity mismatch is not touched and will surface again once commits start reaching the bake.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SD7aiLSo59xng2TzU32xEa

…run (#2916) Auto-merge is performed by GitHub as the identity that ARMED it. `auto-arm.yml` armed with `secrets.GITHUB_TOKEN`, so merges landed as `github-actions[bot]` — and a push created with GITHUB_TOKEN deliberately does not trigger workflow runs. main's HEAD therefore received no `push`-event run of anything: 8cc1cc6 merged by rbuergi Build and Test / Chart Gate / Hosting Operator 1962853 b3e6ae6 04d25ef dc6c5a4 merged by github-actions[bot] -> nothing No push run means no `Consolidate test results` on main's HEAD, which is the evidence main-cd gates publication on. It read `absent/none` and waited, every tick, for six hours across four commits: nothing was promoted and every self-updating install stayed on the previous image, with no red anywhere. The reconciler already separates "running" from "never ran". It cannot separate either from "the trigger was suppressed, so the check will never exist" — the skip-trapdoor shape one level down, where a suppressed trigger renders exactly like a slow one. Cure at the credential, not at the gate: teaching CD to accept the green-tree marker would let it publish while Chart Gate and Hosting Operator stayed silently dead. The arm now runs on a minted GitHub App installation token, requesting contents:write (the merge) and pull-requests:write (arming is the enablePullRequestAutoMerge mutation) so a missing grant fails at the mint, loudly, naming what to provision — with no fallback. ArmedMergeMustTriggerMainsPushLanesGuard fails the build if any workflow merges or arms a PR with GITHUB_TOKEN. It reads configuration, which is normally the weak shape, and says so: the outcome is unobservable from a PR run by construction, since what you would inspect is precisely what is missing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SD7aiLSo59xng2TzU32xEa

Copilot

Pull request overview

This PR fixes a CI/CD outage mode where merges performed (or armed) with secrets.GITHUB_TOKEN suppress downstream push-triggered workflows, causing main’s push lanes (and therefore publishing) to never run.

Changes:

Update .github/workflows/auto-arm.yml to mint and use a GitHub App installation token (with explicit permissions) rather than GITHUB_TOKEN.
Add a guard test (ArmedMergeMustTriggerMainsPushLanesGuard) that fails if any workflow merges/arms using secrets.GITHUB_TOKEN, and asserts auto-arm.yml continues to mint and pass an installation token.
Document the “suppressed trigger” third state in the CD contract and add a What’s New entry describing the incident and fix.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
`.github/workflows/auto-arm.yml`	Switch arming to a minted GitHub App token; add explicit preflight/assertions and better diagnostics.
`test/MeshWeaver.Documentation.Test/ArmedMergeMustTriggerMainsPushLanesGuard.cs`	Adds a guard test preventing reintroduction of `GITHUB_TOKEN` for merge/arm lanes and asserting the minted-token shape remains.
`src/MeshWeaver.Documentation/Data/Architecture/ContinuousDeliveryContract.md`	Documents the suppressed-trigger failure mode and why the fix must be credential-based.
`src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-02-updates-reach-your-installation-again.md`	Adds a user-facing What’s New entry describing impact and resolution.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-09-01T19:17:02Z