Repository navigation
ci(scripts): a platform canary — tell a node repo on the DAY core breaks it, not at the next pin bump - #3806
Conversation
…aks it, not at the next pin bump MeshWeaver.Plugins#1519. Twice on 2026-09-08 a core behaviour change behind an UNCHANGED SIGNATURE reddened that repo, and both times it was found by the pin bump rather than before it: Plugins#1506 core #3681 restricted the legacy access heal to pre-installed partitions — found by reading core's open PRs. Luck, not a gate. Plugins#1518 core #3648 made the platform floor ADVISORY; 14 tests asserted the hold/refuse/skip it replaced — found by the bump itself, WITH THE TRUNK ALREADY RED and every open pull request blocked on a gate none of their diffs could reach. The structural reason is not incidental: no surface detector sees this class by construction — the cross-repo pair gate looks for REMOVED types and members, and nothing was removed — and for a dependent that PINS core, the release event resolves the same pin a pull request does. So the pin bump IS the integration test, and wherever the App lacks `workflows` (Plugins#1464/#1547) that bump is a hand move, which in practice happens when the staleness ratchet has already forced it. This lane runs that same integration test on a schedule, moving nothing. 🚨 IT NEVER GATES. Core main moves under it, so a red canary must never block a pull request that is fine at the pin — a caller that made this required would be blocked by someone else's core merge. It reports; a human decides when to adopt. 🚨 THE CONTROL ARM IS THE LOAD-BEARING HALF. The suites run TWICE — once at the caller's current pin, once at core main — and only tests that PASS at the pin and FAIL at main are reported. Without it every pre-existing red reads as pin drift, the report is noise, and the lane is ignored inside a week: exactly the failure AGENTS.md names for a gate that is red when nothing is wrong, and the one Plugins#1510 had just finished fixing in compile-check.py. Details that are deliberate rather than incidental: · core main is resolved ONCE and both checkouts read that output — two checkouts of a branch name in one run legitimately take different commits, which is what ci.yml's platform-ref job exists for. · a suite that cannot BUILD at one arm is a RESULT ("<suite>: BUILD FAILED"), not a reason to abandon the comparison; a build broken at both arms cancels in the delta, which is the control working. · the tracking issue is edited and re-commented while the delta persists and CLOSED when core main stops breaking anything — the same report shape as the platform-ref bump lane, because a scheduled red surfaces to nobody. · `suites` is required with no default: running everything would cost more and report the same, and guessing which suites matter for a caller is exactly the kind of default that silently measures nothing. Verified offline before shipping: the workflow parses; the failed-test extraction reads real xUnit output including theory cases with parenthesised arguments; and the delta over fixtures reports only the newly-broken test, excluding one already red at the pin and one whose build broke at both arms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
The workflow currently contains a bash syntax error in the pin/main resolution step and uses continue-on-error in a way that can silently misreport a failed run as “clean.”
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a reusable GitHub Actions workflow that runs a “platform canary” for node repos by executing selected test suites twice (once against the repo’s pinned core commit, once against core main) and filing/updating a tracking issue with only the delta (tests that pass at the pin but fail at main).
Changes:
- Introduces
.github/workflows/node-repo-platform-canary.ymlas aworkflow_callreusable workflow with inputs for pin location and targeted test suites. - Resolves the upstream repo’s default-branch HEAD SHA once per run, checks out both core SHAs, runs suites against both, and computes/report deltas.
- Opens/updates/closes a single tracking issue (by marker) to make scheduled failures visible and self-clearing.
File summaries
| File | Description |
|---|---|
| .github/workflows/node-repo-platform-canary.yml | New reusable workflow implementing the pin-vs-main delta canary, with issue-based reporting. |
Review details
Suppressed comments (1)
.github/workflows/node-repo-platform-canary.yml:171
- Same as above:
continue-on-error: truecan hide an arm failing to run at all, which would make the pin-vs-main comparison unreliable (and could lead to closing an existing tracking issue on no evidence).
- name: Run the suites at core MAIN
continue-on-error: true
env:
- Files reviewed: 1/1 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| # legitimately take two different commits; core merges several times an hour. | ||
| echo "pin=$pin" >> "$GITHUB_OUTPUT" | ||
| echo "main=$main" >> "$GITHUB_OUTPUT" | ||
| echo "behind=$(gh api "repos/${UPSTREAM}/compare/${pin}...${main}" --jq .ahead_by)" >> "$GITHUB_OUTPUT" |
|
Additional evidence for the canary comparison at head The workflow's set difference then reports Please make the baseline require positive per-test pass evidence (fresh TRX), and classify build/host failures as an incomplete comparison rather than a test pass. Incomplete arms must not close a prior tracking issue. A fixture covering baseline-build-failure + candidate-test-failure should pin this distinction. This is a local workflow-logic reproduction, not a real failure in a dependent suite. Also, the earlier review's nested-quote warning on |
|
Verified update: d36f769 is now the remote head of this PR. The first push was rejected by the merge queue; I dequeued the PR through dequeuePullRequest, pushed successfully without force, and verified the head with REST. Fifteen focused tests and workflow checks passed locally. Fresh CI/review must complete before this correction merges. |
|
ℹ️ Merge-queue steward: no action — removed from the queue with reason |
🚨 There are now TWO canaries from Plugins#1519 — this lane, and an inline one in MeshWeaver.PluginsFlagging a duplication I am partly responsible for. Both landed on 2026-09-09 from the same issue:
I filed #1519 and built the shared lane; someone independently built the Plugins one. Neither of us They are not redundant copies — each has something the other lacksThe Plugins one is better structured in two ways I did not do:
This lane has what that one does not: it files and closes a tracking issue, which is the half The convergence, and it is the pattern we just used twice todaySame shape as Concretely: this lane takes the matrix and the comparator self-test; Plugins' I am not doing that unilaterally — the Plugins implementation is somebody's fresh work and they may Leaving both as they are until then is the one option that should not stand — two canaries mean two |
Implements the platform-canary proposal in Systemorph/MeshWeaver.Plugins#1519. The caller workflow is a separate change in that repository.
The scheduled canary compares selected dependent suites at their current core pin and at one resolved core-main commit. It reports compatibility regressions without becoming a required PR check or moving the pin.
Evidence and outcomes
Verification
python3 .github/scripts/test-platform-canary.py: 15 tests pass, covering the demonstrated baseline-build-failure false regression, real pass-to-fail comparisons, compiler versus infrastructure failure, missing/skipped tests, partial results, host exits, duplicate names, and theory names. The parser also successfully read an actual five-test xUnit TRX from the local radio-binding reproduction.Workflow shell, YAML-key, timeout, and secret-preflight checks pass. The actual scheduled comparison still needs its first run after the dependent caller is wired; fixture tests do not establish that the dependent suites pass against core main.
No What's New entry: internal CI reporting only.
Pairs-with: none — CI only; no public framework surface changes.