Repository navigation
ci(dependent-suites): request after core's own suites; name the queue when it waits - #6331
Merged
Merged
Conversation
… when it waits The required `Dependent suites (MeshWeaver.Plugins)` gate blocked core merges on runner capacity (core #6320 red on "No verdict in time"; triage rbuergi/Feedback/dependent-suites-gate-times-out-on-queued-candidate-20261008T1740Z). Measured 2026-10-08 17:30Z: 10 live candidate runs, 48 candidate legs queued for the shared dind pool. * `Dependent suites (request)` now needs `build` and `test` and sends the dispatch only once core's own suites are green on the candidate (or the tree is a reused green). Otherwise it is NOT sent and goes RED by name — "waiting for core's own suites" — and Consolidate repeats it. Never a skip. * await-dependent-verdict.py prints a QUEUE CENSUS every 5 min and at the deadline (Actions metadata of Plugins' core-candidate.yml runs, `actions: read` on the same App token): this candidate's jobs, its position among live candidate runs, the keys ahead, Plugins' queued total. A timeout now says "no runner within 42 min (queued behind N candidate run(s): …) — waiting for a runner: <job>", "still running at the deadline", or "no run carries this key". The census never decides anything (self-test + negative control: silence is still no verdict). * The no-verdict step's remedy is corrected: a candidate whose core run has finished stands down (and is now reaped by Plugins' next admission), so re-run the waiter at once, or re-run the request for a fresh key. The 42-minute wait and every cap are unchanged. Doc: Architecture/CrossRepoPairGate § "Runner capacity". The Plugins half (build once per arm; reap) is MeshWeaver.Plugins#3187. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The dependent gate can validate a different merge tree than core tested, and queue metadata handling can expose or misreport private workflow state.
4 open findings
What changed in this PR
This PR reduces dependent-suite runner contention by delaying Plugins validation until core CI passes and improves queue-timeout diagnostics.
Changes:
- Gates dependent-suite dispatch on successful core build/tests.
- Adds a Plugins queue census and clearer timeout reasons.
- Updates architecture documentation and dependency guards.
| File | Description |
|---|---|
.github/workflows/dotnet-test.yml |
Delays dispatch and reports timeout context. |
.github/scripts/await-dependent-verdict.py |
Adds queue census and diagnostic output. |
src/MeshWeaver.Documentation/Data/Architecture/CrossRepoPairGate.md |
Documents the revised workflow. |
test/MeshWeaver.Documentation.Test/PlatformNeverDependsOnPluginsGuard.cs |
Updates the cross-repository API-read ledger. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+3903
to
+3906
| if [ "$REUSED" = "true" ]; then | ||
| echo "this tree was already tested green (precheck reuse) — core's own suites are green; requesting" | ||
| elif [ "$BUILD" = "success" ] && [ "$TEST" = "success" ]; then | ||
| echo "core's own build ($BUILD) and tests ($TEST) are green on this candidate — requesting the dependent's suites" |
Contributor
Author
There was a problem hiding this comment.
This finding is real, but it is not introduced by this diff. The request job has always dispatched github.sha and its first parent: the line it names checks out the run's fixed github.sha, unchanged here. Before this PR, needs did not include build at all.
The exact tested merge cannot be dispatched as written. fresh-merge re-derives that merge commit locally on the runner and never pushes it. Plugins' scope job then runs git fetch origin <candidate> against GitHub, so a local-only merge sha would fail there.
Failing whenever build.outputs.main-sha != github.sha^1 would turn the gate red every time main moves between the PR event and the build, which happens several times an hour. That is the same false red this PR is removing.
What this PR does change: Plugins is now requested only after core's own suites ran green on that run. It does not claim the two trees are byte-identical. Closing the gap needs a reachable ref for the tested merge, for example pushing it to refs/core-candidate-merge/<key>. That is a separate change with its own write-permission decision, and I will file it into triage rather than widen this PR.
Comment on lines
+138
to
+139
| out["queued"] = sorted(j.get("name", "") for j in jobs if j.get("status") in ("queued", "waiting", "pending")) | ||
| out["running"] = sorted(j.get("name", "") for j in jobs if j.get("status") == "in_progress") |
Contributor
Author
There was a problem hiding this comment.
Fixed in 9db6ea2. Job names now pass through an allow-list (_PUBLIC_JOB) of core-candidate.yml's own fixed shapes:
Candidate|Control / … (leg N/M)(build once for N legs)- the four named jobs
Any other name is printed as <job>. A self-test case puts a portal-hosts (Secret.Customer.Test) job in the census and asserts that Secret never reaches the sentence while <job> and the allowed name do. I also checked it live against three current keys: all real names still pass.
Comment on lines
+176
to
+180
| if c["queued"] and not c["running"]: | ||
| return (f"no runner within {minutes} min ({behind}) — waiting for a runner: {_names(c['queued'])} — " | ||
| f"{c.get('pluginsQueuedRuns')} Plugins run(s) queued in total — {c['run']}") | ||
| return (f"the candidate was still running at the deadline ({len(c['running'])} job(s) running: {_names(c['running'])}; " | ||
| f"{len(c['queued'])} waiting for a runner; {behind}) — {c['run']}") |
Contributor
Author
There was a problem hiding this comment.
Fixed in 9db6ea2. A run that is still live (queued/requested/waiting/pending) with no job materialised now gets its own sentence: "no runner within 42 min (queued behind N …) — the candidate run itself never started: it is 'queued' with no job materialised yet (admission / concurrency)". A self-test case pins it, and it is no longer read as "still running".
| | request | `dotnet-test.yml` → `Dependent suites (request)` | **only once core's own build and tests are green on the candidate** (or the tree is a reused green) — otherwise it is NOT sent and goes red as "waiting for core's own suites"; then it proves the receiver exists on Plugins' default branch and sends `repository_dispatch core-candidate-suites` with the candidate (`github.sha`), its **first parent** as the base and a key unique to the run attempt — once for Plugins' default branch, once more at a declared counterpart's head | | ||
| | scope | Plugins `core-candidate.yml` → `scripts/core-candidate-scope.py` | the per-realm, affected-only selection above, printed as an audit table (selected realms with edges, then not-selected realms); `refused` when it cannot compute one; legs cut by the portal-host lane's measured weights | | ||
| | candidate arm | Plugins `core-candidate-arm.yml` | each leg built **from source** against the candidate (`-p:MeshWeaverRoot`, no image — the candidate is unpublished) and run; exit codes and dead hosts recorded exactly as the platform canary records them | | ||
| | candidate arm | Plugins `core-candidate-arm.yml` | the union of the legs' suites built **once, from source** against the candidate (`-p:MeshWeaverRoot`, no image — the candidate is unpublished), shipped to the legs through `scripts/dedupe-tree.py`, and run there with `--no-build`; a one-leg arm builds in its leg; exit codes and dead hosts recorded exactly as the platform canary records them | |
Contributor
Author
There was a problem hiding this comment.
Fixed in 9db6ea2. The three places that describe Plugins-side behaviour now say it lands with MeshWeaver.Plugins#3187:
- the candidate-arm row: "before it, each leg builds its own suites from source"
- point 2 (build once)
- point 4 (the reaper)
The page stays true whichever half lands first.
…us; name a run that never started; doc sequences with Plugins#3187 Review findings on #6331. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
Test Results 22 files ±0 22 suites ±0 50m 1s ⏱️ +17s Results for commit fde1915. ± Comparison against base commit 784c9d5. This pull request removes 40 and adds 16 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
…fter-core-and-census
…entical verdict, -35% dind runner-minutes Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The required
Dependent suites (MeshWeaver.Plugins)gate (#6313 + MeshWeaver.Plugins#3168) blocked core merges on runner capacity. #6320 went red on "No verdict in time" after its 42-minute wait (triage:rbuergi/Feedback/dependent-suites-gate-times-out-on-queued-candidate-20261008T1740Z).Measured 2026-10-08 (REST):
aks-silos-dindpool, beside about 30 queued Plugins-CI dind jobs. 31 dind jobs were running.MW_BUILD_QUEUE=off, so the CI queue's gate tier is bypassed and the order is GitHub's plain FIFO.What changes (core half)
Dependent suites (request)now needsbuildandtest. It dispatches only when both are green, or when the tree is a reused green.Consolidate test resultsrepeats that sentence. It is never a skip.await-dependent-verdict.py).core-candidate.ymlruns, usingpermission-actions: readon the same App token. That covers this candidate's job states, its queue position, the run keys ahead of it, and Plugins' total of queued runs.Dependent suites (request)to get a fresh key.pr-<N>concurrency group withcancel-in-progress. MeshWeaver.Plugins#3187 closes the remaining gap with a reaper.The 42-minute wait and every
timeout-minutesare unchanged.The other half
MeshWeaver.Plugins#3187: build once per arm. The union of the legs is built once and shipped through
dedupe-tree.py, and the legs run--no-build. It also adds the reaper. The two halves are independent and either can land first. There is noPairs-with:on purpose, because a counterpart would double the candidate runs this PR is trying to reduce.Checks run locally
await-dependent-verdict.py --self-test: 26/26.check-plugins-break-declaration.py --self-test,merge-queue-steward.py --self-test,check-workflow-shell.py,check-workflow-timeouts.py --root .(113 jobs, cap 45) andcheck-guard-step-masking.py --root ..actionlintondotnet-test.yml.MeshWeaver.Documentation.Test: Release-warnaserror0 warnings / 0 errors, and the ledger guards (46 tests, filtered) are green.Doc:
Doc/Architecture/CrossRepoPairGate§ "Runner capacity — requested after core's own suites, built once per arm, named when it waits".Mirror-sync: none — no i18n catalog change.
🤖 Generated with Claude Code