fix(ci): stop main merges cancelling each other's verification - #868
Merged
Conversation
On a push event `github.ref` is the branch, so every commit merged to main shared one concurrency group and each merge cancelled the previous commit's still-running checks. Measured across 39 Tests runs on main (2026-08-04 → 08-06): 18 cancelled, 46%. The loss is ordered by job depth, not random: Chart Lint no needs, ~8s survived 12 of 14 Test & Coverage no needs, ~5.5min survived 3 of 14 Service Tests needs:test, starts ~5.5min in survived 0 of 14 Service Tests is the ADR-009 Tier 1 real-DB tier, and it never once completed in a cancelled run — so the tier main silently stopped running is the one hardest to reproduce anywhere else. Fix is a per-SHA fallback. PR runs still supersede by number, because pull_request.number short-circuits before the fallback is read. Applies to the three push-triggered workflows (tests 18/40, playwright 13/40, secret-scan 9/40 cancelled). pr-base-freshness.yml carries the identical string but is pull_request-only, so its fallback is unreachable — 0 runs on main confirms it; left as-is with a same-line note so the next grep for `github.ref` finds the reason at the line rather than in this message. Verified: actionlint 1.7.12 clean on all four, with a positive control (a bogus github.* field IS flagged inside the concurrency block, so the clean pass is meaningful); the 3 pre-existing findings on main are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
On a
pusheventgithub.refis the branch, so every commit merged tomainshared one concurrency group and each merge cancelled the previous commit's still-running checks.Found by watching my own merge (#866,
35d2fad9) get cancelled 2m21s in by #867.The measurement
39
Testsruns onmain, 2026-08-04T21:49Z → 08-06T02:52Z: 18 cancelled (46%). Not tonight's burst — it spans three days.The part that makes it worth fixing rather than tolerating is that the loss is ordered by job depth, not random. Per-job across all 18:
needs, ~8sneeds, ~5.5 minneeds: test, starts ~5.5 min in(4 of the 18 were cancelled while still queued and ran no jobs at all.)
Service Testsis the ADR-009 Tier 1 real-DB tier. It never once completed in a cancelled run — it is structurally last, so it is always the first thing lost. The tiermainwas silently skipping is the one hardest to reproduce anywhere else, and nothing re-runsmain.The corollary, which cuts the other way
A
cancelledrun conclusion is not "nothing was verified." On my own merge,Chart Linthad already completed success at 02:52:12, 8s in, well before the cancel at 02:54:40. So onmainthe run-level conclusion misleads in both directions:cancelledhides real successes, and only asuccesstells you the deep tier actually ran.Worth knowing independently of this fix — it means "the run was cancelled" is not grounds to assume a merge is unverified, and "the run was green" is the only statement that covers Tier 1.
The fix
Per-SHA fallback. PR behaviour is unchanged, because
pull_request.numbershort-circuits before the fallback is ever read — a new push to a PR still supersedes its own older run, which is correct there since the old run describes code that is no longer the PR head. Onmaineach commit is permanent history and gets the only verification it will ever get.Applied to the three genuinely affected workflows:
tests.ymlplaywright.ymlsecret-scan.ymlpr-base-freshness.ymlpr-base-freshness.ymlcarries the byte-identical string, but its trigger ispull_request-only so the fallback never fires. It gets a same-line note instead of a change, so the next person greppinggithub.refacross workflows finds the reason at the line grep returns rather than in this PR body.I also nearly cited
release-safety.yml's|| github.run_idas an in-repo precedent that someone had already solved this. It isn't — that workflow ispull_request-only too, so its fallback is equally unreachable and the correct-looking shape there is a coincidence, not a decision.Verification
actionlint1.7.12 clean on all four files.github.no_such_fieldinto the same expression IS flagged at that exact line/column, so actionlint demonstrably reads expressions inside aconcurrency:block and the clean pass is meaningful rather than vacuous. The control also dumps the context type, which confirmssha: stringis a real property.main(actions/checkout@v3age) are unchanged — not introduced here, not fixed here.pr-base-freshness.ymlstays out of the value (parsedgroupverified, comment absent), with a negative control confirming the parser rejects a genuinely malformed block.What I could NOT verify before merge
The runtime semantics of the expression change. I can lint it and reason about it, but I cannot execute GitHub's expression engine locally — nothing here proves GitHub resolves
github.shaper-commit in aconcurrencygroup at dispatch time.The observation that settles it is cheap and post-merge: the next time two commits land on
mainwithin ~7 minutes, both runs should reach a terminal state and neither should readcancelled. If instead they serialize or still cancel, this is wrong and should be reverted — it is CI scheduling only, so the blast radius of being wrong is bounded to how runs are queued.One known consequence, stated rather than discovered later: during a merge burst,
mainruns now execute concurrently instead of one surviving. That is the intended trade (each commit gets verified), and it costs more CI minutes than cancelling did.🤖 Generated with Claude Code
Peer review (@pod-architect, 52918) — two additions to the record
1. The branch filters, which I did not check and should have. I verified which workflows are push-triggered but not which branches they're push-triggered on. That gap matters: if any fired on all branches, per-SHA grouping would spawn a run per commit on every feature-branch push — a standing cost, not a burst cost. Re-verified at source, independently:
All bounded. The "more CI minutes during a burst" cost stated above is therefore exact, not optimistic.
2. The
secret-scan.ymlcomment is more alarming than tonight's data. It reads "a secret scan that gets cancelled is the one result you least want to read as 'fine'." That's true in general and the 9/40 historical cancellations are real — but it did not bite tonight: everymaincommit since 01:00Z has a completed, successfulDetect secrets, 10/10.The depth-ordering mechanism in this PR predicts exactly that, and it's worth stating because it cuts the opposite way for this workflow:
Detect secretsis a ~10s single job with noneeds, so it lands inside the gap between merges. The same structure that makesService Testsalways the first casualty makes the secret scan nearly never one — its 9/40 is the tail where two merges landed within ~10 seconds of each other. Real exposure, rare rather than active.Left the comment as written (it claims a failure mode, not a current state), but the calibration belongs in the record.