Skip to content

fix(ci): stop main merges cancelling each other's verification - #868

Merged
lilyshen0722 merged 1 commit into
mainfrom
fix/ci-main-run-cancellation
Aug 6, 2026
Merged

fix(ci): stop main merges cancelling each other's verification#868
lilyshen0722 merged 1 commit into
mainfrom
fix/ci-main-run-cancellation

Conversation

@lilyshen0722

@lilyshen0722 lilyshen0722 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

What

On a push event github.ref is the branch, so every commit merged to main shared one concurrency group and each merge cancelled the previous commit's still-running checks.

Found by watching my own merge (#866, 35d2fad9) get cancelled 2m21s in by #867.

The measurement

39 Tests runs on main, 2026-08-04T21:49Z → 08-06T02:52Z: 18 cancelled (46%). Not tonight's burst — it spans three days.

The part that makes it worth fixing rather than tolerating is that the loss is ordered by job depth, not random. Per-job across all 18:

Job Depth Survived cancellation
Chart Lint (Tier 1.5) no needs, ~8s 12 / 14
Test & Coverage no needs, ~5.5 min 3 / 14
Service Tests (Tier 1 — real DBs) needs: test, starts ~5.5 min in 0 / 14

(4 of the 18 were cancelled while still queued and ran no jobs at all.)

Service Tests is the ADR-009 Tier 1 real-DB tier. It never once completed in a cancelled run — it is structurally last, so it is always the first thing lost. The tier main was silently skipping is the one hardest to reproduce anywhere else, and nothing re-runs main.

The corollary, which cuts the other way

A cancelled run conclusion is not "nothing was verified." On my own merge, Chart Lint had already completed success at 02:52:12, 8s in, well before the cancel at 02:54:40. So on main the run-level conclusion misleads in both directions: cancelled hides real successes, and only a success tells you the deep tier actually ran.

Worth knowing independently of this fix — it means "the run was cancelled" is not grounds to assume a merge is unverified, and "the run was green" is the only statement that covers Tier 1.

The fix

Per-SHA fallback. PR behaviour is unchanged, because pull_request.number short-circuits before the fallback is ever read — a new push to a PR still supersedes its own older run, which is correct there since the old run describes code that is no longer the PR head. On main each commit is permanent history and gets the only verification it will ever get.

Applied to the three genuinely affected workflows:

Workflow Trigger Cancelled on main Action
tests.yml push, PR, dispatch 18 / 40 fixed
playwright.yml push, PR, dispatch 13 / 40 fixed
secret-scan.yml push, PR 9 / 40 fixed
pr-base-freshness.yml PR only 0 runs on main not fixed — unreachable

pr-base-freshness.yml carries the byte-identical string, but its trigger is pull_request-only so the fallback never fires. It gets a same-line note instead of a change, so the next person grepping github.ref across workflows finds the reason at the line grep returns rather than in this PR body.

I also nearly cited release-safety.yml's || github.run_id as an in-repo precedent that someone had already solved this. It isn't — that workflow is pull_request-only too, so its fallback is equally unreachable and the correct-looking shape there is a coincidence, not a decision.

Verification

  • actionlint 1.7.12 clean on all four files.
  • Positive control first: substituting github.no_such_field into the same expression IS flagged at that exact line/column, so actionlint demonstrably reads expressions inside a concurrency: block and the clean pass is meaningful rather than vacuous. The control also dumps the context type, which confirms sha: string is a real property.
  • 3 pre-existing actionlint findings on main (actions/checkout@v3 age) are unchanged — not introduced here, not fixed here.
  • YAML parses; the trailing comment on pr-base-freshness.yml stays out of the value (parsed group verified, comment absent), with a negative control confirming the parser rejects a genuinely malformed block.

What I could NOT verify before merge

The runtime semantics of the expression change. I can lint it and reason about it, but I cannot execute GitHub's expression engine locally — nothing here proves GitHub resolves github.sha per-commit in a concurrency group at dispatch time.

The observation that settles it is cheap and post-merge: the next time two commits land on main within ~7 minutes, both runs should reach a terminal state and neither should read cancelled. If instead they serialize or still cancel, this is wrong and should be reverted — it is CI scheduling only, so the blast radius of being wrong is bounded to how runs are queued.

One known consequence, stated rather than discovered later: during a merge burst, main runs now execute concurrently instead of one surviving. That is the intended trade (each commit gets verified), and it costs more CI minutes than cancelling did.

🤖 Generated with Claude Code


Peer review (@pod-architect, 52918) — two additions to the record

1. The branch filters, which I did not check and should have. I verified which workflows are push-triggered but not which branches they're push-triggered on. That gap matters: if any fired on all branches, per-SHA grouping would spawn a run per commit on every feature-branch push — a standing cost, not a burst cost. Re-verified at source, independently:

tests.yml         push branches: [main, v1.0.x]
playwright.yml    push branches: [main]
secret-scan.yml   push branches: [main, v1.0.x]

All bounded. The "more CI minutes during a burst" cost stated above is therefore exact, not optimistic.

2. The secret-scan.yml comment is more alarming than tonight's data. It reads "a secret scan that gets cancelled is the one result you least want to read as 'fine'." That's true in general and the 9/40 historical cancellations are real — but it did not bite tonight: every main commit since 01:00Z has a completed, successful Detect secrets, 10/10.

The depth-ordering mechanism in this PR predicts exactly that, and it's worth stating because it cuts the opposite way for this workflow: Detect secrets is a ~10s single job with no needs, so it lands inside the gap between merges. The same structure that makes Service Tests always the first casualty makes the secret scan nearly never one — its 9/40 is the tail where two merges landed within ~10 seconds of each other. Real exposure, rare rather than active.

Left the comment as written (it claims a failure mode, not a current state), but the calibration belongs in the record.

On a push event `github.ref` is the branch, so every commit merged to main
shared one concurrency group and each merge cancelled the previous commit's
still-running checks. Measured across 39 Tests runs on main (2026-08-04 →
08-06): 18 cancelled, 46%.

The loss is ordered by job depth, not random:

  Chart Lint       no needs,   ~8s               survived 12 of 14
  Test & Coverage  no needs,   ~5.5min           survived  3 of 14
  Service Tests    needs:test, starts ~5.5min in survived  0 of 14

Service Tests is the ADR-009 Tier 1 real-DB tier, and it never once completed
in a cancelled run — so the tier main silently stopped running is the one
hardest to reproduce anywhere else.

Fix is a per-SHA fallback. PR runs still supersede by number, because
pull_request.number short-circuits before the fallback is read.

Applies to the three push-triggered workflows (tests 18/40, playwright 13/40,
secret-scan 9/40 cancelled). pr-base-freshness.yml carries the identical
string but is pull_request-only, so its fallback is unreachable — 0 runs on
main confirms it; left as-is with a same-line note so the next grep for
`github.ref` finds the reason at the line rather than in this message.

Verified: actionlint 1.7.12 clean on all four, with a positive control
(a bogus github.* field IS flagged inside the concurrency block, so the
clean pass is meaningful); the 3 pre-existing findings on main are unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lilyshen0722
lilyshen0722 merged commit 2ece98f into main Aug 6, 2026
11 checks passed
@lilyshen0722
lilyshen0722 deleted the fix/ci-main-run-cancellation branch August 6, 2026 03:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant