CI gates less than it appears to: one required check, and stacked PRs run none

[blooop]

Question

CI gates less than it looks like it does. Two independent holes, both found while
wiring the e2e suite into CI.

1. Only ci (py313) is a required check

main's ruleset requires exactly one context: ci (py313). Everything else —
prek, the other three interpreter jobs, and the new e2e job from
#114 — is advisory. A red e2e
job does not block a merge.

So the work that made the e2e suite trustworthy buys nothing at the gate until
this changes. That is the whole point of putting it in CI, and it is one ruleset
edit away.

Decide which contexts should be required. Not automatically all of them:
a required check that is flaky or slow blocks merges for reasons unrelated to the
change, which is its own failure mode. e2e measures 37s against the matrix's
42–52s, so on today's numbers it is cheap enough to require — but that is a
judgement, and it should be made rather than defaulted into.

2. A pull request onto a non-main base runs no CI at all

.github/workflows/ci.yml has on.pull_request.branches: [main]. That filter is
on the base branch, so a PR targeting anything else never triggers it.

This repo produces stacked PRs — there is a /stack workflow whose entire purpose
is chains of PRs where each targets its predecessor rather than main. Every link
in such a chain except the last currently merges with zero CI.

This is repo-wide and predates every ticket on this map; the e2e work only made it
visible. Note the interaction with hole 1: a required check that never runs is not
"pending", it is absent — and GitHub's behaviour there is worth confirming
rather than assuming, because "required but never triggered" is exactly the state
that silently blocks or silently passes depending on configuration.

Both are described in this repo's own terms in
PR #123's review.

This is a wayfinder:build ticket: go through /tdd.

Test-first, and mind what "test" means for a ruleset. The workflow change is
testable — a PR onto a non-main base must trigger CI, and demonstrating that
means actually opening one, not reading YAML. The ruleset change is a repo
setting, not code: that half needs human hands (gh api against branch
protection is possible but is an outward-facing configuration change to the
repository, not a code change). Park that half with a clear statement of what to
set and why, rather than flipping it.

Relation to the map. Not about developing in a devcontainer at all. It is here
because this map's work surfaced it and because
#114's deliverable is inert
without it — a CI job nobody is required to pass is a job that will rot.

[blooop]

breadcrumb: Claimed. Building the workflow half only; the ruleset half is parked for
human hands and will land as a ### handoff here.

What I will build. (1) The base-branch filter comes off the CI workflow's
pull_request trigger, so a pull request onto any base runs the whole workflow —
which is what a stacked chain needs. (2) A single aggregate gate job that
depends on every other job in that workflow and fails unless all of them
succeeded, so the ruleset has one stable context to require instead of a list
that goes stale every time a matrix leg or a job is added. The prose that
currently documents the gap — both in the workflow's own comments and in the
README's testing section — gets rewritten rather than left describing a state
that no longer holds.

Seam, as the ticket names it. Not a YAML-reading test: the ticket is explicit
that demonstrating this means opening a real pull request onto a non-main base.
So this branch's own pull request is the experiment. It opens onto a scratch base
branch sitting at main's tip, carrying the aggregate job but not the trigger
fix — the red is that pull request showing zero runs of the workflow, the newly
added job included. The trigger fix is then pushed to the same pull request and
the green is the same page listing the matrix, e2e and the gate. Only then is the
base retargeted to main for merge. A deliberate sabotage push proves the gate
goes red when a job it depends on does not succeed, and is taken back out; the
run logs outlive it.

I decided against a test that parses the workflow YAML. No YAML parser is in this
project's lockfile, adding one to assert on six lines of our own config is a
dependency for a guard weaker than the live demonstration, and the ticket already
ruled that seam out.

[blooop]

breadcrumb: Red and green, both on metal, on the same pull request — #128.

Red. The branch was pushed carrying the aggregate gate job but not the trigger
fix, and the pull request opened onto a base branch sitting at main's tip. GitHub
recorded two workflow runs for that head, and neither was the CI workflow: the
lint workflow (which has no base filter and was therefore never affected) and the
publish workflow's push run, which skips itself. Zero runs of CI — not a pending
tick, not a skipped one, none. So the matrix, the e2e suite that #114 exists to
add, and the gate job that had just been written all did not run on a pull request
that was one merge from a chain. That is hole 2 exactly as the ticket describes it,
observed rather than argued.

Green. One commit later — the base-branch filter removed from the trigger and
nothing else changed — the same pull request, still onto the same non-main base,
started a CI run. Worth recording why that works on the pull request that makes
the change: for pull_request events GitHub reads the workflow file from the merge
of head into base, so a trigger change takes effect for the very pull request that
introduces it. No second experiment was needed to confirm the fix, and none will be
needed by whoever next changes a trigger here.

One detail worth carrying beyond this ticket. A branches: filter means
different things on the two triggers in that file, which is most of why this went
unnoticed for so long. On push it matches the branch pushed to; on pull_request
it matches the base, never the head. So push: branches: [main] and
pull_request: branches: [main] read as one symmetrical pair and are not one:
the first is a narrow, correct statement about merges, the second silently
exempted an entire class of pull request. The trigger now carries a comment saying
so, because the next person to reach for a filter there will reach for the same
symmetry.

Next: sabotage the gate to show it goes red when a job it covers did not run, then
retarget this pull request at main.

[blooop]

handoff

Where we are. The workflow half is built and is PR #128 — CI now runs on every
pull request whatever its base, and there is a gate job depending on every other
job in that workflow which fails unless all of them succeeded. Red, green and a
sabotage run are all on metal and linked from the PR. The ruleset half is
outstanding: it is a repository setting, not a code change, and this ticket
should stay open until a human makes it.

Open thread — the one edit to make, and why.

Repository → Settings → Rules → Rulesets → main (ruleset 11900734, targeting
~DEFAULT_BRANCH) → Require status checks to pass.

Today that rule holds exactly one context: ci (py313). Everything else is
advisory — prek, the other three interpreter legs, the e2e job #114 added, and
the new gate. A red e2e does not block a merge.

Set it to exactly two, both from GitHub Actions:

• gate — replacing ci (py313).
• prek — added.

Why gate and not the five job names. A ruleset stores required checks as
literal strings, in a setting no pull request ever shows anyone. Listing
ci (py310)…ci (py313) and e2e there means the list has to be edited by hand,
out of band, every time a matrix leg or a job is added, renamed or dropped here —
and the stale direction is the dangerous one, because a required check whose job no
longer exists stops gating rather than turning anything red. gate is one name that
never changes, and what it covers lives in the workflow file where a pull request
adding a job also adds it to needs, under review. This is the same argument the
ticket makes about e2e being advisory, one level up: the decision should be
somewhere it can be seen.

Why prek has to be listed separately. needs: cannot cross workflow files, so
gate cannot cover a job in prek.yaml. And prek is not redundant with the
matrix: pixi run ci runs ruff-format, ruff, pylint, ty and coverage, while prek
additionally runs the yaml/toml/xml parse checks, private-key detection,
merge-conflict-marker detection, codespell, end-of-file fixing and the
shebang/executable checks. Different failures. It already runs on every pull
request — it has no base filter and so was never part of hole 2.

Why requiring e2e (through gate) is the right call on today's numbers. The
ticket rightly says this should be judged rather than defaulted into, because a
required check that is slow or flaky blocks merges for reasons unrelated to the
change. The measurement says it is neither: the e2e job finished in 37s against a
42–52s matrix on #123's runs and 73s wall-clock for the whole workflow on this
ticket's, and it runs beside the matrix rather than after it, so it adds nothing
to the wait. It is also, since #123, a suite that fails rather than skips when it
cannot do its work — which is the property that makes requiring it mean something.
If it ever does turn flaky, the fix is to drop it from gate's needs in a
reviewed pull request that has to say why, not to quietly stop requiring it in a
settings page.

What to leave advisory, deliberately. GitGuardian Security Checks posts on
every pull request but is a third-party integration this repository does not run.
A required context from an app that stops posting does not fail merges, it stops
them entirely, and there is nobody here to unblock that. publish-job is
Auto-publish's push-event job and is not a pull-request check at all.

Two adjacent settings, and my recommendation on each — but they are yours.

• Require branches to be up to date before merging (strict_required_status_checks_policy)
  is off. Leaving it off is right for now. Turning it on catches the case where
  two individually-green pull requests break main together, at the cost of a
  rebase-and-rerun before every single merge — which serialises the queue, and this
  repository deliberately produces stacked chains where that cost multiplies. The
  workflow runs on pushes to main too, so a semantic conflict shows up on main
  immediately rather than never. Worth revisiting the first time main actually
  breaks that way.
• The ruleset targets ~DEFAULT_BRANCH only, so the intermediate links of a stacked
  chain still are not enforced, even though as of Make CI run on every pull request, and give the ruleset one name to require #128 they now run. I would
  leave that alone: blocking a merge between two feature branches buys little when
  the chain's last link is gated, and the change Make CI run on every pull request, and give the ruleset one name to require #128 makes — red where there was
  previously nothing at all — is most of the value.

One claim in here is not verified on metal, and it should be labelled as such.
The ticket asks what GitHub does with a context that is required but never reported,
on the grounds that it "silently blocks or silently passes depending on
configuration". GitHub documents it as blocking — the check sits as Expected —
waiting for status to be reported and the merge button stays disabled. I have not
confirmed that here, because confirming it means making the very ruleset edit this
handoff is parking. Whoever makes the edit can confirm it in passing at no cost, and
it is worth doing. Note also that #128 removes the only route by which a run went
missing in this repository; the remaining route is a required name that no longer
matches any job, which is precisely what requiring gate is for.

First move on resume. Merge #128 first — gate has to have run on main at
least once before it is a context worth naming — then make the edit above, then
confirm on the next pull request that the required list reads gate + prek and
that a red e2e now blocks. Then this ticket closes.

[blooop]

breadcrumb: Gate proved by sabotage, and the build stage is at its gate.

The aggregate job was pushed with a one-line sabotage — the e2e job made
unconditionally skipped — and the run came back with four green matrix legs, e2e
skipped, and the gate failed, reading back the results it was handed and naming
the one that was not a success. Two things fall out of that beyond "the gate works".
First, if: always() on the gate is load-bearing rather than decoration: without it
the gate would have been skipped alongside the job it depends on and reported
nothing at all, which is the exact silence this ticket is about, reintroduced one
level up. Second, the gate insists on success rather than on the absence of
failure, so "did not run" is a failure there — chosen deliberately, because
failure is the easy case and this ticket is entirely about the other one.

There is a coupling worth knowing about, and the workflow now says so where someone
would trip over it: the e2e job's own comment explains how to make it nightly, and
that recipe now has a third step, because a job made conditional while still listed
in the gate's needs would block every merge. That is the strict semantics working
as intended rather than a defect, but it is only obvious once.

Sabotage reverted, pull request retargeted at main, scratch base branch deleted.
The run logs outlive the commit.

PR #128 — the workflow half in full. The ruleset half is posted above as a
### handoff and is genuinely outstanding: it is a repository setting, so this
ticket does not close on that pull request merging.

[blooop]

breadcrumb: Review posted on #128 — verdict request changes, one blocking finding: the pull request said Closes #126, which would auto-close this ticket on merge and take the parked ruleset half with it. Four Standards findings besides, two of them worth taking here rather than later — the gate passes green when it covers nothing, which is this pull request's own argument turned against its implementation; and the workflow has no concurrency group, a cost this change creates by making every link of a stack run a full matrix.

[blooop]

breadcrumb: Review's findings closed. The one blocking finding was about this ticket rather than about the code: the pull request said Closes #126, and GitHub's keyword auto-closes regardless of the qualifier written after it — so merging it would have shut the only tracker the parked ruleset edit has. It now says Part of, and says why in the first paragraph. This ticket stays open on merge, deliberately.

The substantive fix is that the gate passed green when it covered nothing. An empty dependency list made its check iterate zero times and exit success — the exact thing this ticket is about, rebuilt inside the mechanism built to prevent it, and set -u does not catch it because the variable is set and merely empty. It refuses an empty list now. A concurrency group went in alongside, cancelling superseded pull request runs but never a push to main, because the trigger change is what makes that cost real: rebasing the bottom of a stacked chain pushes every branch in it, and each link now runs a full matrix and an e2e suite.

[blooop]

breadcrumb: #128 is merged to main as a merge commit. This ticket stays open, which is the intended end state: the workflow half is landed and the ruleset half is a repository setting that only its owner can make. The ### handoff above is the whole of what is left.

What now gates, on main. Every pull request runs the interpreter matrix, the e2e suite and the aggregate gate, whatever branch it targets — so a stacked chain shows red where it used to show nothing at all. The gate fails on anything that is not a success, including a job that did not run and a list of jobs that is empty, and it is the one name a ruleset needs for that workflow. The lint workflow was never affected by the trigger bug and stays a second name, because dependencies do not reach across workflow files; it now carries a comment saying so, since renaming it after it is required would stop gating rather than fail.

What is still true and still costs something. None of that is enforced yet. main's ruleset still requires exactly ci (py313), so a red e2e or a red gate remains advisory and a merge goes through over the top of it. That is hole 1, untouched by design.

[blooop]

Both holes are closed on main.

Hole 2 — a PR onto a non-main base ran no CI. Fixed in #128: .github/workflows/ci.yml now has a bare pull_request: with no branches: filter, with the reasoning written into the file, plus a per-PR concurrency group so rebasing the bottom of a stack does not fire one full matrix per link.

Hole 1 — only ci (py313) was required. Settled the way the ticket asked, in the direction it asked: one stable name rather than a list. The gate job (needs: [ci, e2e], if: always()) fails unless every job it covers succeeded, and main's ruleset now requires exactly:

gate
prek

So e2e is required transitively, through gate's needs — adding a job means adding it there, in the same PR, rather than editing a repository setting nobody reviews.

Verified against the live repo, not against the PR description: gh api repos/blooop/devlaunch/rulesets returns those two contexts, and ci.yml on main carries the unfiltered trigger.