docs(research): correct check-safety methodology in #358 to first-attempt semantics - #2092
Conversation
…empt semantics The "which checks are safe to require" section counted a check as safe whenever conclusion=="success" appeared anywhere in its check-run history, which silently absorbed reruns. On PR #216's commit, `check` failed on its first two attempts and only went green on a third rerun, but the success-anywhere filter counted it as always-green. Re-derive using the first attempt (earliest started_at) per check name per commit, which is feasible from the same check-runs REST endpoint already used. Under first-attempt semantics the safe set is 4 names, not 5 - `check` is demoted and named explicitly, distinguishing "never ran red on a first attempt" from "never ran red at all" per the issue. Closes #387 Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
There was a problem hiding this comment.
Reviewed commit cc3677a60d6e80f153bb74f82f7984ae67a7a718 against merge base aef93f2c2acfe9dfe66d22d33f5abb4ac12baa90.
Incomplete
This review is INCOMPLETE and must not be read as a full pass:
- no dimension was actually reviewed: the pipeline ran the 'default_reviewer' stub reviewer, which reports every dimension clean without reading it (a real dimension reviewer is #116)
Containment
No containment findings.
Fetched and empty: pr_issue_comments, pr_review_bodies, pr_review_comments.
Automated containment covers the delimiter boundary and unambiguous injection tells only. It does not cover injection phrased as ordinary, unremarkable prose. The absence of a containment finding is not evidence that this pull request contains no injection attempt.
tucktuck101
left a comment
There was a problem hiding this comment.
Approved — reproduced the correction independently against the live API.
The methodology bug is real. select(.conclusion=="success") over a commit's whole check-run history counts a name as green if any attempt succeeded, which silently absorbs reruns. Confirmed on PR #216's head 43366affa:
check failure 2026-08-18T03:31:37Z id=95586784357
check failure 2026-08-18T03:38:55Z id=95588021110
check success 2026-08-18T03:59:27Z id=95591477097
Two first-attempt failures, then green on a third run — counted as "always green" by the old filter.
Re-derived the corrected intersection myself, grouping by name and taking the earliest started_at/id per name per commit across both PR #308 and PR #216:
4 names:
Dead Token Reference Guard
Detect Changed Paths
adr-boundary
scripts
check in intersection? False
Exactly the corrected set, and check correctly demoted.
The distinction the rewrite draws is the one that matters operationally. "Eventually green" and "always green" are different guarantees, and only the second is safe to require — an eventually-green check still blocks the merge on its first red run until a human notices and reruns it. The doc states this explicitly, and is careful to note that the first-attempt property and the stricter never-red-at-all property merely happen to coincide on this evidence rather than being the same property in general. That caveat is correct and worth keeping.
Scope is right too: the correction is confined to the one evidence section, the admin/permissions and ruleset findings are untouched, and the summary line carries a dated correction pointer to #387 rather than silently rewriting history.
Summary
launchpad/Research/358-who-can-require-a-check.md's "which checks are safe to require" evidence counted a check as safe wheneverconclusion=="success"appeared anywhere in its check-run history, which silently absorbs reruns.43366aff...), the check namedcheckfailed on its first two attempts (03:31:37Z, 03:38:55Z) and only went green on a third rerun (03:59:27Z). The success-anywhere filter counted it as "always green" when it was actually red on the first attempt.started_at/idper check name per commit) — feasible from the exact samecheck-runsREST endpoint the document already used, no new API scope needed.adr-boundary,Dead Token Reference Guard,Detect Changed Paths,scripts.checkis demoted and named explicitly, with its two-failure/one-eventual-success history stated inline, distinguishing "never ran red on a first attempt" from "never ran red at all" per the issue's request.Closes #387
Verification
checkappears in the failure set on PR chore: sync launchpad with upstream block/buzz main (113 commits) #216's commit.gh apifor both PR docs(decisions): record ADR-0021 and ADR-0022 — merge-based adoption, curation scoped to the contested surface #308 and PR chore: sync launchpad with upstream block/buzz main (113 commits) #216's head commits; confirmed the corrected intersection is exactly the 4 names above.checkis the only check name with anyfailureconclusion anywhere in its history (any attempt) on either commit — so the first-attempt property and the stricter "never ran red at all" property happen to coincide here (explained in the doc, with the caveat that they are not the same property in general).gh api/jqcommand now embedded in the rewritten section against the live repo; each reproduces the pasted output exactly.Test plan
gh api .../43366aff.../check-runs --jq '...failure...'reproduces["check"](the issue's own repro)launchpad/depends on the stale 5-name list (checked via grep across files referencing task: find out who can make a status check required here, and whether rulesets are available #358)🤖 Generated with Claude Code