Skip to content

docs(research): correct check-safety methodology in #358 to first-attempt semantics - #2092

Merged
tucktuck101 merged 1 commit into
launchpadfrom
fix/issue-387-check-safety-methodology
Sep 6, 2026
Merged

tucktuck101 merged 1 commit into
launchpadfrom
fix/issue-387-check-safety-methodology

Conversation

@serina-mcfall

Copy link
Copy Markdown

Summary

  • launchpad/Research/358-who-can-require-a-check.md's "which checks are safe to require" evidence counted a check as safe whenever conclusion=="success" appeared anywhere in its check-run history, which silently absorbs reruns.
  • On PR chore: sync launchpad with upstream block/buzz main (113 commits) #216's head commit (43366aff...), the check named check failed on its first two attempts (03:31:37Z, 03:38:55Z) and only went green on a third rerun (03:59:27Z). The success-anywhere filter counted it as "always green" when it was actually red on the first attempt.
  • Re-derived using first-attempt conclusions (earliest started_at/id per check name per commit) — feasible from the exact same check-runs REST endpoint the document already used, no new API scope needed.
  • Under first-attempt semantics the safe set is 4 names, not 5: adr-boundary, Dead Token Reference Guard, Detect Changed Paths, scripts. check is demoted and named explicitly, with its two-failure/one-eventual-success history stated inline, distinguishing "never ran red on a first attempt" from "never ran red at all" per the issue's request.
  • All other findings in the document (admin/permissions, ruleset availability, the "30" denominator) are unchanged — this is scoped to the one evidence section the issue identified.

Closes #387

Verification

Test plan

🤖 Generated with Claude Code

…empt semantics

The "which checks are safe to require" section counted a check as safe
whenever conclusion=="success" appeared anywhere in its check-run
history, which silently absorbed reruns. On PR #216's commit, `check`
failed on its first two attempts and only went green on a third rerun,
but the success-anywhere filter counted it as always-green.

Re-derive using the first attempt (earliest started_at) per check name
per commit, which is feasible from the same check-runs REST endpoint
already used. Under first-attempt semantics the safe set is 4 names,
not 5 - `check` is demoted and named explicitly, distinguishing "never
ran red on a first attempt" from "never ran red at all" per the issue.

Closes #387

Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
@serina-mcfall serina-mcfall added the by:agent Filed or authored by an AI agent, not a human label Sep 4, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit cc3677a60d6e80f153bb74f82f7984ae67a7a718 against merge base aef93f2c2acfe9dfe66d22d33f5abb4ac12baa90.

Incomplete

This review is INCOMPLETE and must not be read as a full pass:

  • no dimension was actually reviewed: the pipeline ran the 'default_reviewer' stub reviewer, which reports every dimension clean without reading it (a real dimension reviewer is #116)

Containment

No containment findings.

Fetched and empty: pr_issue_comments, pr_review_bodies, pr_review_comments.

Automated containment covers the delimiter boundary and unambiguous injection tells only. It does not cover injection phrased as ordinary, unremarkable prose. The absence of a containment finding is not evidence that this pull request contains no injection attempt.

@tucktuck101 tucktuck101 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved — reproduced the correction independently against the live API.

The methodology bug is real. select(.conclusion=="success") over a commit's whole check-run history counts a name as green if any attempt succeeded, which silently absorbs reruns. Confirmed on PR #216's head 43366affa:

check  failure  2026-08-18T03:31:37Z  id=95586784357
check  failure  2026-08-18T03:38:55Z  id=95588021110
check  success  2026-08-18T03:59:27Z  id=95591477097

Two first-attempt failures, then green on a third run — counted as "always green" by the old filter.

Re-derived the corrected intersection myself, grouping by name and taking the earliest started_at/id per name per commit across both PR #308 and PR #216:

4 names:
  Dead Token Reference Guard
  Detect Changed Paths
  adr-boundary
  scripts
check in intersection? False

Exactly the corrected set, and check correctly demoted.

The distinction the rewrite draws is the one that matters operationally. "Eventually green" and "always green" are different guarantees, and only the second is safe to require — an eventually-green check still blocks the merge on its first red run until a human notices and reruns it. The doc states this explicitly, and is careful to note that the first-attempt property and the stricter never-red-at-all property merely happen to coincide on this evidence rather than being the same property in general. That caveat is correct and worth keeping.

Scope is right too: the correction is confined to the one evidence section, the admin/permissions and ruleset findings are untouched, and the summary line carries a dated correction pointer to #387 rather than silently rewriting history.

@tucktuck101
tucktuck101 merged commit 83dbed2 into launchpad Sep 6, 2026
25 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

by:agent Filed or authored by an AI agent, not a human

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: 'safe to require' check set counts eventually-green as always-green

2 participants