Skip to content

ci(#7777): use org-level roles for behaviour-test actors - #7778

Merged
ifireball merged 3 commits into
mainfrom
agent/7777-org-level-actor-roles
Sep 27, 2026
Merged

ifireball merged 3 commits into
mainfrom
agent/7777-org-level-actor-roles

Conversation

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor

Summary

Behaviour-test pool orgs now grant fstest-write and fstest-triage access through organization membership and all-repository roles instead of per-repo collaborator grants. That access survives repoEnsurer delete/recreate, so ensure no longer fails with invitation pending, access not granted.

Related Issue

#7777

Changes

  • hack/setup-new-e2e-org.sh idempotently invites the write/triage actors, assigns the all-repository write/triage roles, and verifies the assignments. It accepts either NN (halfsend-NN) or a full org name (halfsend for STAGE).
  • Runtime provisioning verifies org membership and the matching all-repository role once per org. It no longer calls AddCollaborator.
  • Missing membership or role fails with a pointer to the setup script. fstest-outsider must remain outside the organization.
  • forge.GitHubExtensions gains GetOrgMembership and ListUserOrganizationRoles.
  • Docs in docs/guides/dev/e2e-testing.md and docs/guides/dev/behaviour-testing.md describe the org-level actor model.

Testing

  • go test ./internal/forge/ ./internal/forge/github/ ./pkg/behaviourtest/drivers/install/
  • go vet on those packages
  • shellcheck on hack/setup-new-e2e-org.sh
  • Direct execution of local pre-commit hooks (gofmt, gitleaks, lychee, docs-link lint). pre-commit run itself could not fetch remote hook repos (HTTP 403).

Checklist

  • PR title follows Conventional Commits (ci(#7777): ...; fix(e2e) is forbidden)
  • Autonomous agent commit; DCO sign-off omitted by design

Closes #7777

Post-script verification

  • Branch is not main/master (agent/7777-org-level-actor-roles)
  • Secret scan passed (gitleaks — 1a803ca6d8950689e9a43cbecf1de88f18864523..HEAD)
  • PR body secret scan passed (gitleaks — no-git)

Pool repo delete/recreate dropped per-repo collaborator grants for
fstest-write and fstest-triage. Re-applying AddCollaborator then
created pending invitations and failed ensure with "invitation
pending, access not granted".

Grant those actors organization membership plus the all-repository
write/triage roles in hack/setup-new-e2e-org.sh instead. The
runtime ensurer now verifies that org-level access once per org
and no longer calls AddCollaborator. fstest-outsider remains
outside the organization.

Closes #7777
@fullsend-ai-coder
fullsend-ai-coder Bot requested a review from a team September 27, 2026 14:49
@fullsend-ai-coder fullsend-ai-coder Bot added the ready-for-review Triggers review agent dispatch label Sep 27, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 2:51 PM UTC · Completed 3:09 PM UTC

Commit: ef170e6 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $7.44

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Site preview

Preview: https://d9d2cf97-site.fullsend-ai.workers.dev

Commit: d47b4881dc30da1e320873c17f2c9016275f0248

@codecov

codecov Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@fullsend-ai-review fullsend-ai-review Bot added the risk/moderate PR risk: moderate label Sep 27, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Risk Assessment: moderate (2/5)

Details

Moderate risk: large change size across test drivers and forge abstractions with moderate churn and coupling is balanced by zero protected or security-sensitive path modifications, established bot author identity, and close alignment with linked issue #7777 (anchored to prior assessment; one incremental commit since then does not change Tier 1 signals).

Previous run

Risk Assessment: moderate (2/5)

Details

Moderate risk: large change size across test drivers and forge abstractions with moderate churn and coupling is balanced by zero protected or security-sensitive path modifications, established bot author identity, and close alignment with linked issue #7777.

Previous run (2)

Risk Assessment: moderate (2/5)

Details

Moderate risk: large change size across test drivers and forge abstractions with moderate churn is balanced by zero protected/security path changes, an established bot author, and close alignment with the linked issue's criteria.

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Looks good to me


Labels: PR modifies behaviour-test install driver code (pkg/behaviourtest/drivers/install/) and e2e org setup tooling (hack/setup-new-e2e-org.sh, docs/guides/dev/e2e-testing.md) but carries neither component/install nor component/e2e

Previous run

Looks good to me

Previous run (2)

Review

Findings

Medium

  • [missing-permission] pkg/behaviourtest/drivers/install/actor_grants.go:125 — verifyActorOrgAccess calls gh.ListUserOrganizationRoles, which hits GET /orgs/{org}/organization-roles/users/{username} using the suite's minted e2e GitHub App installation token. The e2e role's permission maps (internal/mintcore/github.go) and AppPermissions (internal/forge/github/types.go) grant organization_administration, members, organization_actions_variables, etc., but there is no organization_custom_roles grant, which is GitHub's documented permission for the organization-roles endpoints. If that's the required permission, every verifyActors call would 403, breaking the exact feature this PR adds — and the failure would surface as setupOrgHint ("run hack/setup-new-e2e-org.sh"), which cannot fix a missing App permission. Unit tests use FakeClient and cannot catch this.
    Remediation: Confirm the required App permission for the organization-roles endpoints. If it's organization_custom_roles, add it to the e2e role maps in internal/mintcore/github.go and to AppPermissions/the e2e manifest in internal/forge/github/types.go, then re-accept the updated permissions on each pool-org installation. Distinguish a permission-related 403 from the generic setupOrgHint message.

  • [fail-open] hack/setup-new-e2e-org.sh:269 — The outsider-exclusion checks in the setup script fail open and never set a non-zero exit code, unlike the write/triage assignment steps which exit 1 on failure: (1) outsider_state=$(gh api .../memberships/${TEST_OUTSIDER_USER} --jq '.state' 2>/dev/null || echo "none") treats any API/auth error as "not a member." (2) A real membership state (including pending) only prints a WARNING and the script continues. (3) outsider_roles=$(gh api .../organization-roles/users/${TEST_OUTSIDER_USER} ... 2>/dev/null || true) turns any API failure into an empty role list, so the script prints "OK: no all-repository role" without the check having actually run. A successful (exit 0) run of this script is therefore not reliable evidence that fstest-outsider is excluded.
    Remediation: Fail closed — exit 1 on any API error in these checks instead of defaulting to "none"/empty, and exit 1 (not just WARNING) when the outsider is found to be a member or holds an all-repository role. This is a security invariant for the test suite (Use organization-level roles for behaviour-test actors in ephemeral pool orgs #7777), not an advisory check.

  • [fail-open] pkg/behaviourtest/drivers/install/actor_grants.go:60 — outsiderLoginFromEnv returns "" both when TEST_ACTOR_OUTSIDER_PAT is unset and when it's set but GetAuthenticatedUser fails (it logs and returns "" either way). repoEnsurer.verifyActors only calls verifyOutsider if e.outsiderLogin != "", and login resolution happens once at ensurer construction. A transient lookup failure therefore silently disables the outsider-exclusion check for the whole run, while EnsureRepo proceeds and the org gets cached as verified. This is a fail-open path for the outsider invariant, in contrast to verifyOutsider/verifyActorOrgAccess, which correctly fail closed once a login is known.
    Remediation: If TEST_ACTOR_OUTSIDER_PAT is set but GetAuthenticatedUser fails, propagate an error instead of silently treating it as "no outsider to check."

Low

  • [logic-error] hack/setup-new-e2e-org.sh:292 — The outsider all-repository-role confirmation greps role names case-sensitively and doesn't route them through normalize_role_name, unlike resolve_all_repo_role_id's assignment path in the same file. A display-name variant (e.g. "All-repository write") would be missed. GitHub's documented role slug (all_repo_write/all_repo_triage) is still matched by the current pattern, so this is a robustness gap rather than a demonstrated miss today.
    Remediation: Pipe outsider_roles through normalize_role_name before grepping, matching all_repo_write, all_repo_triage, all_repository_write, all_repository_triage — the same forms resolve_all_repo_role_id already recognizes.

  • [permission-expansion] hack/setup-new-e2e-org.sh:130 — Argument handling now accepts any non-numeric string as a literal org name with no allowlist (needed to support the STAGE org halfsend). A mistyped or copy-pasted org argument would invite fstest-write/fstest-triage and assign all-repository write/triage roles in that org. Exploitability is low — this is an interactive, org-owner-run script with multiple confirmation prompts — but there is no validation gate against the documented pool/STAGE org names.
    Remediation: Validate ORG against ^halfsend(-[0-9]+)?$ (or an explicit allowlist of pool + STAGE org names) before proceeding, and abort with a clear error otherwise.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR

fullsend-ai-review[bot]

This comment was marked as outdated.

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-coder

fullsend-ai-coder Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 3:11 PM UTC · Completed 3:35 PM UTC

Commit: ef170e6 · View workflow run →

Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $7.57

Grant the e2e GitHub App the organization_custom_roles permission that
GetOrgMembership/ListUserOrganizationRoles require, and treat both a 403
and (as observed in this PR's own CI run) a 404 from the roles endpoint
as forge.ErrForbidden so the ensurer's error message points at the
missing App permission instead of the misleading setup-script hint.
Document the required operator permission-acceptance rollout in
docs/guides/dev/e2e-testing.md.

Make the outsider-exclusion checks fail closed instead of fail open:
hack/setup-new-e2e-org.sh now aborts on any API error while checking
fstest-outsider's membership/roles (rather than defaulting to "none"),
treats a pending membership as a hard failure rather than a WARNING,
and normalizes role names before matching so a display-name role
variant isn't missed. outsiderLoginFromEnv now propagates a login
resolution error instead of silently disabling the outsider check for
the ensurer's lifetime.

Restrict hack/setup-new-e2e-org.sh's org argument to the halfsend-NN /
halfsend allowlist so a mistyped argument can't grant test-actor access
in an unintended org.

Addresses #7778
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 1 (bot-triggered)

Granted the e2e App the organization_custom_roles permission the new org-role verification endpoints require (confirmed via this PR's failing CI run, which hit exactly this gap as a 404), gave the ensurer a permission-specific error message instead of pointing at the setup script, made the setup script's outsider-exclusion checks and outsider-login resolution fail closed instead of fail open, normalized role-name matching for the outsider check, and restricted the setup script's org argument to an allowlist. All five review findings were addressed as code fixes; no disagreements.

Fixed (5):

  1. missing-permission: verifyActorOrgAccess calls ListUserOrganizationRoles, but the e2e GitHub App role has no organization_custom_roles permission grant (internal/mintcore/github.go): Added organization_custom_roles (write/read) to the e2e role's canonical permission maps in internal/mintcore/github.go and to the e2e App manifest in internal/forge/github/types.go, and synced the mintcore embed copy used by the GCF deployment. Also made LiveClient.ListUserOrganizationRoles wrap both 403 and 404 responses as forge.ErrForbidden (this PR's own E2E CI run showed the failure surfaces as a 404, not 403), and updated verifyActorOrgAccess in pkg/behaviourtest/drivers/install/actor_grants.go to show a permission-specific hint instead of the misleading 'run hack/setup-new-e2e-org.sh' message in that case. Documented the required operator permission-acceptance rollout in docs/guides/dev/e2e-testing.md.
  2. fail-open: hack/setup-new-e2e-org.sh's outsider-exclusion checks treat API errors as "not a member"/empty and only WARN on a real membership or role (hack/setup-new-e2e-org.sh): Rewrote both outsider checks to fail closed: any gh api error other than a genuine 404 now aborts the script with exit 1, and a real membership (active or pending) or an assigned all-repository role now exits 1 instead of printing a WARNING and continuing.
  3. fail-open: outsiderLoginFromEnv silently disables the outsider check for the whole run when TEST_ACTOR_OUTSIDER_PAT is set but login resolution fails (pkg/behaviourtest/drivers/install/actor_grants.go): outsiderLoginFromEnv now returns an error instead of swallowing it, and newRepoEnsurer/newRepoEnsurerWithOpts (ensure.go) propagate that error to their callers (the three driver factory functions, which already return (Driver, error)) instead of constructing an ensurer that silently skips the outsider check.
  4. logic-error: the outsider all-repository-role confirmation greps role names case-sensitively without routing them through normalize_role_name (hack/setup-new-e2e-org.sh): The outsider role check now normalizes each returned role name the same way resolve_all_repo_role_id already does before comparing, so a display-name variant like "All-repository write" is caught.
  5. permission-expansion: the ORG argument accepts any non-numeric string as a literal org name with no allowlist (hack/setup-new-e2e-org.sh): Added a validation gate requiring ORG to match ^halfsend(-[0-9]+)?$ (the DEV pool naming convention or the STAGE org name) before the script proceeds, aborting with a clear error otherwise.

Tests: passed

CI inspections (13):

  1. E2E Tests / behaviour (pr-related, failure): Every behaviour scenario failed via an after-scenario repo-allocation hook error tracing to the new org-role verification call returning 404 Not Found, consistent with the e2e App installation lacking the organization_custom_roles permission this PR's new endpoint calls require. Added the organization_custom_roles permission to the e2e role/App definitions and treated 403/404 from the roles endpoint as a permission error with a corrected hint. This requires an operator to accept the updated e2e App permission on each pool-org installation (documented in docs/guides/dev/e2e-testing.md) before a rerun can pass; recommend rerunning this job after that manual step.
  2. E2E Tests / e2e (passing, success)
  3. E2E Tests / gate (passing, success)
  4. Functional Tests / functional-tests (passing, success)
  5. Functional Tests / gate (passing, success)
  6. CI / test (passing, success)
  7. CI / test-sandbox-darwin (passing, success)
  8. CI / commit-lint (passing, success)
  9. CI / web (passing, success)
  10. Vouch Check / vouch-gate (passing, success)
  11. Build Site / build (passing, success)
  12. codecov/patch (passing, success)
  13. DCO (passing, success)
Decision points
  • This PR's own E2E behaviour-test CI run (head SHA ef170e6) failed with the exact missing-permission symptom the review predicted, but as a 404 rather than a 403 (alternatives: Only special-case 403 as forge.ErrForbidden, matching the review's literal wording, Treat both 403 and 404 from ListUserOrganizationRoles as forge.ErrForbidden, since a member with zero roles legitimately returns 200 with an empty list, not 404; rationale: The CI log showed the actual runtime error was a 404 hitting the generic setupOrgHint branch, which would have left the misleading message finding unresolved in production. Extended the fix to cover both status codes and added a regression test for the 404 case.)

Next steps:

  • /fs-review — request a re-review of the changes
  • /fs-fix <your instruction> — run another fix pass with specific guidance
  • Push commits directly — review re-runs automatically on push
    Updated by fullsend fix agent

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 3:37 PM UTC · Completed 3:53 PM UTC

Commit: 4592b21 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $6.22

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Sep 27, 2026
@ifireball

Copy link
Copy Markdown
Member

/fs-fix

Please simplify this before merging if possible. The org-level actor-role fix should not require granting the e2e GitHub App organization_custom_roles or updating every pool-org installation. That permission is only needed because runtime calls ListUserOrganizationRoles.

Prefer having the setup script create and verify membership/roles, then verify effective access on the newly created repository via the collaborator-permission endpoint using role_name, or keep role verification setup-only. Retain the fail-closed outsider checks. Remove the App permission expansion if this works.

@fullsend-ai-coder

fullsend-ai-coder Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 5:00 PM UTC · Completed 5:15 PM UTC

Commit: 4592b21 · View workflow run →

Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $4.10

…oles

Per human review feedback on PR #7778: runtime org-level actor
verification called ListUserOrganizationRoles, which requires the e2e
GitHub App installation on every pool org to hold the
organization_custom_roles permission. Pool orgs that have not yet
accepted that permission update fail closed, as seen in this PR's own
CI (list organization roles returning 404 Not Found).

hack/setup-new-e2e-org.sh already creates and verifies the
all-repository write/triage roles using an org-admin-authenticated gh
session, independent of the e2e App's permissions. The runtime ensurer
now only re-verifies org membership via GetOrgMembership before each
lease, and trusts the setup script for the role assignment itself,
since the org-level role does not change with the per-repo
delete/recreate cycle the ensurer drives. This removes the need to
expand the e2e App's permissions or update every pool-org installation.

Drops ListUserOrganizationRoles and OrganizationRole from
forge.GitHubExtensions and its GitHub/fake implementations, the
organization_custom_roles App permission from the e2e role config
(mintcore and its GCF embed copy), and the now-unreachable
role-matching helpers and docs describing the permission rollout. The
fail-closed outsider check, which only uses GetOrgMembership, is
unaffected.

Addresses #7778
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 2 (human-triggered)

Removed the runtime dependency on GitHub's organization-roles API (ListUserOrganizationRoles) that required granting the e2e GitHub App organization_custom_roles and updating every pool-org installation. The ensurer now verifies only org membership at runtime; the all-repository write/triage role is verified solely by hack/setup-new-e2e-org.sh, which already runs with an org-admin gh session and needed no changes. Reverted the organization_custom_roles App permission grant (mintcore + GCF embed copy + types.go), updated docs, and trimmed the now-unreachable role-matching code and its tests. The fail-closed outsider membership check is untouched. This also fixes the PR's own failing 'behaviour' CI job, which failed with exactly this 404/organization_custom_roles error against pool org halfsend-07.

Fixed (4):

  1. Runtime org-level actor verification requires the e2e GitHub App to hold organization_custom_roles on every pool org (pkg/behaviourtest/drivers/install/actor_grants.go): Removed the ListUserOrganizationRoles call from verifyActorOrgAccess. The ensurer now only verifies active org membership (GetOrgMembership) before each lease; the all-repository write/triage role itself is verified only by hack/setup-new-e2e-org.sh, which already runs with an org-admin-authenticated gh session, independent of the e2e App's installed permissions. Removed the now-unreachable hasAllRepoRole/matchesAllRepoRole/normalizeRoleName helpers and the missingOrgRolesPermissionHint constant. Also removed ListUserOrganizationRoles/OrganizationRole from forge.GitHubExtensions (internal/forge/forge.go), its LiveClient implementation (internal/forge/github/github.go), and its FakeClient implementation/tests (internal/forge/fake.go, fake_test.go, github_test.go).
  2. Remove the App permission expansion now that the org-level role check no longer needs it (internal/forge/github/types.go): Reverted the OrganizationCustomRoles field on AppPermissions and its grant in AgentAppConfig's e2e role, and the matching canonicalRolePermissions entries in internal/mintcore/github.go and its GCF embed copy internal/dispatch/gcf/mintsrc/mintcore/github.go.embed (kept byte-for-byte in sync, verified via hack/lint-mint-embed-sync). Existing pool-org installations no longer need to accept a permission update for this change.
  3. Docs describe the organization_custom_roles requirement and a permission-rollout procedure that no longer applies (docs/guides/dev/e2e-testing.md): Removed the organization_custom_roles bullet from pool-org provisioning step 4, removed the 'Permission rollout for the e2e App' operator-notes section, and rewrote the Test actor permissions section to state that only org membership is re-verified at ensure time and the all-repository role is verified once by the setup script. Also reverted the organization_custom_roles mention in docs/guides/infrastructure/infrastructure-reference.md and updated the package doc comment in pkg/behaviourtest/drivers/install/doc.go to match.
  4. Test coverage must follow the runtime/behavioural change (pkg/behaviourtest/drivers/install/actor_grants_test.go): Removed tests exercising ListUserOrganizationRoles-based role verification and the role-name matching helper, and updated the remaining membership-only tests (including the stub client and the recreated-repo integration test) to no longer expect a role lookup. internal/forge/github/types_test.go, internal/forge/fake_test.go, internal/forge/github/github_test.go, and internal/mintcore/github_test.go were updated to drop the corresponding assertions.

Tests: passed

CI inspections (11):

  1. behaviour (pr-related, failure): All 25 failing scenarios trace to one root cause: allocating a pool repo on halfsend-07 failed when verifyActorOrgAccess called ListUserOrganizationRoles and GitHub returned 404, because the e2e App installation on that org lacks organization_custom_roles. This is the exact defect the human's /fs-fix instruction asked to eliminate. Removed the runtime ListUserOrganizationRoles call entirely; the ensurer now verifies only org membership, which does not require organization_custom_roles. Recommend rerunning the behaviour job to confirm.
  2. build (passing, success)
  3. test (passing, success)
  4. e2e (passing, success)
  5. functional-tests (passing, success)
  6. gate (passing, success)
  7. test-sandbox-darwin (passing, success)
  8. web (passing, success)
  9. commit-lint (passing, success)
  10. codecov/patch (passing, success)
  11. DCO (passing, success)
Decision points

Next steps:

  • /fs-review — request a re-review of the changes
  • /fs-fix <your instruction> — run another fix pass with specific guidance
  • Push commits directly — review re-runs automatically on push
    Updated by fullsend fix agent

@fullsend-ai-review

fullsend-ai-review Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 5:17 PM UTC · Completed 5:33 PM UTC

Commit: d47b488 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $4.52

@fullsend-ai-review
fullsend-ai-review Bot dismissed their stale review September 27, 2026 17:33

Superseded by updated review

@fullsend-ai-review fullsend-ai-review Bot added component/e2e End-to-end tests component/install CLI install and app setup labels Sep 27, 2026
@ifireball
ifireball enabled auto-merge September 27, 2026 17:47
@ifireball
ifireball added this pull request to the merge queue Sep 27, 2026
Merged via the queue into main with commit 877ba79 Sep 27, 2026
66 checks passed
@ifireball
ifireball deleted the agent/7777-org-level-actor-roles branch September 27, 2026 18:03
@fullsend-ai-retro

fullsend-ai-retro Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 6:05 PM UTC · Completed 6:13 PM UTC

Commit: d47b488 · View workflow run →

Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $1.63

@fullsend-ai-retro

Copy link
Copy Markdown

PR #7778 (fullsend-ai/fullsend) closed issue #7777, replacing per-repo AddCollaborator grants for e2e test actors (fstest-write/fstest-triage) with org-level all-repository roles so access survives pool-repo delete/recreate cycles. Full chain: triage ($0.75) → code ($12.37) → review #1 ($7.44, found a real fail-open bug in the outsider-exclusion check plus a missing-permission logic error) → bot-triggered fix #1 ($7.57, fixed both findings by granting the e2e GitHub App a new organization_custom_roles permission across every pool-org installation) → review #2 ($6.22, approved with no new findings) → CI's behaviour job kept failing → human (@ifireball) posted /fs-fix, explicitly rejecting the org-wide permission expansion as operationally heavy and redirecting to a simpler design → human-triggered fix #2 ($4.10, dropped the runtime role-verification dependency entirely, trusting the org-admin-run setup script instead) → review #3 ($4.52, approved) → human approval → merge. Total agent spend ~$43 across 7 runs, ~3h14m wall clock, two full rework cycles. The final code and docs are well-reasoned (see ensure.go/doc.go comments and updated docs/guides/dev/{e2e-testing,behaviour-testing}.md), and review #1's findings were genuinely good, security-relevant catches — this was not a case of a sloppy review. The gap is narrower: neither review #1 nor review #2 flagged that fix #1's remedy depended on live infrastructure state (a GitHub App installation permission grant) unverifiable from the diff, in a repo with a well-documented recurring history of exactly this failure class (open: #6701, #6702, #7488; closed lineage: #6025, #6064). It took human judgment, not agent review, to catch that the approach (not just the code) was wrong. Note: #6701 remains open and is unaffected by #7778 — different actor/mechanism (GitHub App installation permission graph vs. test-actor org roles) — it should not be assumed resolved by this merge. One proposal below on the review/fix agent gap; no duplicate issue was found covering it. Related-but-distinct existing issues for context: #2644 (deeper security analysis on authz code paths, production-focused) and #6891 (pre-flight effort heuristic for trivially-scoped PRs) — both adjacent in spirit but neither substantially overlaps this finding.

Proposals filed

This branch was successfully deployed

1 active and 1 inactive deployments
site-preview — d47b4881 Deployed Sep 27, 2026 by github-actions[bot]
dev — d47b4881 Deployed Sep 27, 2026 by fullsend-ai-coder[bot] via behaviour #14233
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/e2e End-to-end tests component/install CLI install and app setup ready-for-merge All reviewers approved — ready to merge ready-for-review Triggers review agent dispatch risk/moderate PR risk: moderate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use organization-level roles for behaviour-test actors in ephemeral pool orgs

1 participant