Repository navigation
ci(#7777): use org-level roles for behaviour-test actors - #7778
Conversation
Pool repo delete/recreate dropped per-repo collaborator grants for fstest-write and fstest-triage. Re-applying AddCollaborator then created pending invitations and failed ensure with "invitation pending, access not granted". Grant those actors organization membership plus the all-repository write/triage roles in hack/setup-new-e2e-org.sh instead. The runtime ensurer now verifies that org-level access once per org and no longer calls AddCollaborator. fstest-outsider remains outside the organization. Closes #7777
|
🤖 Finished Review · ✅ Success · Started 2:51 PM UTC · Completed 3:09 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $7.44 |
Site previewPreview: https://d9d2cf97-site.fullsend-ai.workers.dev Commit: |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
Risk Assessment: moderate (2/5) DetailsModerate risk: large change size across test drivers and forge abstractions with moderate churn and coupling is balanced by zero protected or security-sensitive path modifications, established bot author identity, and close alignment with linked issue #7777 (anchored to prior assessment; one incremental commit since then does not change Tier 1 signals). Previous runRisk Assessment: moderate (2/5) DetailsModerate risk: large change size across test drivers and forge abstractions with moderate churn and coupling is balanced by zero protected or security-sensitive path modifications, established bot author identity, and close alignment with linked issue #7777. Previous run (2)Risk Assessment: moderate (2/5) DetailsModerate risk: large change size across test drivers and forge abstractions with moderate churn is balanced by zero protected/security path changes, an established bot author, and close alignment with the linked issue's criteria. |
|
Looks good to me Labels: PR modifies behaviour-test install driver code (pkg/behaviourtest/drivers/install/) and e2e org setup tooling (hack/setup-new-e2e-org.sh, docs/guides/dev/e2e-testing.md) but carries neither component/install nor component/e2e Previous runLooks good to me Previous run (2)ReviewFindingsMedium
Low
Next steps:
|
|
🤖 Finished Fix · ✅ Success · Started 3:11 PM UTC · Completed 3:35 PM UTC Commit: Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $7.57 |
Grant the e2e GitHub App the organization_custom_roles permission that GetOrgMembership/ListUserOrganizationRoles require, and treat both a 403 and (as observed in this PR's own CI run) a 404 from the roles endpoint as forge.ErrForbidden so the ensurer's error message points at the missing App permission instead of the misleading setup-script hint. Document the required operator permission-acceptance rollout in docs/guides/dev/e2e-testing.md. Make the outsider-exclusion checks fail closed instead of fail open: hack/setup-new-e2e-org.sh now aborts on any API error while checking fstest-outsider's membership/roles (rather than defaulting to "none"), treats a pending membership as a hard failure rather than a WARNING, and normalizes role names before matching so a display-name role variant isn't missed. outsiderLoginFromEnv now propagates a login resolution error instead of silently disabling the outsider check for the ensurer's lifetime. Restrict hack/setup-new-e2e-org.sh's org argument to the halfsend-NN / halfsend allowlist so a mistyped argument can't grant test-actor access in an unintended org. Addresses #7778
🔧 Fix agent — iteration 1 (bot-triggered)Granted the e2e App the organization_custom_roles permission the new org-role verification endpoints require (confirmed via this PR's failing CI run, which hit exactly this gap as a 404), gave the ensurer a permission-specific error message instead of pointing at the setup script, made the setup script's outsider-exclusion checks and outsider-login resolution fail closed instead of fail open, normalized role-name matching for the outsider check, and restricted the setup script's org argument to an allowlist. All five review findings were addressed as code fixes; no disagreements. Fixed (5):
Tests: passed CI inspections (13):
Decision points
Next steps:
|
|
🤖 Finished Review · ✅ Success · Started 3:37 PM UTC · Completed 3:53 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $6.22 |
Superseded by updated review
|
/fs-fix Please simplify this before merging if possible. The org-level actor-role fix should not require granting the e2e GitHub App Prefer having the setup script create and verify membership/roles, then verify effective access on the newly created repository via the collaborator-permission endpoint using |
|
🤖 Finished Fix · ✅ Success · Started 5:00 PM UTC · Completed 5:15 PM UTC Commit: Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $4.10 |
…oles Per human review feedback on PR #7778: runtime org-level actor verification called ListUserOrganizationRoles, which requires the e2e GitHub App installation on every pool org to hold the organization_custom_roles permission. Pool orgs that have not yet accepted that permission update fail closed, as seen in this PR's own CI (list organization roles returning 404 Not Found). hack/setup-new-e2e-org.sh already creates and verifies the all-repository write/triage roles using an org-admin-authenticated gh session, independent of the e2e App's permissions. The runtime ensurer now only re-verifies org membership via GetOrgMembership before each lease, and trusts the setup script for the role assignment itself, since the org-level role does not change with the per-repo delete/recreate cycle the ensurer drives. This removes the need to expand the e2e App's permissions or update every pool-org installation. Drops ListUserOrganizationRoles and OrganizationRole from forge.GitHubExtensions and its GitHub/fake implementations, the organization_custom_roles App permission from the e2e role config (mintcore and its GCF embed copy), and the now-unreachable role-matching helpers and docs describing the permission rollout. The fail-closed outsider check, which only uses GetOrgMembership, is unaffected. Addresses #7778
🔧 Fix agent — iteration 2 (human-triggered)Removed the runtime dependency on GitHub's organization-roles API (ListUserOrganizationRoles) that required granting the e2e GitHub App organization_custom_roles and updating every pool-org installation. The ensurer now verifies only org membership at runtime; the all-repository write/triage role is verified solely by hack/setup-new-e2e-org.sh, which already runs with an org-admin gh session and needed no changes. Reverted the organization_custom_roles App permission grant (mintcore + GCF embed copy + types.go), updated docs, and trimmed the now-unreachable role-matching code and its tests. The fail-closed outsider membership check is untouched. This also fixes the PR's own failing 'behaviour' CI job, which failed with exactly this 404/organization_custom_roles error against pool org halfsend-07. Fixed (4):
Tests: passed CI inspections (11):
Decision points
Next steps:
|
|
🤖 Finished Review · ✅ Success · Started 5:17 PM UTC · Completed 5:33 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $4.52 |
Superseded by updated review
|
🤖 Finished Retro · ✅ Success · Started 6:05 PM UTC · Completed 6:13 PM UTC Commit: Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $1.63 |
|
PR #7778 (fullsend-ai/fullsend) closed issue #7777, replacing per-repo Proposals filed |
Summary
Behaviour-test pool orgs now grant
fstest-writeandfstest-triageaccess through organization membership and all-repository roles instead of per-repo collaborator grants. That access survivesrepoEnsurerdelete/recreate, so ensure no longer fails withinvitation pending, access not granted.Related Issue
#7777
Changes
hack/setup-new-e2e-org.shidempotently invites the write/triage actors, assigns the all-repository write/triage roles, and verifies the assignments. It accepts eitherNN(halfsend-NN) or a full org name (halfsendfor STAGE).AddCollaborator.fstest-outsidermust remain outside the organization.forge.GitHubExtensionsgainsGetOrgMembershipandListUserOrganizationRoles.docs/guides/dev/e2e-testing.mdanddocs/guides/dev/behaviour-testing.mddescribe the org-level actor model.Testing
go test ./internal/forge/ ./internal/forge/github/ ./pkg/behaviourtest/drivers/install/go veton those packagesshellcheckonhack/setup-new-e2e-org.shpre-commit runitself could not fetch remote hook repos (HTTP 403).Checklist
ci(#7777): ...;fix(e2e)is forbidden)Closes #7777
Post-script verification
agent/7777-org-level-actor-roles)1a803ca6d8950689e9a43cbecf1de88f18864523..HEAD)