Repository navigation
fix(gcp): delete only a deployment's own custom roles (ALIEN-1203) - #839
Conversation
Deleting a deployment removed every custom role whose ID started with role_<namespace>_, which also matched the roles of any deployment in the same project whose prefix extends this one (acme and acme-prod). Prefixes sharing their first 18 characters also got the same role IDs. Cleanup now deletes a role only when its ID is in one of this deployment's namespaces and its description names this deployment's exact resource prefix. Runtime role names use the Terraform module's namespace (long prefixes keep 9 characters plus a SHA-256 suffix). Deployments whose GCP service accounts were created before the naming rule was recorded keep the truncated namespace their roles use.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Return stored-state decoding failures before naming or creating identities. Warn when cleanup cannot verify a candidate owner without a description. Keep unverified roles in place and preserve existing role naming choices.
|
@greptileai review Please review current head 264ef8d. It fixes both reported P2 findings: GCP identity state errors now propagate before naming or creation, and cleanup warns when a matching-namespace role has no description without deleting it. Both threads have replies linking the fixing commit and validation. All 794 infrastructure tests pass, including five additional regressions. |
…agates (#1035) Addresses ALIEN-1515. ## Problem A GCP setup failed with a non-retryable error after it reused soft-deleted custom roles: ``` Step failed for 'alien-storage': Failed to apply resource-scoped IAM policy to bucket '<prefix>-alien-storage' INVALID_INPUT: Cannot add new grants to deleted role (projects/<p>/roles/role_<prefix>_storage_trigger_management). HTTP 400 Non-retryable error for 'alien-storage', transitioning to failure immediately ``` ## Root cause Setup already treats a soft-deleted role correctly. Since #838, `ensure_gcp_custom_role` sees `deleted: true`, calls `roles.undelete`, and then patches the role to the desired title, description, stage and permissions. In the failing run it did this for all 15 roles. The failure came from a propagation delay after undelete: - 14 roles were undeleted and then bound 45 s or more later. All of those bindings succeeded. - `storage_trigger_management` was undeleted inside the bucket's own step and bound 1.5 s later. Cloud Storage still saw the role as deleted and returned 400. - `map_gcp_error` maps every 400 to `InvalidInput`, which is not retryable. The executor therefore failed the bucket instead of retrying the step. **Reproduced against a GCP test project (2026-10-14).** I created a throwaway role and bucket, deleted the role, and then undeleted it. `PUT /b/{bucket}/iam` granting the role then returned `400 "Cannot add new grants to deleted role (...)"` on every attempt for about 12-18 s after `roles.undelete` returned. After that it returned 200. I then removed the throwaway role and bucket. ## Fix `map_gcp_error` now classifies a 400 whose message starts with `Cannot add new grants to deleted role` as `RemoteServiceUnavailable`, which is retryable. This follows the existing AWS Lambda mapping for IAM eventual-consistency 400s. The executor's bounded step retry (exponential backoff, at most 10 attempts) then converges once the undelete propagates. All other 400s stay `InvalidInput` and non-retryable. The change sits at the client mapping, so the fix covers every IAM policy write that goes through `map_gcp_error`: buckets, Pub/Sub, Secret Manager, Artifact Registry and the project policy. Trade-off: a role that is really still deleted now fails after the retry budget runs out, not immediately. That case is setup-owned and not expected, because setup undeletes before it binds. ## Alternatives considered - **Sleep or poll after undelete in `ensure_gcp_custom_role`.** IAM gives no readiness signal for other services' policy checks: `roles.get` already reports `deleted: false`. A fixed sleep would be a guess, and it would slow every reuse. - **Map it to `RemoteResourceConflict`.** Several IAM writers treat a conflict as a stale-etag race and re-read in a local loop. That is the wrong meaning for this error. - **Stop deleting roles at teardown.** Rejected: the roles would leak and use up the project's 300-role quota (see #838 and #839). ## Tests - `storage::gcp::tests::bucket_policy_write_rejected_while_role_undelete_propagates_is_retryable` (controller): the first bucket policy write is rejected this way, the step fails as retryable while the bucket stays `Provisioning`, and the retried step reaches `Running`. **Fails before the fix:** on the tests-only commit 7af30cb it was the only failure in CI (4359 passed, 1 failed). It failed with exactly the e2e error chain: `CLOUD_PLATFORM_ERROR` (retryable: false) caused by `INVALID_INPUT: Cannot add new grants to deleted role`. - `gcp_request_utils::iam_policy_write_error_tests` (client, in-crate so CI runs it) drives a real `GcsClient` against an HTTP mock server that returns Cloud Storage's exact 400 body. It asserts that the error is retryable `REMOTE_SERVICE_UNAVAILABLE`. A second test asserts that an unrelated invalid-policy 400 stays non-retryable `INVALID_INPUT`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Fixes ALIEN-1203.
What changes for users
Several GCP deployments can now share one project safely:
acmealso deleted every role ofacme-prodin the same project, because cleanup deleted every role whose ID started withrole_acme_. The other deployment kept running without its roles, and the delete reported no error.Cleanup design
delete_gcp_custom_roleslists the project's custom roles. It deletes a role only when both of these are true:role_<namespace>_for one of this deployment's namespaces (the hashed one and the truncated one).Resource prefix: <this deployment's exact prefix>.Every role that the runtime generator or the Terraform module creates ends its description that way. The ID check alone cannot prove ownership, because
role_acme_also starts every role ofacme-prod.I chose listing plus this exact owner check over regenerating role IDs from the current stack. Regenerated IDs miss roles from permission sets that an earlier update removed, and roles whose generated suffix changed when a permission set's contents changed. Prefix listing used to clean those roles up. The owner check keeps them covered and is exact. If a role's description was edited by hand, the role is left in place. Leaving a role behind is safer than deleting another deployment's role.
Naming and compatibility
custom_role_namespace_for_prefixnow matches the generated Terraform module'sgcp_custom_role_prefixlocal. A prefix of up to 18 characters is used unchanged. A longer prefix becomes its first 9 characters plus_plus the first 8 hex characters ofsha256(prefix). Runtime setup and Terraform setup now name a deployment's roles the same way.ServiceAccountandRemoteStackManagementcontrollers now record acustomRoleNamingvalue (hashedLongPrefixortruncatedPrefix) when they create their service account.GcpCustomRoleNaming::for_deploymentreads this value from stored state:truncatedPrefix, the 18-character cut. Switching would bind its service accounts to roles that do not exist, and setup-owned roles cannot be recreated after setup.hashedLongPrefix. Terraform imports recordhashedLongPrefix, because the module names roles that way.alien-infragets the namespace fromResourcePermissionsHelper::gcp_permission_context, or fromgcp_custom_role_namespacein the vault controller. This replaces five copies of the same context-building code.local_resource_preplaceholder namespace explicitly, asCUSTOM_ROLE_NAMESPACE_PLACEHOLDER. Before, it relied on the placeholder prefix being truncated. The rendered modules do not change.Prefixes of 18 characters or fewer produce the same IDs as before. This includes every generated prefix (8 or 9 characters).
Other prefix-based GCP cleanups checked
remove_gcp_project_member_bindings) also matches role names by prefix. It only edits the bindings of this deployment's own service account member, so it cannot change another deployment's bindings. I did not change it.Follow-ups (not changed here):
RemoteStackManagementdelete flow deletes custom roles. A GCP deployment without that resource never deletes its custom roles.gcp_custom_role_prefixto manage roles outside the module also stays unknown to the runtime.Verification
Run on the parent of this commit with only the new tests added:
With this change:
The
alien-terraformsuite includes the existing GCP generator tests (63 of them, run withterraform validate). They pass without snapshot changes, so the rendered GCP modules are byte-for-byte unchanged.The new tests:
deleting_a_deployment_keeps_roles_of_a_deployment_whose_prefix_extends_itruns the realRemoteStackManagementdelete flow. The mocked IAM API lists roles across two pages foracmeandacme-prod. Each role has the ID and description the generator produces for that deployment, andacmealso has a role from a permission set no longer in its stack. The test asserts that exactlyacme's roles are deleted.deleting_a_deployment_set_up_before_hashed_names_removes_its_truncated_rolescovers a long-prefix deployment that has no recorded naming value. Its truncated-ID roles are deleted. The roles ofcustomer-acme-prodstay, even though that deployment's namespace is the same 18 characters.deleting_a_long_prefix_deployment_removes_its_hashed_rolesdeletes a hashed-ID deployment's roles and keeps those of its sibling prefix.gcp_custom_role_naming::testschecks the naming choice against stored controller JSON. One case uses the JSON of a service account created before this change, which has nocustomRoleNamingfield.gcp_custom_role_namespace_matches_terraformcompares the runtime namespace with values thatterraform consoleproduced from the module'sgcp_custom_role_prefixexpression. The cases cover 4, 9, 18, 19, 21 and 39 characters, including a hyphen at the 9-character cut:gcp_custom_role_namespace_override_keeps_existing_role_idschecks that the truncated namespace reproduces the exact role IDs the previous code generated for a 21-character prefix.I also listed custom roles in a real GCP project, read-only. The default
BASICview ofprojects.roles.listreturnsdescription, and Terraform-created roles end it withResource prefix: <prefix>.Review follow-up
Role naming now returns
RESOURCE_STATE_SERIALIZATION_FAILEDwhen either identity controller's stored JSON cannot be read. The error includes the affected resource ID and preserves the decoding error. All GCP identity states are checked before selecting a namespace, even when an earlier identity has a recorded rule. Both service-account creation flows determine naming before calling GCP.Cleanup logs a warning with the role name and resource prefix for a matching-namespace role that has no description. The role stays in place, and the exact description ownership requirement remains.
Five additional regressions cover unreadable service-account state, unknown management-identity naming, a valid recorded identity followed by an unreadable one, identities explicitly owned by another platform, and controller deletion with missing or edited role descriptions.
Validation of the follow-up passed all 794 infrastructure tests, including all five added regressions. Changed Rust files pass
rustfmt --check, andgit diff --checkis clean. The local build disabled debug symbols and incremental compilation to fit the available disk space:CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_INCREMENTAL=0 \ cargo nextest run -p alien-infra --all-features --lib # 794 tests run: 794 passed, 0 skippedFor the next maintainer
GcpCustomRoleNamingvariant. Do not change an existing one, because stored deployments depend on it.custom_role_description_names_prefix.