Skip to content

fix(gcp): delete only a deployment's own custom roles (ALIEN-1203) - #839

Merged
lilienblum merged 3 commits into
mainfrom
dan/alien-1203-gcp-role-cleanup-exact-ids
Oct 3, 2026
Merged

lilienblum merged 3 commits into
mainfrom
dan/alien-1203-gcp-role-cleanup-exact-ids

Conversation

@lilienblum

@lilienblum lilienblum commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Fixes ALIEN-1203.

What changes for users

Several GCP deployments can now share one project safely:

  • Deleting a deployment removes only its own custom roles. Before this change, deleting acme also deleted every role of acme-prod in the same project, because cleanup deleted every role whose ID started with role_acme_. The other deployment kept running without its roles, and the delete reported no error.
  • Two resource prefixes that share their first 18 characters now get different role IDs. Before, both deployments used the same roles, so one overwrote the other's permissions, and deleting either one removed both sets.

Cleanup design

delete_gcp_custom_roles lists the project's custom roles. It deletes a role only when both of these are true:

  1. The role ID starts with role_<namespace>_ for one of this deployment's namespaces (the hashed one and the truncated one).
  2. The role description ends with Resource prefix: <this deployment's exact prefix>.

Every role that the runtime generator or the Terraform module creates ends its description that way. The ID check alone cannot prove ownership, because role_acme_ also starts every role of acme-prod.

I chose listing plus this exact owner check over regenerating role IDs from the current stack. Regenerated IDs miss roles from permission sets that an earlier update removed, and roles whose generated suffix changed when a permission set's contents changed. Prefix listing used to clean those roles up. The owner check keeps them covered and is exact. If a role's description was edited by hand, the role is left in place. Leaving a role behind is safer than deleting another deployment's role.

Naming and compatibility

  • custom_role_namespace_for_prefix now matches the generated Terraform module's gcp_custom_role_prefix local. A prefix of up to 18 characters is used unchanged. A longer prefix becomes its first 9 characters plus _ plus the first 8 hex characters of sha256(prefix). Runtime setup and Terraform setup now name a deployment's roles the same way.
  • Deployments created before this change with prefixes longer than 18 characters keep their existing role IDs. The GCP ServiceAccount and RemoteStackManagement controllers now record a customRoleNaming value (hashedLongPrefix or truncatedPrefix) when they create their service account. GcpCustomRoleNaming::for_deployment reads this value from stored state:
    • If any service account recorded a value, every controller in the deployment uses it.
    • If a GCP service account exists but has no recorded value, it was created before this change, so the deployment keeps truncatedPrefix, the 18-character cut. Switching would bind its service accounts to roles that do not exist, and setup-owned roles cannot be recreated after setup.
    • Otherwise, the deployment is new and uses hashedLongPrefix. Terraform imports record hashedLongPrefix, because the module names roles that way.
  • Every GCP permission context in alien-infra gets the namespace from ResourcePermissionsHelper::gcp_permission_context, or from gcp_custom_role_namespace in the vault controller. This replaces five copies of the same context-building code.
  • The Terraform emitter now passes its local_resource_pre placeholder namespace explicitly, as CUSTOM_ROLE_NAMESPACE_PLACEHOLDER. Before, it relied on the placeholder prefix being truncated. The rendered modules do not change.

Prefixes of 18 characters or fewer produce the same IDs as before. This includes every generated prefix (8 or 9 characters).

Other prefix-based GCP cleanups checked

  • Custom roles were the only GCP cleanup that lists project resources and deletes them by name prefix.
  • Project IAM binding reconciliation (remove_gcp_project_member_bindings) also matches role names by prefix. It only edits the bindings of this deployment's own service account member, so it cannot change another deployment's bindings. I did not change it.
  • Sandbox template reaping lists templates under this deployment's own engine only. GCS notifications are matched exactly.

Follow-ups (not changed here):

  • Only the RemoteStackManagement delete flow deletes custom roles. A GCP deployment without that resource never deletes its custom roles.
  • Deployments imported from Terraform before this change, with prefixes longer than 18 characters, record no naming value, so they keep the truncated IDs at runtime. Terraform created hashed IDs for them, so they still disagree, as they did before this change. A Terraform module that sets gcp_custom_role_prefix to manage roles outside the module also stays unknown to the runtime.

Verification

Run on the parent of this commit with only the new tests added:

$ cargo nextest run -p alien-infra --all-features --lib -E 'test(deleting_a_deployment_keeps_roles)'
FAIL remote_stack_management::gcp::tests::deleting_a_deployment_keeps_roles_of_a_deployment_whose_prefix_extends_it
  left (deleted):  role_acme_manage_cloud_run_services, role_acme_prod_manage_cloud_run_services,
                   role_acme_prod_read_cloud_storage_objects, role_acme_prod_sign_cloud_storage_download_urls,
                   role_acme_read_cloud_storage_objects, role_acme_sign_cloud_storage_download_urls
  right (expected): the three role_acme_* roles only

$ cargo nextest run -p alien-permissions --test gcp_runtime -E 'test(gcp_custom_role_ids_differ)'
FAIL gcp_custom_role_ids_differ_for_prefixes_sharing_their_first_18_characters
  'build/management' gives both deployments the role ID 'role_customer_acme_prod_build_management'

With this change:

$ cargo nextest run -p alien-infra --all-features --lib -E 'test(/remote_stack_management::gcp::tests|gcp_custom_role_naming/)'
8 tests run: 8 passed
$ cargo nextest run -p alien-permissions --all-features
191 tests run: 191 passed
$ cargo nextest run -p alien-terraform --all-features --test-threads 2
253 tests run: 253 passed
$ cargo nextest run -p alien-infra --all-features --lib
789 tests run: 789 passed

The alien-terraform suite includes the existing GCP generator tests (63 of them, run with terraform validate). They pass without snapshot changes, so the rendered GCP modules are byte-for-byte unchanged.

The new tests:

  • deleting_a_deployment_keeps_roles_of_a_deployment_whose_prefix_extends_it runs the real RemoteStackManagement delete flow. The mocked IAM API lists roles across two pages for acme and acme-prod. Each role has the ID and description the generator produces for that deployment, and acme also has a role from a permission set no longer in its stack. The test asserts that exactly acme's roles are deleted.

  • deleting_a_deployment_set_up_before_hashed_names_removes_its_truncated_roles covers a long-prefix deployment that has no recorded naming value. Its truncated-ID roles are deleted. The roles of customer-acme-prod stay, even though that deployment's namespace is the same 18 characters.

  • deleting_a_long_prefix_deployment_removes_its_hashed_roles deletes a hashed-ID deployment's roles and keeps those of its sibling prefix.

  • gcp_custom_role_naming::tests checks the naming choice against stored controller JSON. One case uses the JSON of a service account created before this change, which has no customRoleNaming field.

  • gcp_custom_role_namespace_matches_terraform compares the runtime namespace with values that terraform console produced from the module's gcp_custom_role_prefix expression. The cases cover 4, 9, 18, 19, 21 and 39 characters, including a hyphen at the 9-character cut:

    echo local.gcp_custom_role_prefix | terraform console -var resource_prefix=customer-acme-prod-eu
    "customer__2e6bb8cf"
    
  • gcp_custom_role_namespace_override_keeps_existing_role_ids checks that the truncated namespace reproduces the exact role IDs the previous code generated for a 21-character prefix.

I also listed custom roles in a real GCP project, read-only. The default BASIC view of projects.roles.list returns description, and Terraform-created roles end it with Resource prefix: <prefix>.

Review follow-up

Role naming now returns RESOURCE_STATE_SERIALIZATION_FAILED when either identity controller's stored JSON cannot be read. The error includes the affected resource ID and preserves the decoding error. All GCP identity states are checked before selecting a namespace, even when an earlier identity has a recorded rule. Both service-account creation flows determine naming before calling GCP.

Cleanup logs a warning with the role name and resource prefix for a matching-namespace role that has no description. The role stays in place, and the exact description ownership requirement remains.

Five additional regressions cover unreadable service-account state, unknown management-identity naming, a valid recorded identity followed by an unreadable one, identities explicitly owned by another platform, and controller deletion with missing or edited role descriptions.

Validation of the follow-up passed all 794 infrastructure tests, including all five added regressions. Changed Rust files pass rustfmt --check, and git diff --check is clean. The local build disabled debug symbols and incremental compilation to fit the available disk space:

CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_INCREMENTAL=0 \
cargo nextest run -p alien-infra --all-features --lib
# 794 tests run: 794 passed, 0 skipped

For the next maintainer

  • To change how role IDs are derived, add a GcpCustomRoleNaming variant. Do not change an existing one, because stored deployments depend on it.
  • Never select roles for deletion by ID prefix alone. Use custom_role_description_names_prefix.

Deleting a deployment removed every custom role whose ID started with
role_<namespace>_, which also matched the roles of any deployment in the
same project whose prefix extends this one (acme and acme-prod). Prefixes
sharing their first 18 characters also got the same role IDs.

Cleanup now deletes a role only when its ID is in one of this
deployment's namespaces and its description names this deployment's
exact resource prefix. Runtime role names use the Terraform module's
namespace (long prefixes keep 9 characters plus a SHA-256 suffix).
Deployments whose GCP service accounts were created before the naming
rule was recorded keep the truncated namespace their roles use.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-03T14:00:06.786877Z a8fd785 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@greptile-apps

greptile-apps Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Critical risk] Changes how GCP custom roles are identified and deleted during cleanup.

The PR appears safe to merge; both previously reported issues are fixed and no new actionable issue was established.

Summary

The PR makes GCP custom-role IDs distinct for long deployment prefixes and limits deletion to roles whose descriptions identify the deployment. The follow-up propagates unreadable identity-state errors before service-account creation and warns when a matching role has no description. Both previous findings are fixed.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Select deployment role naming from identity state] --> B{Identity state readable?}
  B -- No --> C[Return resource-state error]
  B -- Yes --> D[Use recorded, legacy, or hashed namespace]
  D --> E[List project custom roles]
  E --> F{ID matches either deployment namespace?}
  F -- No --> G[Keep role]
  F -- Yes --> H{Description ends with exact resource prefix?}
  H -- Yes --> I[Delete role]
  H -- No --> G
Loading

Reviews (2) · Last reviewed commit: "fix(gcp): surface role naming and cleanu..."

Comment thread crates/alien-infra/src/core/gcp_custom_role_naming.rs Outdated
Comment thread crates/alien-infra/src/core/resource_permissions_helper.rs
Return stored-state decoding failures before naming or creating identities.
Warn when cleanup cannot verify a candidate owner without a description.
Keep unverified roles in place and preserve existing role naming choices.
@lilienblum

Copy link
Copy Markdown
Contributor Author

@greptileai review

Please review current head 264ef8d. It fixes both reported P2 findings: GCP identity state errors now propagate before naming or creation, and cleanup warns when a matching-namespace role has no description without deleting it. Both threads have replies linking the fixing commit and validation. All 794 infrastructure tests pass, including five additional regressions.

@lilienblum
lilienblum merged commit f2f2a80 into main Oct 3, 2026
29 checks passed
@lilienblum
lilienblum deleted the dan/alien-1203-gcp-role-cleanup-exact-ids branch October 3, 2026 23:25
alongubkin added a commit that referenced this pull request Oct 14, 2026
…agates (#1035)

Addresses ALIEN-1515.

## Problem

A GCP setup failed with a non-retryable error after it reused
soft-deleted custom roles:

```
Step failed for 'alien-storage': Failed to apply resource-scoped IAM policy to bucket '<prefix>-alien-storage'
INVALID_INPUT: Cannot add new grants to deleted role (projects/<p>/roles/role_<prefix>_storage_trigger_management). HTTP 400
Non-retryable error for 'alien-storage', transitioning to failure immediately
```

## Root cause

Setup already treats a soft-deleted role correctly. Since #838,
`ensure_gcp_custom_role` sees `deleted: true`, calls `roles.undelete`,
and then patches the role to the desired title, description, stage and
permissions. In the failing run it did this for all 15 roles. The
failure came from a propagation delay after undelete:

- 14 roles were undeleted and then bound 45 s or more later. All of
those bindings succeeded.
- `storage_trigger_management` was undeleted inside the bucket's own
step and bound 1.5 s later. Cloud Storage still saw the role as deleted
and returned 400.
- `map_gcp_error` maps every 400 to `InvalidInput`, which is not
retryable. The executor therefore failed the bucket instead of retrying
the step.

**Reproduced against a GCP test project (2026-10-14).** I created a
throwaway role and bucket, deleted the role, and then undeleted it. `PUT
/b/{bucket}/iam` granting the role then returned `400 "Cannot add new
grants to deleted role (...)"` on every attempt for about 12-18 s after
`roles.undelete` returned. After that it returned 200. I then removed
the throwaway role and bucket.

## Fix

`map_gcp_error` now classifies a 400 whose message starts with `Cannot
add new grants to deleted role` as `RemoteServiceUnavailable`, which is
retryable. This follows the existing AWS Lambda mapping for IAM
eventual-consistency 400s. The executor's bounded step retry
(exponential backoff, at most 10 attempts) then converges once the
undelete propagates. All other 400s stay `InvalidInput` and
non-retryable.

The change sits at the client mapping, so the fix covers every IAM
policy write that goes through `map_gcp_error`: buckets, Pub/Sub, Secret
Manager, Artifact Registry and the project policy.

Trade-off: a role that is really still deleted now fails after the retry
budget runs out, not immediately. That case is setup-owned and not
expected, because setup undeletes before it binds.

## Alternatives considered

- **Sleep or poll after undelete in `ensure_gcp_custom_role`.** IAM
gives no readiness signal for other services' policy checks: `roles.get`
already reports `deleted: false`. A fixed sleep would be a guess, and it
would slow every reuse.
- **Map it to `RemoteResourceConflict`.** Several IAM writers treat a
conflict as a stale-etag race and re-read in a local loop. That is the
wrong meaning for this error.
- **Stop deleting roles at teardown.** Rejected: the roles would leak
and use up the project's 300-role quota (see #838 and #839).

## Tests

-
`storage::gcp::tests::bucket_policy_write_rejected_while_role_undelete_propagates_is_retryable`
(controller): the first bucket policy write is rejected this way, the
step fails as retryable while the bucket stays `Provisioning`, and the
retried step reaches `Running`. **Fails before the fix:** on the
tests-only commit 7af30cb it was the only failure in CI (4359 passed,
1 failed). It failed with exactly the e2e error chain:
`CLOUD_PLATFORM_ERROR` (retryable: false) caused by `INVALID_INPUT:
Cannot add new grants to deleted role`.
- `gcp_request_utils::iam_policy_write_error_tests` (client, in-crate so
CI runs it) drives a real `GcsClient` against an HTTP mock server that
returns Cloud Storage's exact 400 body. It asserts that the error is
retryable `REMOTE_SERVICE_UNAVAILABLE`. A second test asserts that an
unrelated invalid-policy 400 stays non-retryable `INVALID_INPUT`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant