Skip to content

fix: reconcile vault grants across cloud updates - #932

Merged
alongubkin merged 19 commits into
mainfrom
alon/alien-1305-cross-cloud-vault-updates
Oct 8, 2026
Merged

alongubkin merged 19 commits into
mainfrom
alon/alien-1305-cross-cloud-vault-updates

Conversation

@alongubkin

@alongubkin alongubkin commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Existing vaults could miss newly added grants during updates. Azure and GCP update handlers now reconcile their grants before reporting success. AWS, Azure, and GCP vaults also detect permission-only changes, so setup schedules reconciliation even when resource config and dependencies are unchanged.

Each controller records the resolved-grant revision only after success and stops scheduling updates once it converges. Setup authority checks remain in place. GCP reconciliation preserves other vaults' conditional grants and shared unconditional bindings. Retries compare binding and member sets, so a lost response can adopt a committed policy even if another writer reorders it. New and known-empty GCP profiles can skip namespace IAM work without project metadata; legacy checkpoints with unknown grant history must have metadata before setup can report cleanup success.

State compatibility: adds one optional permissionsRevision field to the three vault controllers with serde(default). Older checkpoints load without it; direct setup reconciles them once. Importers initialize it to None. No state variants are added or renamed.

Validation: cargo test -p alien-infra --all-features --lib passed in an isolated Linux sandbox: 947 passed, 0 failed, 1 existing test ignored. Regressions cover new consumers, permission-only changes, convergence, legacy checkpoints, missing saved identifiers, setup handoff, Live resources, empty GCP profiles, unrelated vault grants, and serialized retry after a lost response. Fail-then-pass validation: disabling permission-change scheduling caused all three cloud regressions to fail; restoring it passed all three. The legacy cleanup and reordered-policy retry regressions also failed against the old behavior and passed with the fixes. All Greptile review threads are resolved. Greptile scored the final commit 5/5 after findings were addressed before CI. GitHub CI passed: Rust crates, every-package/target checks, worker integration, and PR conventions. Cloud E2E selected no deployment jobs for this change.

Closes ALIEN-1305

Live AWS validation: added an opt-in controller integration test. Reading the synthetic SecureString is denied before the permission-only update, then succeeds after the real executor reconciles IAM. Reading another namespace and reading from an unrelated role remain denied. All fixture roles and parameters are deleted and their absence verified. Disabling permission-change detection makes the same cloud test fail; restoring the candidate passes again. This validates AWS vault reconciliation, not full workload startup. Live Azure and GCP validation remains pending.

@alongubkin

Copy link
Copy Markdown
Member Author

@greptile review

Please review the update paths, setup authority checks, saved-state retry behavior, and GCP preservation of other vaults' conditional grants. CI is deferred until review findings are addressed; regression tests are being validated in an isolated Linux sandbox.

@alongubkin

alongubkin commented Oct 7, 2026 •

Copy link
Copy Markdown
Member Author

@greptileai please review

Please review the current head, 48d3572. The original handlers now fail nine sandbox regressions; validation of the fixed handlers is running. This push corrects only the new GCP test fixture's optional condition title. CI remains deferred until review findings are addressed.

@greptile-apps

greptile-apps Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Critical risk] Vault permission reconciliation across cloud platforms.

The PR appears safe to merge; the previous write-access finding is addressed, and no new actionable issues were found.

What we checked:

  • Another vault keeps write access: The server checks grants across every desired vault before removing the shared role. It removes only the management member when none still needs it.
  • Role lookup has required inputs: The caller supplies the required names before generating plans for registered vault grants. The lookup generates plans without making cloud writes.

Summary

The PR makes vault updates apply changed grants and saves permissions_revision only after success. Older checkpoints reconcile once before converging.

  • AWS removes stale owned policies. Azure applies grants during updates.
  • GCP preserves grants shared with other vaults and removes write access when no desired vault grant needs it.
  • The latest change addresses the previous finding about retained project-wide write access. No new actionable issues were found.
  • This review checked code and regression tests; it did not run cloud validation.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Setup plans vault update] --> B{Grant revision changed?}
  B -->|No| C[No permission-only update]
  B -->|Yes| D[Check setup authority]
  D --> E[Apply desired grants and remove owned stale grants]
  E --> F{Cloud work succeeded?}
  F -->|No| G[Keep previous revision for retry]
  F -->|Yes| H[Save permissions_revision]
  H --> C
Loading

Reviews (9) · Last reviewed commit: "fix: remove dropped GCP vault write gran..." · Reviewed by Greptile

Comment thread crates/alien-infra/src/vault/gcp.rs Outdated
Comment thread crates/alien-infra/src/vault/gcp.rs Outdated
Comment thread crates/alien-infra/src/vault/gcp.rs
Comment thread crates/alien-infra/src/vault/gcp.rs Outdated
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai please review the current head.

Both findings are addressed: the fixture title now uses Some(...), and Azure/GCP vaults record the resolved-grant revision after successful reconciliation. Their needs_update hooks schedule permission-only changes, then stop after convergence. The new revision fields default to None for older checkpoints, and importers initialize them without granting runtime setup authority.

New executor regressions cover matching saved dependencies and checkpoints without the revision field. All 15 preceding vault regressions passed in the Linux sandbox; the expanded suite is running now. CI remains deferred until review findings are resolved.

@alongubkin alongubkin changed the title fix: reconcile Azure and GCP vault grants on update fix: reconcile vault grants across cloud updates Oct 7, 2026
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai please review the current head, 1d1eca4.

The empty-profile finding is fixed and has a create/update regression. Permission-only scheduling now covers AWS as well as Azure and GCP. GCP preserves other vault namespaces and shared unconditional roles, and retries can adopt an already-committed policy without writing again.

The new grant revision defaults to None on older checkpoints and is recorded only after successful reconciliation. The preceding expanded vault suite passed all 19 tests. The final full infrastructure unit suite is running in the Linux sandbox. CI remains deferred until review findings are resolved.

@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai please review the latest commit. All 947 infrastructure unit tests pass in an isolated Linux sandbox (one existing test ignored). All review findings have been addressed and resolved, including permission-only scheduling and empty GCP grant profiles. Please update the confidence score for the latest head.

Comment thread crates/alien-infra/src/vault/gcp.rs Outdated
Comment thread crates/alien-infra/src/vault/gcp.rs Outdated
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai please review the latest head. The delayed findings on legacy empty-profile cleanup and order-sensitive retries are now fixed. The strengthened regression tests fail against the old behavior (6 passed, 2 failed) and the final full-suite rerun is underway. Existing unknown grant history requires project metadata; new or known-empty profiles remain supported. Binding and member order no longer cause redundant IAM writes.

@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai please review the latest head. This is an empty commit to trigger GitHub CI after addressing review findings first. Its code tree is identical to ecd16e6, which received 5/5. Sandbox validation passed all 947 infrastructure unit tests, with one existing ignored test; all review threads are resolved.

alongubkin and others added 5 commits October 8, 2026 08:09
…ss concurrent writers

AWS: a vault reconcile now deletes its own inline policies
(alien-<vault>-vault-* on service-account roles, alien-mgmt-<vault>-vault-*
on the management role) that the desired stack no longer grants, before it
records permissionsRevision. Only exact registered names are removed;
NotFound counts as removed, AccessDenied fails with a setup-required
message, and a lost delete response resumes cleanly.

GCP: a stale etag on the project IAM write (409) re-reads and re-merges in
the same state for up to 5 attempts instead of failing the step. The
management identity's reconcile now leaves bindings conditioned on a vault
namespace in this stack to the vault controllers, so its update can no
longer strip a vault's conditional grants after the vault recorded its
revision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t-updates' into alon/alien-1305-cross-cloud-vault-updates-wt
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

Comment thread crates/alien-infra/src/vault/gcp.rs
…e bindings [skip ci]

Vault custom role IDs come from grant labels, so the permission-set prefix
never matched them and a dropped vault/data-write left both its
namespace-scoped custom-role binding and its unconditional
secretmanager.secrets.create binding on the management identity. The vault
now names its custom roles exactly, removes stale namespace bindings to
them, and drops the member from a shared unconditional vault role only
when no vault grant in the desired stack still produces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin
alongubkin merged commit f8ee218 into main Oct 8, 2026
23 checks passed
@alongubkin
alongubkin deleted the alon/alien-1305-cross-cloud-vault-updates branch October 8, 2026 06:42
alongubkin added a commit that referenced this pull request Oct 8, 2026
Container workload roles trusted only the retained legacy node identity,
so nodes using the isolation identity could not vend credentials and
workloads failed at startup. Trust both exact stack-scoped node role
ARNs while retaining the account condition and caller-side AssumeRole
requirement. Privileged setup reconciles older workload-role
checkpoints; runtime authority rejects saved trust/policy write states
before IAM access.

Live AWS reproduction: old trust allows legacy nodes but denies isolated
nodes. The candidate allows both node identities for newly created roles
and for a preexisting legacy role repaired through the actual
controller. An unrelated same-account node remains denied despite having
its own caller-side AssumeRole grant. Every task-owned role is deleted
and absence verified. Three focused tests also pass.

Legacy checkpoint validation deserializes the older JSON without the new
optional field and converges through the existing controller states.
Full generated Terraform and CloudFormation compute renders pass
terraform fmt/validate and cfn-lint. Full workload startup on isolated
nodes is not yet exercised end to end; the STS role-chaining proof above
covers the trust change itself.

After merging main: the vault permission-update fixtures added by #932
now record the applied default trust policy, so ready roles are not
scheduled for legacy trust repair (including the runner-owned role in
the ignored live vault test). `alien-infra` service_account (15) and
vault (31) tests and all `alien-cloudformation` tests pass.

Refs ALIEN-1316, ALIEN-1314



**New controller state variants:**
`AwsServiceAccountState::UpdatingStackPermissions` and
`UpdatingResourcePermissions`. Updates go `UpdateStart ->
UpdatingStackPermissions -> UpdatingResourcePermissions -> Ready`,
report `Updating`, and fail into `UpdateFailed`, so a failed policy
write can't put a live role into `CreateFailed`, where the executor
would replace it. Only privileged setup writes these states. Older
binaries can't read them, so a checkpoint that stopped mid-update needs
this version to resume. Create states are unchanged.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant