fix: revoke registry access before handing a deletion to setup teardown

[alongubkin]

Problem

Deleting an AWS deployment whose Lambda pulls from the manager's ECR registry could hang for 45 minutes and then time out. This happened when the manager's registry role was not allowed to delete a repository policy.

1. The manager revokes the deployment's registry grant at TeardownRequired. It persisted that state first and revoked afterwards. When the registry denied the revoke, the step failed as retryable, and the runner retried every 30 s while holding the deployment lease. alien-deploy destroy waited on "runtime cleanup handoff" until it timed out.
2. If the manager restarted inside that loop, the deployment stayed at teardown-required with the grant still in place. Setup teardown then deleted the role and the record, and nothing revoked the grant afterwards. The account root principal stayed in the repository policy.
3. The deploy CLI drives setup teardown through POST /v1/sync/reconcile. On its final Deleted step the route deleted the record and then ran the revoke, which takes tens of seconds when the policy is already gone. A lease renewal during that window got 404, and destroy failed with DEPLOYMENT_LEASE_LOST even though everything had been deleted. This was reproduced twice against a real manager.

Root cause

The revoke ran after the state that hands the deployment on was saved. Any revoke failure was also treated as a retryable checkpoint failure, including a denial that no retry can fix.

Fix

• On AWS, revoke_before_persisting runs the revoke before TeardownRequired or Deleted is persisted. It is used by the manager loop transport, the reconcile route and the runtime-less destroy path.
• GCP keeps revoking after the state is persisted (revoke_after_persisting, same as main). Its shared Cloud Run grant is removed only when no other deployment in the project is saved as active, and each deletion is saved before it checks. So of two deletions finishing together, the one that checks last sees both deleted and removes the grant.
• A denied revoke (RemoteAccessDenied from the registry) becomes REGISTRY_ACCESS_REVOKE_DENIED. The deployment is persisted as DeleteFailed with that error, which names the repository and the missing permission. The runner and the deploy CLI adopt that state and stop. Once the permission is granted, retry runs runtime cleanup again and revokes.
• Any other AWS revoke failure persists nothing and returns as before, so the runner retries with the deployment still Deleting.
• The operator sync path, where pull-mode state belongs to the operator, is unchanged.

Alternatives considered:

• Making the cleanup error inherit retryable. It doesn't help: REMOTE_ACCESS_DENIED is retryable for IAM propagation. It also leaks silently, because the runner would exit, setup teardown would take the deployment, and the grant would stay.
• Revoking best-effort and continuing. Same leak.
• Bounding the checkpoint retry in the runner. That would weaken a durability barrier every step relies on.

Proof

Each new test fails on main and passes with this change. Both runs used the same test code, in a sandbox.

• registry_access::tests::revoke_before_handoff drives the real ManagerTransport against a real SQLite store. Its fake registry records the deployment's persisted status at the moment of each revoke.
  • the_revoke_runs_before_setup_teardown_can_take_the_deployment: main revokes when the record already says teardown-required; with the fix it still says deleting.
  • a_denied_revoke_fails_the_delete_and_a_retry_hands_off_once_granted: main returns REGISTRY_ACCESS_CLEANUP_FAILED with retryable: true, which is the endless retry. With the fix the record is delete-failed and carries the error. After the permission is granted, the retried handoff revokes and persists teardown-required.
  • a_failed_revoke_persists_nothing_for_the_runner_to_retry: main leaves teardown-required persisted; with the fix it stays deleting.
  • gcp_revokes_the_shared_grant_only_after_its_deletion_is_saved guards the GCP ordering. It passes on main and with the fix. It fails if GCP is moved before persisting: the revoke then runs while the record is still deleting.
• tests/registry_revoke_handoff.rs runs the same scenario through POST /v1/sync/reconcile:
  • On the final Deleted step, main revokes after the record is deleted; with the fix it revokes while the record is still teardown-required.
  • A denied revoke returns 500 on main. With the fix it returns 200 with current.status = delete-failed.
• cargo test -p alien-manager --no-fail-fast: everything passes except routes::airgap::tests::a_backlog_is_forwarded_..., a timing-budget test. It fails under full parallel load and passes 3/3 alone. It is unrelated to this change.

Live, against a real hosted manager deploying a Lambda worker into a second AWS account, with this PR's base and the registry permission present:

• The grant was written on the source and replica repositories, then revoked at TeardownRequired. Both policies were deleted and the target resources removed.
• Each setup-teardown reconcile re-ran the revoke, taking 30–40 s. The final one ran after the record was deleted, and destroy exited with DEPLOYMENT_LEASE_LOST. That is item 3 above.
• With the permission missing, the same flow looped on REGISTRY_ACCESS_CLEANUP_FAILED every 30 s for over 40 minutes. That is items 1 and 2.

Rollout

Manager-only change. A deployment already stuck at teardown-required with a denied revoke is picked up again only when something re-runs its delete. After this ships, a denied revoke shows as a failed delete with the missing permission named.

🤖 Generated with Claude Code

[enclave-ai]

Enclave skipped this draft pull request. It will run automatically when you mark the PR as Ready for review.

[alongubkin]

@greptileai review

[greptile-apps]

Confidence Score: 5/5

[High impact] The PR appears safe to merge; no blocking findings remain.

Summary

The PR moves AWS registry cleanup before the manager saves the setup handoff. A denied revoke becomes DeleteFailed.

• AWS deletes revoke registry access before setup teardown takes over.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Runtime cleanup finishes] --> B{Cloud provider}
    B -->|AWS| C[Revoke registry access]
    C -->|Success| D[Save teardown handoff or Deleted]
    C -->|Permission denied| E[Save DeleteFailed]
    C -->|Other error| F[Return error without saving]
    B -->|GCP| G[Save Deleted]
    G --> H[Check for other active deployments]
    H -->|None remain| I[Remove shared registry access]
    H -->|Another remains| J[Keep shared registry access]

Loading

_{Reviews (3) · Last reviewed commit: "fix: keep GCP's shared-grant revoke afte..." · Reviewed by Greptile}

[alongubkin]

@greptileai review