Skip to content

fix: retry failed generated CLI teardown - #620

Merged
alongubkin merged 4 commits into
mainfrom
alon/alien-793-generated-deploy-cli-cannot-retry-failed-setup-teardown
Sep 14, 2026
Merged

alongubkin merged 4 commits into
mainfrom
alon/alien-793-generated-deploy-cli-cannot-retry-failed-setup-teardown

Conversation

@alongubkin

Copy link
Copy Markdown
Member

Summary

  • request a new delete attempt when setup-owned teardown previously failed
  • keep teardown-required idempotent while work is still active
  • preserve the tracked deployment until teardown succeeds

Validation

  • cargo test -p alien-deploy-cli failed_setup_teardown_requests_a_new_delete_attempt
  • real generated CLI retry against a failed AWS setup: the API re-armed the same delete operation, the CLI resumed from the failed Aurora delete state, removed the remaining IAM resources, and completed successfully
  • verified no IAM roles, IAM policies, Aurora clusters, Aurora instances, or DB subnet groups remained for the deployment prefix

Linear: ALIEN-793

@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai Please review the current head.

@greptile-apps

greptile-apps Bot commented Sep 14, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

The PR appears safe to merge; the combined teardown retry and lease-renewal lifecycle is coherent and has focused coverage.

Summary

  • Requests deletion again for teardown-failed, while preserving idempotent handling of active teardown-required work.
  • Extracts reusable lease-renewal handling and applies it to setup teardown.
  • Retains the last stack checkpoint while provider deletion is in flight so lease-loss cancellation can resume safely.
  • Adds focused lifecycle tests for teardown retry ordering and lease renewal.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[CLI down] --> B{Deployment status}
    B -->|teardown-required| D[Acquire existing setup teardown]
    B -->|teardown-failed or other| C[Request cleanup / re-arm operation]
    C --> D
    D --> E[Run setup teardown]
    E --> F[Renew lease periodically]
    E --> G[Clone retained stack checkpoint]
    G --> H[Execute provider deletion step]
    H -->|Success| I[Persist next checkpoint]
    H -->|Failure| J[Mark teardown failed]
    F -->|Lease lost| K[Cancel operation and retain prior checkpoint]
    I --> L{Teardown complete?}
    L -->|No| G
    L -->|Yes| M[Final reconcile and release]
Loading

Reviews (4) · Last reviewed commit: "fix: renew setup teardown leases (#621)"

Comment thread crates/alien-deploy-cli/src/commands/down.rs Outdated
Comment thread crates/alien-deploy-cli/src/commands/down.rs Outdated
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai Please review the current head. The retry test now exercises the request-to-acquisition sequence, and the lifecycle comment reflects the API re-arming the canonical failed delete operation.

…ted-deploy-cli-cannot-retry-failed-setup-teardown
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai

## Summary

- reuse the shared lease-renewal guard around setup-owned teardown
- cancel setup teardown before ownership can expire when renewals fail
- cover lease renewal during an operation longer than the renewal
interval

## Root cause

Setup-owned teardown has its own long-running resource loop. Unlike the
normal deployment runner, it did not renew the deployment lease. A real
generated-CLI AWS destroy therefore lost its claim after exactly five
minutes while Aurora was still deleting.

## Validation

- `cargo test -p alien-deployment --lib` (103 passed)
- Real pre-fix reproduction: deployment
`dep_timzeyh4w5ia0owdlytoow3wes0k` acquired at 06:45:20 UTC, lease
expired at 06:50:20.625, and its next checkpoint returned
`DEPLOYMENT_UPDATE_CLAIM_LOST` while RDS remained in `deleting`.
- Supported retry then completed teardown; exact-prefix AWS and Alien
metadata audits were empty. A post-fix >5-minute real teardown remains
the final acceptance gate.

## Stack

Stacked on #620 because the generated CLI retry behavior is needed to
recover the pre-fix failure fixture cleanly.

Linear: ALIEN-794
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai Please review the current head after the stacked lease-renewal PR merged into this branch. Both changes have focused lifecycle coverage; please assess the combined diff.

@alongubkin
alongubkin merged commit 743504d into main Sep 14, 2026
21 checks passed
@alongubkin
alongubkin deleted the alon/alien-793-generated-deploy-cli-cannot-retry-failed-setup-teardown branch September 14, 2026 10:03
alongubkin added a commit that referenced this pull request Sep 14, 2026
## Summary

- return a typed delete response that tells setup clients whether
setup-owned cleanup is still required
- let generated deployment CLIs finish cleanly when the manager already
removed a never-provisioned deployment
- preserve compatibility with older managers by treating an absent
cleanup result as cleanup required
- regenerate the Manager OpenAPI and Rust SDK inputs from the route
schema

## Why

A deployment that fails initial preflights before creating stack state
owns no cloud resources. The Platform can safely remove that empty
record, but the generated CLI previously continued into setup teardown
and received a 404 because the deployment was already gone.

The response is explicit rather than inferred from messages or errors.
Existing managers remain safe: a new CLI defaults a missing field to
continuing cleanup.

## Validation

- `cargo test -p alien-manager routes::deployments::tests --lib` (6
passed, including both route response outcomes)
- `cargo test -p alien-deploy-cli commands::down::tests` (7 passed)
- `pnpm run generate:manager-rust-sdk` reproduced the checked-in OpenAPI
inputs with no diff
- real malformed generated-CLI fixture
`dep_np5lyeeke2aqndq5raweleoi3cak`: one ordinary destroy removed the
deployment, operation, certificate, and local tracker without setup
teardown; an exact-tag AWS audit was empty

## Stack

Stacked on #620 until its teardown retry and lease-renewal changes
merge.

Linear: ALIEN-796
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant