fix: recover interrupted volumes and network deletion

[alongubkin]

An interrupted EBS resize returns AWS IncorrectModificationState as an untyped HTTP error, so the caller cannot wait for the accepted modification. Decode it as a volume conflict with the provider code retained, and retry ModifyVolume only for throttling responses so ambiguous outcomes return to the controller for inspection.

Network teardown retains children after a denied deletion but advances to the VPC, where retries never revisit those children. Revisit retained child cleanup before the parent, bound complete cleanup passes, and preserve a denied VPC as a domain failure rather than reporting deletion. The new counter defaults to zero for old checkpoints and resets on manual retry; no state variants change.

Validation: real signed EC2 HTTP error decoding, permission-restoration deletion, old parent-delete checkpoint recovery, bounded waits, and the existing AWS network lifecycle tests. Regression checks restore the offending behavior and require the focused cases to fail. The local regression suite is supplemented by the live AWS test below.

The provider behavior is documented in EC2 error codes and DeleteVpc.

Executed 83 AWS network/EC2 tests with cargo nextest run -p alien-infra -p alien-aws-clients --lib --all-features -E "test(network::aws::) | test(modify_volume_decodes_in_progress)": all passed after rebasing onto current main. The three focused shared-crate regression cases fail when the offending behavior is restored.

Live AWS validation passed: delete_resumes_retained_children_after_real_permission_restoration creates a complete real network, switches to an STS session explicitly denying security-group deletion, verifies independent subnets disappear while the denied group and parent VPC remain, restores permission, reloads an older parent-delete checkpoint, and finishes with an empty tagged inventory. No sweep leftovers remained. Run with cargo test -p alien-infra --features all-platforms --test aws_network_live delete_resumes_retained_children_after_real_permission_restoration -- --ignored --exact --nocapture and the documented live/restricted-session credentials. TLS verification stays enabled; ALIEN_LIVE_TEST_CA_CERT optionally trusts an explicit runner CA.

The actual EC2 endpoint also returned IncorrectModificationState after an accepted 10→20 GiB resize; the live signed client decoded it as REMOTE_RESOURCE_CONFLICT. Both created test volumes were independently observed deleted. Live results were captured at 2d2131324; the final rebase adds the newer main fixes without changing the tested network/EC2 paths.

Greptile's resize-retry coverage finding is addressed by direct signed HTTP tests: modify_volume_does_not_resend_an_ambiguous_failure verifies one request for HTTP 503, and modify_volume_retries_an_explicit_throttling_response verifies throttling retries. All three resize tests pass. Restoring the old generic retry policy makes the ambiguous-response and conflict request-count tests fail with four sends instead of one.

Final rebased regression run passed 82 AWS network unit tests (one existing ignored test), 17 persistent-storage controller tests, three Azure handler/metadata tests, and the SDK conflict test; the two added retry tests also pass.

[enclave-ai]

Enclave skipped this draft pull request. It will run automatically when you mark the PR as Ready for review.

[alongubkin]

@greptileai review

[greptile-apps]

Confidence Score: 5/5

[High risk] Changes how the infrastructure controller retries volume and network deletion.

The PR appears safe to merge; the added resize tests address the previous finding.

What we checked:

• Resize tests match retry behavior: The client retries only recognized throttling errors. Its retry limit allows three retries, so the tests correctly expect one request for an ambiguous failure and four for throttling.

Summary

This PR makes interrupted volume resizing and network deletion recoverable.

• ModifyVolume retains the AWS conflict code and retries only explicit throttling responses.
• Network deletion revisits retained children before deleting their VPC and bounds those cleanup passes.
• A denied VPC deletion now returns ResourceDeleteBlocked instead of reporting success.
• Since the previous review, two tests were added to cover ambiguous resize failures and throttling retries. They address the previous finding. No new issues were found.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Network reaches parent deletion] --> B{Retained children?}
  B -->|Yes| C{Cleanup limit reached?}
  C -->|No| D[Revisit child deletion]
  D --> A
  C -->|Yes| E[Return ResourceDeleteBlocked]
  B -->|No| F[Delete VPC]
  F -->|Permission denied| E
  F -->|Deleted or already absent| G[Mark network deleted]

Loading

_{Reviews (2) · Last reviewed commit: "test: verify resize retries only follow ..." · Reviewed by Greptile}