fix: resume setup-required failures when setup reruns

[alongubkin]

Problem

Some controller errors can only be fixed by rerunning setup. Examples: EBS growth on a management role without ec2:DescribeVolumesModifications (MANAGEMENT_PERMISSION_MISSING), or a workload role whose trust does not admit the compute nodes. Rerunning setup did not finish the deployment:

1. The engine reported the failure like any other (update-failed, headline DEPLOYMENT_FAILED). A control plane could only record the operation as failed, so a setup run had nothing to claim and got a bare statusMismatch.
2. After a retry and then setup, setup handed the deployment back at Provisioning. handle_provisioning found the resource still failed, failed again, and reported provisioning-failed for an installed deployment. Only a second retry finished it.

Change

• alien-infra: requires_setup_rerun(error) recognizes the errors only a setup run resolves: MANAGEMENT_PERMISSION_MISSING and a new SETUP_RERUN_REQUIRED (setup-owned configuration such as workload role trust).
• alien-deployment:
  • When every real failure in a failed provisioning or update needs setup, the deployment error is DEPLOYMENT_RESOURCE_SETUP_REQUIRED, naming the resources. If any other failure is mixed in, behavior is unchanged and a retry is still asked for. The message also says that CloudFormation/Terraform installs retry after updating the stack, because a stack update does not hand the deployment back.
  • On entry to Provisioning with such failures (reachable only through a setup handoff), the runtime resumes them at their saved step instead of failing again. Authority is unchanged: setup grants, and the runtime only continues its own controller steps.
  • Provisioning now uses the same completion check as Updating (same change as fix: complete provisioning after removed resources finish deletion #925). A removed resource's Deleted tombstone therefore no longer holds a resumed update open (Greptile P1).
  • An installed deployment that fails at Provisioning reports update-failed; its installed release keeps serving.
  • A setup run refused with statusMismatch on a failed deployment now returns SETUP_RUN_AFTER_FAILED_OPERATION, which says to retry first. The refusal stays in the error chain. If the status lookup fails, the plain refusal is returned.
• Test worker controller: withhold_test_worker_setup_grant / give_test_worker_setup_grant simulate a step that needs setup.

Alternatives considered: retrying automatically in the runtime would keep failing until the grant exists. Having setup request a retry would give setup-run authority to runtime retries. Resuming at the existing setup handoff is the smallest change that keeps authority where it is.

Proof

Tests ran in a Linux sandbox on this commit, rebased on main 7b9a37011:

• alien-deployment + alien-infra (--features alien-infra/test) all pass; rustfmt --check passes on the changed files.
• test_platform:
  • an_update_waiting_for_a_setup_grant_finishes_when_setup_reruns: a mid-update step fails with a missing grant. An idle step changes nothing. A setup run that does not grant it leaves the deployment update-failed and waiting (never provisioning-failed). A setup run that grants it finishes the update from the failed step, with no retry.
  • an_update_that_removes_a_resource_finishes_after_setup_reruns: the same flow while removing a worker.
  • a_first_install_waiting_for_setup_continues_when_setup_reruns.
• Fail-without-fix:
  • with provisioning.rs + updating.rs at main, all 3 fail;
  • with only the Provisioning completion change reverted, the removal test fails (stuck in Provisioning);
  • with only the handoff resume disabled, all 3 fail;
  • with only the transport lookup removed, setup_run_refused_after_a_failed_operation_names_the_retry fails.
• Unit tests: only_failures_that_all_need_setup_wait_for_setup, setup_rerun_errors_are_recognized_through_wrapping, plus mocked acquire+GET tests for the refusal (failed status, non-failed status, lookup failure, other refusal reason).

Rollout

New error codes: SETUP_RERUN_REQUIRED, DEPLOYMENT_RESOURCE_SETUP_REQUIRED, SETUP_RUN_AFTER_FAILED_OPERATION. No controller state variants changed. A control plane that does not know DEPLOYMENT_RESOURCE_SETUP_REQUIRED treats it as an ordinary failure, which is today's behavior.

🤖 Generated with Claude Code

[enclave-ai]

Enclave skipped this draft pull request. It will run automatically when you mark the PR as Ready for review.

[alongubkin]

@greptileai review

[greptile-apps]

Confidence Score: 5/5

[High impact] The PR appears safe to merge; no actionable issue remains from this review.

Summary

The PR lets deployments continue from saved controller steps after setup reruns.

• Deployments wait for setup when every real failure needs it.
• The runtime resumes setup-blocked resources from their saved steps.
• A failed setup handoff now tells operators to retry first.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Resource step fails] --> B{Every real failure needs setup?}
  B -->|No| C[Ordinary failure: retry required]
  B -->|Yes| D[Report setup required]
  D --> E[Setup reruns]
  E --> F[Setup hands back at Provisioning]
  F --> G[Resume saved steps]
  G --> H{Work completed?}
  H -->|Yes| I[Running on target release]
  H -->|No: failure| J[UpdateFailed if installed; otherwise ProvisioningFailed]

Loading

_{Reviews (3) · Last reviewed commit: "fix: resume setup-required failures when..." · Reviewed by Greptile}

[alongubkin]

@greptileai review