Repository navigation
fix: complete provisioning after removed resources finish deletion - #925
Merged
Merged
Conversation
|
Enclave skipped this draft pull request. It will run automatically when you mark the PR as Ready for review. |
Member
Author
|
@greptileai review |
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Member
Author
|
@greptileai review |
alongubkin
marked this pull request as ready for review
October 7, 2026 12:14
Provisioning aggregated every retained resource record, so a removed live resource left in Deleted state next to Running desired resources kept the stack InProgress and the release was never promoted. Use the update completion policy over the executor's reconciliation scope instead: ignore Deleted records absent from the target, keep desired deleted resources, desired config drift and deferred deletions as outstanding work. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alongubkin
force-pushed
the
alon/alien-1209-provisioning-completion
branch
from
October 9, 2026 08:46
c150055 to
a7db14b
Compare
Member
Author
|
@greptileai review |
alongubkin
added a commit
that referenced
this pull request
Oct 9, 2026
## Problem Some controller errors can only be fixed by rerunning setup. Examples: EBS growth on a management role without `ec2:DescribeVolumesModifications` (`MANAGEMENT_PERMISSION_MISSING`), or a workload role whose trust does not admit the compute nodes. Rerunning setup did not finish the deployment: 1. The engine reported the failure like any other (`update-failed`, headline `DEPLOYMENT_FAILED`). A control plane could only record the operation as failed, so a setup run had nothing to claim and got a bare `statusMismatch`. 2. After a retry and then setup, setup handed the deployment back at `Provisioning`. `handle_provisioning` found the resource still failed, failed again, and reported `provisioning-failed` for an installed deployment. Only a second retry finished it. ## Change - `alien-infra`: `requires_setup_rerun(error)` recognizes the errors only a setup run resolves: `MANAGEMENT_PERMISSION_MISSING` and a new `SETUP_RERUN_REQUIRED` (setup-owned configuration such as workload role trust). - `alien-deployment`: - When every real failure in a failed provisioning or update needs setup, the deployment error is `DEPLOYMENT_RESOURCE_SETUP_REQUIRED`, naming the resources. If any other failure is mixed in, behavior is unchanged and a retry is still asked for. The message also says that CloudFormation/Terraform installs retry after updating the stack, because a stack update does not hand the deployment back. - On entry to `Provisioning` with such failures (reachable only through a setup handoff), the runtime resumes them at their saved step instead of failing again. Authority is unchanged: setup grants, and the runtime only continues its own controller steps. - `Provisioning` now uses the same completion check as `Updating` (same change as #925). A removed resource's `Deleted` tombstone therefore no longer holds a resumed update open (Greptile P1). - An installed deployment that fails at `Provisioning` reports `update-failed`; its installed release keeps serving. - A setup run refused with `statusMismatch` on a failed deployment now returns `SETUP_RUN_AFTER_FAILED_OPERATION`, which says to retry first. The refusal stays in the error chain. If the status lookup fails, the plain refusal is returned. - Test worker controller: `withhold_test_worker_setup_grant` / `give_test_worker_setup_grant` simulate a step that needs setup. Alternatives considered: retrying automatically in the runtime would keep failing until the grant exists. Having setup request a retry would give setup-run authority to runtime retries. Resuming at the existing setup handoff is the smallest change that keeps authority where it is. ## Proof Tests ran in a Linux sandbox on this commit, rebased on main `7b9a37011`: - `alien-deployment` + `alien-infra` (`--features alien-infra/test`) all pass; `rustfmt --check` passes on the changed files. - `test_platform`: - `an_update_waiting_for_a_setup_grant_finishes_when_setup_reruns`: a mid-update step fails with a missing grant. An idle step changes nothing. A setup run that does not grant it leaves the deployment `update-failed` and waiting (never `provisioning-failed`). A setup run that grants it finishes the update from the failed step, with no retry. - `an_update_that_removes_a_resource_finishes_after_setup_reruns`: the same flow while removing a worker. - `a_first_install_waiting_for_setup_continues_when_setup_reruns`. - Fail-without-fix: - with `provisioning.rs` + `updating.rs` at main, all 3 fail; - with only the Provisioning completion change reverted, the removal test fails (stuck in `Provisioning`); - with only the handoff resume disabled, all 3 fail; - with only the transport lookup removed, `setup_run_refused_after_a_failed_operation_names_the_retry` fails. - Unit tests: `only_failures_that_all_need_setup_wait_for_setup`, `setup_rerun_errors_are_recognized_through_wrapping`, plus mocked acquire+GET tests for the refusal (failed status, non-failed status, lookup failure, other refusal reason). ## Rollout New error codes: `SETUP_RERUN_REQUIRED`, `DEPLOYMENT_RESOURCE_SETUP_REQUIRED`, `SETUP_RUN_AFTER_FAILED_OPERATION`. No controller state variants changed. A control plane that does not know `DEPLOYMENT_RESOURCE_SETUP_REQUIRED` treats it as an ordinary failure, which is today's behavior. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem. Provisioning could stay open forever after a removed resource finished deleting: its retained
Deletedrecord was aggregated with the desiredRunningresources bycompute_stack_status, so the stack never readRunning. It could also promote the release too early while a deferred removal (a dependency still held by a consumer) was outstanding, because every resource readRunning.Root cause. Provisioning computed completion with the global
compute_stack_statusplus a separate convergence check, while updates already use a target-aware policy (compute_update_status) that ignores retainedDeletedrecords outside the target and waits for pending deletions and config drift.Alternatives. Dropping
Deletedrecords from state would change deletion ownership for every caller; adding another provisioning-only special case would duplicate the update policy. Reusingcompute_update_statuskeeps one completion rule.Fix. Provisioning now uses
compute_update_statuswith the executor's reconciled scope andpending_deletions. This replaces the separatestack_has_convergedblock on main (that check now runs insidecompute_update_status). Global stack status and deletion ownership are unchanged.Proof. Four real-controller tests in
provisioning::tests(they drivehandle_provisioningwith the test platform executor). On main with only the tests applied, two fail:provisioning_finishes_after_removed_live_resource_is_deleted(staysProvisioning) andprovisioning_waits_for_a_removed_dependency_to_finish_deleting(promotes toRunningwith the old dependency undeleted). With the fix all four pass. The other two (desired deleted resource is recreated; removed-resource delete failure is preserved) guard behavior that already holds.Rollout. Controller-only change; no schema or API change.
Refs ALIEN-1209.
🤖 Generated with Claude Code