Repository navigation
fix: grant and decode EBS volume modification checks - #954
Merged
Merged
Conversation
Growing a container's EBS volume reads the volume's latest modification before calling ModifyVolume, so a retry never resends a resize AWS has already accepted. No permission set granted ec2:DescribeVolumesModifications, so the call was denied on every install. - container/provision (AWS) grants ec2:DescribeVolumesModifications. The action supports no resource types, so it joins the Resource "*" read statement. - EC2 client: InvalidVolumeModification.NotFound (the volume was never modified) decodes as RemoteResourceNotFound, and VolumeModificationRateExceeded (more than four modifications in a rolling 24 hours) decodes as QuotaExceeded with the provider code and the time AWS allows the next one. Neither is resent. - alien-infra: MANAGEMENT_PERMISSION_MISSING, a non-retryable error for a step whose grant only a setup rerun can add. It names the action. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Enclave skipped this draft pull request. It will run automatically when you mark the PR as Ready for review. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
@greptileai review |
|
… contract Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
@greptileai review |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Growing a stateful container's EBS volume on AWS fails on every install. Before it calls
ModifyVolume, the growth step reads the volume's latest modification withDescribeVolumesModifications, so a retry never resends a resize that AWS has already accepted. Two things break it:ec2:DescribeVolumesModifications. The management role getsUnauthorizedOperation, which is retried as a generic error.VolumeId.Nabout a volume that was never modified, AWS returns HTTP 400InvalidVolumeModification.NotFound, not an empty set. The client left it as an undecoded HTTP error.I checked this against real AWS (eu-west-1) on two throwaway 1 GiB gp3 volumes:
DescribeVolumesModifications --volume-ids <never modified>InvalidVolumeModification.NotFound--filters Name=volume-id,Values=<ids>(any mix)ModifyVolumewhilemodifyingIncorrectModificationStateModifyVolumein 24 h (a no-change modify counts)VolumeModificationRateExceeded"Wait until "DescribeVolumessize duringmodifyingThe limit is four modifications per volume in a rolling 24 hours (AWS, Jan 2026). The IAM service reference lists
DescribeVolumesModificationswith no resource types.What
container/provision(AWS) grantsec2:DescribeVolumesModificationsin the existingResource: "*"read statement.InvalidVolumeModification.NotFoundasRemoteResourceNotFoundandVolumeModificationRateExceededasQuotaExceeded. The quota keeps the provider code and AWS's "Wait until" time, and it is not resent as throttling.MANAGEMENT_PERMISSION_MISSING: a non-retryable error for a step whose grant only a setup rerun can add. It names the action and says to rerun setup, then retry.Existing installs
The management permissions revision is a hash of the resolved permission sets, so this change marks it stale. The grant arrives when the installation's setup runs again: a direct
alien deploy, or a CloudFormation/Terraform update. Until then, the controller fails growth once withMANAGEMENT_PERMISSION_MISSING, instead of retrying a generic 403. A retry after setup resumes the failed step.Validation
cargo test -p alien-aws-clients -p alien-permissions -p alien-infra: results in a comment below.🤖 Generated with Claude Code