Skip to content

fix: grant and decode EBS volume modification checks - #954

Merged
alongubkin merged 4 commits into
mainfrom
alon/alien-1332-ebs-volume-growth
Oct 8, 2026
Merged

alongubkin merged 4 commits into
mainfrom
alon/alien-1332-ebs-volume-growth

Conversation

@alongubkin

Copy link
Copy Markdown
Member

Why

Growing a stateful container's EBS volume on AWS fails on every install. Before it calls ModifyVolume, the growth step reads the volume's latest modification with DescribeVolumesModifications, so a retry never resends a resize that AWS has already accepted. Two things break it:

  1. No permission set grants ec2:DescribeVolumesModifications. The management role gets UnauthorizedOperation, which is retried as a generic error.
  2. Asked with VolumeId.N about a volume that was never modified, AWS returns HTTP 400 InvalidVolumeModification.NotFound, not an empty set. The client left it as an undecoded HTTP error.

I checked this against real AWS (eu-west-1) on two throwaway 1 GiB gp3 volumes:

Request Result
DescribeVolumesModifications --volume-ids <never modified> 400 InvalidVolumeModification.NotFound
same, with one modified and one never-modified volume 400 NotFound for the whole call
--filters Name=volume-id,Values=<ids> (any mix) 200, only the modified volumes
ModifyVolume while modifying 400 IncorrectModificationState
5th ModifyVolume in 24 h (a no-change modify counts) 400 VolumeModificationRateExceeded "Wait until "
DescribeVolumes size during modifying still the old size

The limit is four modifications per volume in a rolling 24 hours (AWS, Jan 2026). The IAM service reference lists DescribeVolumesModifications with no resource types.

What

  • container/provision (AWS) grants ec2:DescribeVolumesModifications in the existing Resource: "*" read statement.
  • The EC2 client decodes InvalidVolumeModification.NotFound as RemoteResourceNotFound and VolumeModificationRateExceeded as QuotaExceeded. The quota keeps the provider code and AWS's "Wait until" time, and it is not resent as throttling.
  • alien-infra adds MANAGEMENT_PERMISSION_MISSING: a non-retryable error for a step whose grant only a setup rerun can add. It names the action and says to rerun setup, then retry.

Existing installs

The management permissions revision is a hash of the resolved permission sets, so this change marks it stale. The grant arrives when the installation's setup runs again: a direct alien deploy, or a CloudFormation/Terraform update. Until then, the controller fails growth once with MANAGEMENT_PERMISSION_MISSING, instead of retrying a generic 403. A retry after setup resumes the failed step.

Validation

  • Client tests decode the exact response bodies AWS returned above, and assert that a rate-limited modify is sent once.
  • cargo test -p alien-aws-clients -p alien-permissions -p alien-infra: results in a comment below.

🤖 Generated with Claude Code

Growing a container's EBS volume reads the volume's latest modification
before calling ModifyVolume, so a retry never resends a resize AWS has
already accepted. No permission set granted
ec2:DescribeVolumesModifications, so the call was denied on every
install.

- container/provision (AWS) grants ec2:DescribeVolumesModifications.
  The action supports no resource types, so it joins the Resource "*"
  read statement.
- EC2 client: InvalidVolumeModification.NotFound (the volume was never
  modified) decodes as RemoteResourceNotFound, and
  VolumeModificationRateExceeded (more than four modifications in a
  rolling 24 hours) decodes as QuotaExceeded with the provider code and
  the time AWS allows the next one. Neither is resent.
- alien-infra: MANAGEMENT_PERMISSION_MISSING, a non-retryable error for
  a step whose grant only a setup rerun can add. It names the action.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@enclave-ai

enclave-ai Bot commented Oct 8, 2026

Copy link
Copy Markdown

Enclave skipped this draft pull request. It will run automatically when you mark the PR as Ready for review.

alongubkin and others added 2 commits October 8, 2026 21:27
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

@greptile-apps

greptile-apps Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Critical impact] The PR appears safe to merge, with no outstanding findings.

Summary

Adds the missing EBS read permission, decodes modification errors, and provides errors that tell callers when setup or waiting is required.

  • Since the last review, errors now name the requested volumes, and tests check that wrapped errors stop retries.
  • Both earlier comments are cleared. alongubkin explained that downstream controllers use the public error variants; Greptile withdrew the caller finding. The missing-volume-name finding is fixed.
  • No new actionable issues found. Tests were inspected but not run.

Reviews (3) · Last reviewed commit: "fix: name the volumes in modification er..." · Reviewed by Greptile

Comment thread crates/alien-infra/src/error.rs
Comment thread crates/alien-aws-clients/src/aws/ec2.rs
… contract

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

@alongubkin
alongubkin marked this pull request as ready for review October 8, 2026 21:04
@alongubkin
alongubkin merged commit c4cac8b into main Oct 8, 2026
23 of 24 checks passed
@alongubkin
alongubkin deleted the alon/alien-1332-ebs-volume-growth branch October 8, 2026 21:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant