Skip to content

feat: backups for container persistent volumes - #885

Merged
alongubkin merged 35 commits into
mainfrom
alon/alien-1232-volume-backups
Oct 6, 2026
Merged

alongubkin merged 35 commits into
mainfrom
alon/alien-1232-volume-backups

Conversation

@alongubkin

@alongubkin alongubkin commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Tracks ALIEN-1232.

Containers with .persistentStorage() get one block volume per replica. Before this change nothing snapshotted those volumes, and deleting the container deleted them immediately. This branch adds the types, cloud clients, permissions and CLI for volume backups and restore on AWS, GCP and Azure.

Config

  • persistentStorage(size, { backups }). Backups are on by default: every 24 hours, kept 7 days. Tune them with intervalHours (1, 2, 4, 6, 8, 12 or 24) and retentionDays (at most 365), or turn them off with backups: false.
  • A compile-time preflight keeps the schedule within what AWS Data Lifecycle Manager, Compute Engine snapshot schedules and Azure Disk Backup all support, including at most 450 snapshots per volume.

Outputs

  • ContainerOutputs.volumes: each replica's volume, zone, latest snapshot, and last restore.
  • ContainerOutputs.volumeBackups: active, disabled or setupRequired, with a message.

Restore

  • DeploymentConfig.volumeRestores carries restore requests (request ID, resource, ordinal, snapshot) to the controller. The manager's sync route passes them through.
  • CLI: alien deployments volumes, alien deployments restore-volume <deployment> --resource <id> --ordinal <n> --snapshot <id>, and alien deployments cancel-volume-restore.

Controllers in this repository

  • The types, clients and permissions here are what a container controller for AWS, GCP or Azure uses to schedule snapshots, report them and restore a volume.
  • The Kubernetes and local container controllers don't snapshot volumes. They report no volumes and no volumeBackups status. A restore request can only target a volume listed in ContainerOutputs.volumes, so it can't target these controllers.

Cloud clients

  • AWS: a Data Lifecycle Manager client, plus EC2 CreateSnapshot, DescribeSnapshots, DeleteSnapshot, CreateTags and DeleteTags. DescribeVolumes now reads attachment status.
  • GCP: resource policies (snapshot schedules), add and remove policies on disks, and snapshots.
  • Azure: Data Protection (Backup vaults, policies, backup instances, suspend and resume) and managed disk snapshots. The snapshot listing follows a nextLink only on the management endpoint's host, so the bearer token never goes to another host.

Permissions

  • container/provision grants the schedule and snapshot calls. AWS grants are bound to the deployment's resource tags, and IAM role grants to the <prefix>-dlm-* name with PassRole limited to dlm.amazonaws.com.
  • container/management grants only the status reads.
  • Installations set up before this change don't have these grants. The container keeps running, and its outputs report setupRequired until the setup is updated, for example with a CloudFormation stack update or terraform apply. The remote stack management controllers now record a revision of the management permissions they applied, so an Alien direct setup re-applies them when the permission set changes. A running direct install has no command yet that re-runs setup: alien deploy waits on the deployment lock.

Tests

  • Core, preflight, permission-set, cloud client and CLI unit tests.
  • Results in the sandbox: alien-core 633, alien-preflights 399, AWS clients 91, Azure clients 77, GCP clients 43, alien-permissions 33, remote stack management 15. All passed.
  • Scheduling, status, restore and delete were run live on AWS and GCP in a test account. Azure couldn't be run live: the test subscription has no VM sizes available in the regions with the machine image.

Generated files: packages/core/src/generated was regenerated with pnpm generate. The platform TypeScript SDK still needs regenerating with Speakeasy 1.680.11 (pnpm run generate:api-platform-sdk) before merge.

🤖 Generated with Claude Code

alongubkin and others added 2 commits October 5, 2026 15:54
Persistent volumes get a snapshot schedule that is on by default
(every 24 hours, kept 7 days) and can be tuned or turned off. The
schedule is validated against the limits every cloud's native
snapshot scheduler shares, and container outputs gain the volume
list with each volume's latest snapshot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@enclave-ai

enclave-ai Bot commented Oct 5, 2026

Copy link
Copy Markdown

Enclave skipped this draft pull request. It will run automatically when you mark the PR as Ready for review.

alongubkin and others added 27 commits October 5, 2026 16:04
A restore request names a container, a replica ordinal and a snapshot.
Controllers act on each request once and report the result per volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Volume backups on AWS are scheduled with Data Lifecycle Manager and
need snapshot create/list/delete plus tagging of existing volumes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…me backups

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CreateLifecyclePolicy has no resource type in IAM, so a policy ARN in the
statement would never match. Bound it by request tags instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Azure Disk Backup keeps operational snapshots for at most a year, so a
daily schedule kept 450 days would deploy on AWS and GCP but fail on
Azure. The portable limit is now 365 days.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n api

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uming backups

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A role that does not exist has no tags, so tag-conditioned IAM calls on
it are denied instead of reporting it missing. The role name now carries
a hash of the stack prefix and container ID, so the grant covers
{prefix}-dlm-*. Volume growth gets ec2:ModifyVolume, and the EC2 client
reads the attachment state from <status>, as EC2 sends it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alongubkin and others added 5 commits October 5, 2026 21:06
Installations set up before volume backups lack the management
permissions the schedule needs. Containers report that state instead
of failing, so it can be shown and fixed with a setup update.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Management permissions are installed by setup and a release update never
refreshes them. The remote stack management controllers now record a
revision of the permissions they applied, and an Alien direct setup
re-applies them when the revision changes. Runtime credentials never
trigger this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

@greptile-apps

greptile-apps Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[High risk] Adds volume backup and restore operations across cloud providers.

The PR appears safe to merge based on this review; no new actionable issue or outstanding previous finding remains.

Summary

The PR adds persistent-volume backup configuration, cloud clients and permissions, restore plumbing, and CLI commands. Since the previous review, it also restricts Azure snapshot pagination to the management endpoint and clarifies which controllers report volumes.

Reviews (4) · Last reviewed commit: "fix: check the host of Azure snapshot ne..."

Comment thread crates/alien-core/src/resources/container.rs
Comment thread crates/alien-core/src/deployment/config.rs
Comment thread crates/alien-infra/src/container/kubernetes.rs
Comment thread crates/alien-azure-clients/src/azure/snapshots.rs Outdated
Following a nextLink on another host would send it the management token.
Also documents that Kubernetes and local controllers report no volumes,
so they never get a backup status or a restore request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

1 similar comment
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

@alongubkin
alongubkin marked this pull request as ready for review October 6, 2026 03:35
@alongubkin
alongubkin merged commit 0c64c9f into main Oct 6, 2026
24 checks passed
@alongubkin
alongubkin deleted the alon/alien-1232-volume-backups branch October 6, 2026 05:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant