Skip to content

feat(sandbox): deploy AWS sandboxes through direct deployment - #716

Merged
ItamarZand88 merged 113 commits into
mainfrom
itamar/alien-966-make-aws-sandboxes-work-through-direct-deployment-alien
Sep 25, 2026
Merged

ItamarZand88 merged 113 commits into
mainfrom
itamar/alien-966-make-aws-sandboxes-work-through-direct-deployment-alien

Conversation

@ItamarZand88

@ItamarZand88 ItamarZand88 commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

AWS sandboxes can now be deployed with alien-deploy, the direct path with no Terraform or CloudFormation. Setup creates everything the runtime identity is never allowed to create itself, and the runtime builds only what it is granted.

Step-by-step flow when alien-deploy deploy sets up a stack with a sandbox:

  1. Setup, using the deployer's credentials, creates the sandbox's build role. For egress: deny it also creates an operator role, a loopback-only security group and a network connector. It records each object and passes them to the sandbox's controller. ← the heart
  2. A Frozen sandbox's image is built during setup and rolled once when setup runs again with a new bundle. A Live sandbox's image is built later by the runtime, which passes the setup-made role.
  3. Destroy removes the images, then everything setup created, in dependency order.

This PR changes AWS sandboxes from template-only to deployable directly, with the same objects the templates create.

What was broken

alien-deploy could not deploy an AWS sandbox. The build role, the deny egress objects and the remote-access grant exist only in the Terraform and CloudFormation setups. The runtime identity is deliberately never given iam:CreateRole, so it cannot make them.

What I did

  • Setup scaffolding. A setup step creates the build role, and for deny egress an operator role, a loopback-only security group and a network connector (through Cloud Control). It also writes a published sandbox's remote-access grant onto the Remote Bindings role. Each object is tagged, recorded in the deployment's runtime metadata and verified before it is adopted.
  • Frozen sandboxes on the direct path. Setup builds the image. A setup re-run with a new bundle re-grants the role and rolls the image once. The runtime never builds or deletes a Frozen image.
  • Guards. Two Deny statements on the management identity block it from writing to setup's sandbox roles and to its own role. The drift check lets an update add either guard, since a Deny only narrows the identity, but not remove one. An earlier version must accept stacks carrying them, and a separate PR (fix(preflights): accept the management role guards in either direction #718) adds that tolerance on main. It has to land first.
  • Updates that need setup are refused. A runtime update that would change scaffolding (a new sandbox, a different egress, network or build policy, or publishing and unpublishing) is refused with the reason, until setup runs again.
  • Teardown. It deletes the image, then the connector, the security group (after the connector's network interfaces are released), the roles and the grant. Anything already gone is skipped.
  • Destroy after a failed setup. When setup fails before the management identity exists, and nothing runtime cleanup acts on has come up, the manager skips runtime cleanup. The destroy goes straight to setup teardown. Before this, it ended in delete-failed, and the build role stayed in the account.

Files touched

  • crates/alien-infra/src/setup_scaffolding/: the scaffolding step, AWS sandbox roles and egress, seeding and teardown.
  • crates/alien-infra/src/sandbox/aws.rs: Frozen build, roll and ownership rules on the direct path.
  • crates/alien-deployment/src/: runs the scaffolding in initial setup, refuses updates that need setup, and tears scaffolding down.
  • crates/alien-core/src/: the runtime-metadata record and the build role naming.
  • crates/alien-permissions/permission-sets/: the two management guards.
  • crates/alien-preflights/src/: the build role name budget and the guard exemption.
  • crates/alien-aws-clients/src/: the Cloud Control, EC2 and IAM calls the scaffolding needs.
  • CloudFormation and Terraform generator snapshots, updated for the guards.

How I tested

  • Manually (us-west-2): did a direct deployment with the local control plane, using a Live and a Frozen sandbox with open egress.
    • Setup created both build roles and built the Frozen image under setup tags.
    • A remote-access session on the Live sandbox started in 2.4s and ran a probe.
    • Setup re-run with a new Frozen bundle rolled that image exactly once.
    • Added Live and Frozen deny sandboxes. Setup created their operator roles, loopback-only groups and ACTIVE connectors. Sessions started and TCP out was blocked.
    • alien-deploy destroy removed every object setup and the runtime created. The account inventory matched its pre-run baseline.
    • With a standalone manager, alien deploy failed in setup after creating the build role. alien destroy then reached deleted and removed the role.
  • Unit tests: alien-infra 741, alien-deployment 141, alien-manager 263, plus preflights and core. Tests use strict mocks for every IAM, EC2, Cloud Control and MicroVM call, and the behavioural fixes are mutation-tested.
  • Anything I couldn't test:
    • Deny sessions were started with admin credentials; the compute grant comes from sandbox/management and was checked by configuration only.
    • Under egress: deny, DNS still resolves, because the security group doesn't cover the VPC resolver. That gap also exists in the template setups and is tracked separately.

I also ran a security review on the diff. What it checked:

  • The runtime minting a role: sandbox/provision grants iam:PassRole on the setup-made role only and never iam:CreateRole. The management guards deny writes to setup's sandbox roles and to the management role (aws_management_role_guards snapshot).
  • Adopting a role someone else made: a same-named role is adopted only if it carries setup's tags and its trust, inline and attached policies match exactly. Otherwise setup refuses (aws_sandbox.rs adoption tests).
  • Leaving a remote grant behind: unpublishing requires setup, and setup revokes the grant even when the stack no longer declares the Remote Bindings role.
  • A destroy that skips cleanup it owes: the rule returns nothing once a Live resource, a synced secrets vault or a Frozen type with runtime cleanup has come up. It also returns nothing on any resource without a lifecycle (deleting.rs tests).
  • Deny egress holes: the group has no ingress and egress only to 127.0.0.1/32, and a tagged group that allows more is repaired or refused (aws_sandbox_egress.rs tests).
    Nothing turned up.

Depends on #700 (base), #715 (image tagging) and the #712 → #713 → #714 chain (image read, build inputs, teardown). This branch merges #714 and #715 and will rebase once they land.

Three tests asserted the bare image name, and one matched with `ends_with`,
which passes for a name and an ARN alike. They fail against the current code.
The pre-create probe passed the derived image name where the API takes an
ARN, so AWS answered 400 Invalid ARN format. Only a 404 counts as absence,
so the create was unreachable and every Live sandbox retried until it gave
up. The two delete paths passed the same bare name on their fallback, which
left an unrecorded image behind instead of sweeping it.

The name is account-unique, so its ARN is derivable before the image exists.
The executor rebuilds a resource through `new_pending` when a failed create's
config changes, which keeps no controller state. This starts from that state and
fails against the current code, which reads the build inputs back from what
setup registered.
A create that restarts after a failure comes back with a default controller, so
reading the build role and bundle from what setup registered left it holding
neither and failing on its own guard. Nothing could restore them, and the
read-before-create adopt that makes the restart safe sits behind those guards.

Both are functions of the desired config and the deployment's own names. The
build role name is the one both generators emit and the one `iam:PassRole` is
already scoped to, and a preflight refuses any id long enough to have it
clamped. The recorded bundle stays where it belongs, as the last-applied value a
roll diffs against.
Teardown deleted every version and then the image. AWS refuses to delete an
image's last version — "This is the last version. Please delete the entire
image" — so every sandbox that built once and was never rolled failed to tear
down, burned its retry budget, and left the customer's stack undeletable.

The sweep was there because the image delete was believed to remove nothing
while versions remained. It does remove them: against the live API, deleting an
image carrying two versions left neither behind. So the sweep goes rather than
being taught which version to skip.

The retired-version reaper is untouched. It deletes versions a roll replaced
while the image keeps serving, which is not what teardown does.
… names

A grant keyed on aws:ResourceTag is bounded only while the resource is also
pinned by name; otherwise whoever can set the tag widens their own reach. Every
Lambda statement satisfies this today, so this passes as written and fails the
moment one stops.
CreateMicrovmImage sends the tags every other sandbox statement scopes against,
and AWS authorizes the implied TagResource the way it authorizes the create:
against no resource type. The grant sat on the image ARN instead, which cannot
match an image that does not exist yet, so the build was refused with a 403 and
no Live sandbox on AWS could ever be provisioned.

The tag now travels with the create it belongs to, bounded by the same request
tags. On "*" that is safe only while a forged tag reaches nothing, which the
name-pinning test holds.
@greptile-apps

greptile-apps Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Critical risk] Rebuilds AWS sandbox deployment and infrastructure setup.

The PR appears safe to merge based on this review; no new actionable issue or outstanding previous finding was identified.

Summary

The PR adds direct AWS sandbox setup, image lifecycle management, permission guards, and ordered teardown. Changes since the previous review also add application-image observations and adjust stable-release publishing and resume handling.

  • Direct setup creates and records sandbox roles and deny-egress resources.
  • The operator can report observed Kubernetes charts and container images.
  • Stable publication now uploads qualified Python wheels from the top-level workflow.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  D[Deployer credentials] --> S[Sandbox setup]
  S --> R[Build role and optional deny-egress resources]
  S --> F[Frozen image build]
  R --> L[Live runtime build and sessions]
  F --> T[Destroy: images, connector, group, roles and grant]
  L --> T
Loading

Reviews (6) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..."

Comment thread crates/alien-preflights/src/compatibility/permission_profiles_unchanged.rs Outdated
Comment thread crates/alien-infra/src/setup_scaffolding/aws_sandbox.rs
…ith derived inputs

The restart test covered only an image that was never built. A create whose
response and state were both lost finds its image and must adopt and roll it
with the recomputed role and bundle.
Teardown deleted every version and then the image. AWS refuses to delete an
image's last version — "This is the last version. Please delete the entire
image" — so every sandbox that built once and was never rolled failed to tear
down, burned its retry budget, and left the customer's stack undeletable.

The sweep was there because the image delete was believed to remove nothing
while versions remained. It does remove them: against the live API, deleting an
image carrying two versions left neither behind. So the sweep goes rather than
being taught which version to skip.

The retired-version reaper is untouched. It deletes versions a roll replaced
while the image keeps serving, which is not what teardown does.
…re it is described

Two comments still claimed the image delete removes nothing while versions
remain, which is what the removed sweep was built on. The CloudControl probe
keeps its claim, scoped to CloudControl.
@ItamarZand88

Copy link
Copy Markdown
Contributor Author

@greptile review

Base automatically changed from itamar/alien-941-make-the-sandbox-image-follow-the-worker-build-and-release to main September 24, 2026 18:39
…amed

The tagged CreateMicrovmImage performs an implicit TagResource that AWS
authorizes against no resource, so the grant sat on "*". That also let the
identity tag any existing Lambda resource, with any extra keys, as long as
the request carried this stack's tags.

The tag now binds to NotResource of every Lambda ARN, which leaves only the
resourceless check of a create. Permission sets gain notResources for this,
and the build refuses it on anything but a request-tag-bound TagResource.
Verified live: the tagged create succeeds, and a direct TagResource on an
image or a function is refused.
@ItamarZand88

Copy link
Copy Markdown
Contributor Author

@greptile review

…ion-sweep-cannot' into itamar/alien-966-make-aws-sandboxes-work-through-direct-deployment-alien
…rce-to-an-arn-so-tag' into itamar/alien-966-make-aws-sandboxes-work-through-direct-deployment-alien
A caller creating the AWS sandbox image-build role through the IAM API
needs the same inline policy and trust policy the CloudFormation and
Terraform emitters write. SandboxBuildRole resolves both from concrete
values (partition, account, region, the declared bundle URI, and
whether the image is runtime-built), refusing what the emitters refuse.

Parity tests render each emitter for Frozen and Live sandboxes, with
and without the {region} token, resolve the template intrinsics to
fixed values, and require the result to equal the builder's output.
The adopt check filters inline policies by name. The existing test paired the
build policy with a foreign one, so a count check would have passed it while
adopting a role that carries only someone else's policy.
Two #[test] attributes had stacked on one function, leaving
state_written_before_versions_were_tracked_keeps_its_binding with none. It never
ran, and its neighbour ran twice. It passes once restored.
The adopt refusal was tested in the scaffolding module alone. Nothing proved
setup stops on it, so treating the error as done would have handed a refused
role to the controller unnoticed.
Create, read, delete and list any CloudFormation-registry resource type
through Cloud Control (AWS JSON 1.0, target CloudApiService, signing
name cloudcontrolapi), and read a request's progress. Resource documents
travel as JSON strings. A FAILED or cancelled ProgressEvent maps to a
typed error carrying AWS's handler code and status message, with
NotFound and AlreadyExists mapped as their synchronous refusals are.
A direct setup creating a deny sandbox's egress objects through the IAM
and Cloud Control APIs needs what the emitters write: the operator
role's trust and sandbox-egress-connector policy, and the network
connector's properties. Names for the operator role, deny group and
connector come from one place, and the tags setup puts on what it
creates move to alien-core, where the connector reads them.

Parity tests resolve the CloudFormation operator role and connector
(created and bring-your-own VPC) and the Terraform operator role to
fixed values and require them to equal the builders' output.
A rule naming a prefix list carries no CIDR, so a reader that dropped
prefixListIds would see a group reaching nothing where it reaches a
whole AWS service. The deny-group check needs to see them.
For egress: deny, direct setup now creates what the templates do: the
operator role (sandbox-egress-connector policy), a security group in
the stack network's VPC permitting egress only to 127.0.0.1/32, and the
AWS::Lambda::NetworkConnector through Cloud Control with the same
desired state CloudFormation emits. Each call makes at most one
mutating call, between verifying the build role and applying its
policy, and records each object as soon as it exists.

EC2's default allow-all egress is revoked from a group setup recorded;
a same-named group it did not record is adopted only when its egress is
exactly loopback, and a same-named operator role only after the build
role's checks. A connector created before a crash is found again by its
unique name rather than created twice. Deny on a network with no
private subnets, and allowDomains, are refused as the emitters refuse.

Teardown deletes the connector, then the group, then the operator
role, before the build role. A group AWS has not released yet is
retried on a later call, and Frozen teardown waits for it.
…the bundle

A Frozen sandbox direct setup registered owned its builds whoever stepped
it, so the runtime identity could roll it through a Live sibling's
provision grant. Its builds now need direct-setup authority. A roll waits
until setup has granted the build role the new bundle, since that policy
names one object; setup's roll reaps the versions earlier rolls retired;
and the image is created with setup's tags, as the templates tag it.
Recovery runs on every teardown pass and derived the image from its name,
so an image teardown had already deleted came back into the record and was
deleted again on each pass of a deny sandbox's longer teardown.
…direct setup

The template setups attach the sandbox/remote-execute grant to the Remote
Bindings role; direct setup never did, so a published sandbox loaded its
binding but could not start a session. Setup scaffolding now writes the
grant and revokes it on unpublish, and publishing or unpublishing requires
setup. The Remote Bindings delete tolerates a policy already removed.
…ion-sweep-cannot' into itamar/alien-966-make-aws-sandboxes-work-through-direct-deployment-alien
The raw JSON values reordered their keys whenever a workspace build turned on
serde_json's preserve_order, so the snapshot passed alone and failed in CI.
…rce-to-an-arn-so-tag' into itamar/alien-966-make-aws-sandboxes-work-through-direct-deployment-alien
Comment thread crates/alien-permissions/permission-sets/sandbox/provision.jsonc Outdated
@ItamarZand88
ItamarZand88 force-pushed the itamar/alien-966-make-aws-sandboxes-work-through-direct-deployment-alien branch from 9b65f73 to 9b31f4e Compare September 24, 2026 20:33
An exclusion fails open: NotResource of arn:aws:lambda excludes nothing in
GovCloud or China, where the grant would again reach every Lambda resource.
The build now requires a wildcard partition on notResources.
…rce-to-an-arn-so-tag' into itamar/alien-966-make-aws-sandboxes-work-through-direct-deployment-alien
@ItamarZand88

Copy link
Copy Markdown
Contributor Author

@greptile review

…ment identity existed

Runtime cleanup needs the management identity to reach the target, so a
destroy after setup failed on it ended in delete-failed and never reached
setup teardown, leaving the sandbox build role and egress objects behind.
With no identity outputs and no Live resource provisioned, the runtime never
started: the manager now moves such a destroy, or a retried failed one,
straight to setup teardown.
A sandbox deployment prepared before either role guard existed gains both on
its next update, and only the management-role guard was exempt, so the update
was refused as a management permissions change. Both guards are Denies that
only narrow the identity; removing either is still refused.
A retried teardown of a record without stack state checkpointed
TeardownRequired and cleared the earlier error before failing.
…ild role and bundle

Also runs the persisted-state test that lacked its #[test] attribute.
…reate-is-unrecoverable' into itamar/alien-957-sandbox-teardown-fails-on-aws-the-version-sweep-cannot
…ion-sweep-cannot' into itamar/alien-966-make-aws-sandboxes-work-through-direct-deployment-alien

# Conflicts:
#	crates/alien-infra/src/sandbox/aws.rs
…me up

A stack prepared with no management configuration has no management identity
at all, so its absence alone did not show the runtime never ran: a secrets
vault or compute cluster setup brought up still needs its runtime cleanup.
The destroy now goes straight to setup teardown only when no Live resource,
synced secrets vault or Frozen type with a runtime share of its delete came
up, and fails closed on a resource with no lifecycle. A destroy that lands on
Deleted this way also revokes the deployment's registry access.
…ion-sweep-cannot' into itamar/alien-966-make-aws-sandboxes-work-through-direct-deployment-alien
@ItamarZand88

Copy link
Copy Markdown
Contributor Author

@greptile review

…-aws-sandboxes-work-through-direct-deployment-alien

# Conflicts:
#	crates/alien-infra/src/sandbox/aws.rs
#	crates/alien-permissions/tests/aws_runtime.rs
#	crates/alien-preflights/src/compatibility/permission_profiles_unchanged.rs
…-aws-sandboxes-work-through-direct-deployment-alien

# Conflicts:
#	crates/alien-infra/src/sandbox/aws.rs
…-aws-sandboxes-work-through-direct-deployment-alien
@ItamarZand88

Copy link
Copy Markdown
Contributor Author

@greptile review

@ItamarZand88
ItamarZand88 merged commit f32aef2 into main Sep 25, 2026
29 checks passed
@ItamarZand88
ItamarZand88 deleted the itamar/alien-966-make-aws-sandboxes-work-through-direct-deployment-alien branch September 25, 2026 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant