Skip to content

docs(setup): recovery for EntityAlreadyExists after a failed CloudFront VPC origin - #650

Open
vivibui wants to merge 2 commits into
aws-samples:mainfrom
vivibui:docs/vpc-origin-recovery
Open

vivibui wants to merge 2 commits into
aws-samples:mainfrom
vivibui:docs/vpc-origin-recovery

Conversation

@vivibui

@vivibui vivibui commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Issue #, if available: Refs #648 (workaround; the code fix is tracked there)

Description of changes:

Problem

If a CloudFront VPC origin create fails (for example in an account with VPC Block Public Access, #644), every later deployment fails with:

Error: creating CloudFront VPC Origin
… EntityAlreadyExists: Another vpc origin with the same name already exists within the aws account.

The failed origin is tainted, create_before_destroy makes Terraform create the replacement first, and VPC origin names are unique per account. Until someone deletes the failed origin by hand, the deployment stays stuck. #648 discusses a code fix.

Solution

A troubleshooting entry in docs/getting-started/setup.md with the recovery steps: read the origin ID from state, check that no CloudFront distribution references it, delete it with its ETag only if none does, wait for EntityNotFound, and re-run the deployment. Terraform then drops the missing origin from state and creates a fresh one. On a first deployment no distribution uses the failed origin yet. If one does, the steps refuse to delete it.

Testing

  • These are the steps we used to unblock a real deployment after a failed VPC origin create.
  • The origin-ID lookup and the distribution guard were run against a live stack: for an origin in use, the guard reports the distribution and does not delete it.
  • uv run zensical build: no issues.
  • npm run format, npm run lint and npm run test pass.

Note

#645 adds a troubleshooting entry at the same spot in setup.md (just above "Frontend shows authentication errors") and points readers to this entry, so please merge this PR first. I'll rebase #645 right after.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…nt VPC origin

A failed VPC origin create leaves the resource tainted. Because the
origin uses create_before_destroy and a fixed, account-unique name, the
replacement collides with the failed origin on every re-run. Document how
to delete the unreferenced failed origin and re-run the deployment until
the code fix lands.

Refs aws-samples#648
…re deleting it

The recovery steps deleted the failed VPC origin after a prose-only
condition ("when no distribution uses the failed origin"). Look up the
distributions that reference the origin first and delete it only when
there are none, then re-run the deployment as a separate step once the
delete has finished.

Refs aws-samples#648
@vivibui

vivibui commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Update from the review on #645: 6679d83 adds a guard before the delete. The steps now look up the distributions that reference the failed origin and delete it only when there are none, and the re-run is a separate step once the delete has finished. I checked the lookup and the guard against a live stack, where they report the referencing distribution and refuse to delete.

#645's Block Public Access entry now points readers here, so it would help to merge this one first. I'll rebase #645 on top right after.

@jeromevdl jeromevdl added area/docs Documentation and docs site size/XS < 50 changed lines (lockfiles excluded) labels Oct 30, 2026

@jeromevdl jeromevdl left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a separate documentation PR for this workaround? Could we put the recovery steps in #648, then include any recovery guidance still needed alongside the Terraform fix?

If we publish these steps, the distribution guard needs a correction: when an account has no CloudFront distributions, the query returns None, so [ -n "$USED_BY" ] incorrectly refuses deletion. That blocks the first-deployment case described here. Normalize the query result to an empty array and check command success before proceeding.

If the check reports a distribution, the origin is in use and deleting it would break that distribution, so leave it in place. Otherwise, wait until `aws cloudfront get-vpc-origin --id "$VPC_ORIGIN_ID"` returns `EntityNotFound` (several minutes), then re-run the deployment:

```bash
./scripts/deploy-terraform.sh dev

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Recovery can redeploy a different environment

If the failed deployment used prod or staging, the origin lookup above reads that environment’s initialized state, but this command switches Terraform to dev’s backend and variables. That can apply changes to dev while leaving the failed environment unrecovered. Using <environment> here and specifying that the retry must retain the original AWS profile, configuration paths and command-line overrides would keep recovery scoped to the affected installation. This also applies if these steps move into #645.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs Documentation and docs site size/XS < 50 changed lines (lockfiles excluded)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants