[vivibui]

Issue #, if available: Fixes #644

Description of changes:

Problem

In an account with VPC Block Public Access (BPA) enabled, the first apply fails about 25 minutes in, when the CloudFront VPC origin for the internal Yjs ALB is created:

unexpected state 'Failed : You can't update an endpoint group that has
endpoints in a subnet with Block Public Access enabled. ...', wanted target 'Deployed'

CloudFront VPC origins can't use subnets that BPA covers, and the stack has no way to exclude the ALB's subnets. Details in #644.

Solution

• New root variable vpc_block_public_access_exclusion (bool, default false), passed to the networking module.
• When true, module.networking creates:
  • aws_vpc_block_public_access_exclusion.private: allow-bidirectional for the two private subnets.
  • aws_vpc_block_public_access_exclusion.nat: allow-egress for each NAT gateway's subnet (1 outside prod, 2 in prod). In block-bidirectional mode BPA also blocks NAT gateways, so without this the private subnets would lose egress. It has no effect in block-ingress mode.
  • The VPC itself is never excluded, and no public subnet gets inbound access.
• New networking output vpc_origin_subnet_ids: the same IDs as private_subnet_ids, with depends_on both exclusions. module.yjs_server now takes its subnets from it, so the ALB and the VPC origin are created, and the Yjs task pulls its image, only after the exclusions exist.
• dev.tfvars.example documents the variable, and docs/getting-started/setup.md gets a troubleshooting entry for the error above. If the deployment already failed, the entry first sends readers to the recovery steps for the tainted VPC origin (docs(setup): recovery for EntityAlreadyExists after a failed CloudFront VPC origin #650, [Bug]: A failed CloudFront VPC origin create can't be retried (EntityAlreadyExists on replacement) #648) before they re-run.

Security notes

• Opt-in. With the default false, nothing is created and the plan doesn't change.
• The private subnets' only default route is the NAT gateway, with no internet gateway route, so excluding them adds no inbound path from the internet.
• The Yjs ALB is internal = true, and its security group admits only the CloudFront origin-facing managed prefix list.
• allow-bidirectional is the only exclusion mode that lifts ingress blocking, which the VPC origin needs. The NAT gateway subnets get only allow-egress, which keeps inbound internet traffic blocked and grants nothing beyond what block-ingress mode already allows.
• Reachability is the same as the intended design without BPA: CloudFront → VPC origin → internal ALB.

Testing

• New test in scripts/test/terraform-hardening.test.mjs pins the false default, both exclusions (subnets, modes, counts), and the ordering through vpc_origin_subnet_ids. node --test scripts/test/terraform-hardening.test.mjs: 9 passed.
• A throwaway mock-provider terraform test plan of the networking module gave 2 private + 1 NAT exclusions in dev, 2 NAT exclusions in prod, and none with the default false.
• terraform fmt -check -recursive, terraform validate -no-tests, and the auth-domain, waf, yjs, managed-environments and lambda-packaging terraform test suites pass locally (Terraform 1.12.1).
• npm run format, npm run lint and npm run test pass.
• Live, block-ingress account (first commit): the private-subnet exclusions were created in 4–5 minutes, the VPC origin reached Deployed, the Yjs target group was healthy, the AgentCore runtime was READY, and Yjs and realtime WebSockets connected through CloudFront → VPC origin → internal ALB → ECS. In a second account without BPA, the default false produced no plan diff.
• Live, block-bidirectional account (752575e, fresh deploy): all three exclusions active; VPC origin Deployed; Yjs task healthy; AgentCore READY; the collaborative editor, a CodeCommit-backed space and an agent intent (Claude Code, bugfix) ran through reverse-engineering to the first gate and on to requirements-analysis. The AgentCore runtime's own subnets need no exclusion: they egress through the excluded NAT subnet.
• Negative control in the same account: deleting only the NAT egress exclusion made VPC Lambda calls to STS time out (API Gateway 504) and new Yjs tasks fail with ResourceInitializationError … unable to retrieve secrets from ssm. Re-applying Terraform recreated just that exclusion and everything recovered.

Impact

None unless the variable is set to true. Moving module.yjs_server to the new output doesn't change any subnet IDs, so existing deployments see no diff.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[jeromevdl]

@vivibui thanks for this PR. The approach makes sense to me: CloudFront VPC origins are blocked by BPA, and allow-bidirectional on the private subnets is the only way to let CloudFront in.

Did you test the application after the deployment (opening a document in the collaborative editor, running an agent stage), and which BPA mode was the account in (block-ingress or block-bidirectional)?

Looking at the docs, I think the fix only fully works in block-ingress mode:

• In ingress-only mode, outbound traffic through NAT gateways is allowed, so the private subnets keep their internet egress.
• In bidirectional mode, the service impact list says "NAT gateway: All inbound and outbound traffic is blocked". Our NAT gateway is in the public subnets, which this PR intentionally doesn't exclude. The advanced example (scenario 6) shows private-subnet instances losing internet access in that setup.

Since we have no VPC endpoints, the Yjs ECS task (ECR pull, Cognito JWKS), the VPC Lambdas and the AgentCore runtime in VPC mode all rely on the NAT gateway. In a block-bidirectional account, I'd expect the VPC origin to deploy but the application not to work.

I think we would need to add an allow-egress exclusion for the NAT gateway subnets when the flag is on. Egress-only exclusions keep inbound internet traffic blocked. The public subnets contain only the NAT gateways, so this grants nothing beyond what block-ingress mode already allows, and it's ignored in block-ingress mode. Something like:

resource "aws_vpc_block_public_access_exclusion" "nat" {
  count = var.vpc_block_public_access_exclusion ? length(aws_nat_gateway.main) : 0

  subnet_id                       = aws_nat_gateway.main[count.index].subnet_id
  internet_gateway_exclusion_mode = "allow-egress"
}

with vpc_origin_subnet_ids depending on it too, so the Yjs task can pull its image on the first apply.
Could you verify in a block-bidirectional account that the app works end to end? If you'd rather scope this PR to block-ingress only, then please say so in the docs and variable descriptions instead but I'd recommend to support both if we can.

[eipasteur]

The two points still need addressing before merge: the BPA modes this option supports, as Jérôme noted, and recovery after the failed VPC origin creation shown in the new troubleshooting entry.

[eipasteur]

[P2] Retrying the failed VPC origin still hits a name collision

This entry starts after VPC origin creation has already failed, but enabling the flag and rerunning does not recover the tainted origin. Its fixed name and create_before_destroy cause the replacement to hit EntityAlreadyExists, as tracked in #648. Connecting these instructions to the recovery procedure in #650 would let users complete the retry; that procedure also needs its guard that no distribution references the failed origin.

[vivibui]

Good catch, thank you! That's exactly what happened on our first deployment: enabling the flag and re-running hit EntityAlreadyExists until we deleted the failed origin by hand.

fea3239 adds a step to this entry: the failed create leaves the VPC origin tainted, so before re-running, follow Re-running after a failed VPC Origin fails with EntityAlreadyExists, the recovery entry from #650. #650 (6679d83) now has the guard you asked for. It looks up the distributions that reference the origin (list-distributions filtered on VpcOriginConfig.VpcOriginId) and deletes the origin only when there are none. The re-run is a separate step once the delete has finished. I ran the lookup and the guard against a live stack: for an origin a distribution uses, it reports the distribution and refuses to delete.

The two entries sit at the same spot in setup.md, so the reference resolves once #650 is in. If you merge #650 first, I'll rebase this PR on it straight away so the two entries end up next to each other.

[vivibui]

Thank you @jeromevdl and @eipasteur! Both points are addressed now (752575e for the BPA modes, fea3239 for the retry path, details in the inline thread). @jeromevdl, you were right about bidirectional mode, and a live test confirms it.

Original test. The first account was in block-ingress mode (ExclusionsAllowed: allowed). There the VPC origin reached Deployed, the Yjs target group was healthy, the AgentCore runtime was READY, and the Yjs and realtime WebSockets connected through CloudFront → VPC origin → internal ALB → ECS. I hadn't opened a document in the editor UI or run an agent stage in that account.

The fix. 752575e follows your suggestion:

• aws_vpc_block_public_access_exclusion.nat: one allow-egress exclusion per NAT gateway subnet (length(aws_nat_gateway.main), so 1 outside prod and 2 in prod).
• vpc_origin_subnet_ids now depends on both exclusions.
• The variable descriptions, dev.tfvars.example and the setup troubleshooting entry cover both modes, and the hardening test checks both exclusions (subnets, modes, counts) and the output's dependencies.

Live test in block-bidirectional mode. I deployed 752575e from scratch into an empty region of a test account after switching it to block-bidirectional, with vpc_block_public_access_exclusion = true. The plan was 1,746 to add, 0 to change, 0 to destroy.

• All three exclusions became active in under 3 minutes. Effective subnet modes: private subnets off, the NAT subnet block-ingress, everything else block-bidirectional.
• The VPC origin reached Deployed, the Yjs task started and its target group was healthy, and the AgentCore runtime was READY.
• In the app: sign-in, the Admin → Agents platform Bedrock key, a CodeCommit-backed space (role assumed, file tree listed), and live editing on the collaborative Compose page all worked.
• An intent (scope bugfix, Claude Code) cloned the repository in the AgentCore runtime and ran workspace-scaffold, workspace-detection, state-init and reverse-engineering to the first validation gate, with about 55k input and 5k output tokens through Bedrock and nine artifacts produced. After approval, it continued to requirements-analysis.

One thing I wanted to check: the AgentCore runtime uses its own subnets, which have no exclusion and stay block-bidirectional. They still reach CodeCommit and Bedrock through the NAT gateway once its subnet has the egress-only exclusion, so no extra exclusion is needed for them.

Negative control. I then deleted only the NAT subnet's egress exclusion, which leaves the stack as it was with the first commit:

• Re-binding the space's repository (a VPC Lambda calling STS) and starting an intent both returned API Gateway 504 after 29 s.
• A new Yjs task failed three times with TaskFailedToStart: ResourceInitializationError: unable to pull secrets or registry auth: unable to retrieve secrets from ssm … context deadline exceeded, and the deployment rolled back.

The VPC has S3 and DynamoDB gateway endpoints, but SSM, the ECR API, STS, Cognito, CodeCommit and Bedrock all go through the NAT gateway, so this is the failure you predicted. Re-applying Terraform recreated only the exclusion, and every check passed again, including the Yjs service recovering on its own.

Screenshots of the collaborative editor, the agent settings and the intent run follow in the next comment.

[vivibui]

Screenshots from the block-bidirectional test (752575e, vpc_block_public_access_exclusion = true; hostnames, account IDs and emails masked):

Collaborative Compose page: live edit through CloudFront → VPC origin → internal ALB → Yjs (status bar: Connected)

Admin → Agents: platform Bedrock connection ready

Intent run: Inception completed (reverse engineering + requirements analysis), waiting at the requirements review gate

[vivibui]

@eipasteur @jeromevdl let me know once either of you have reviewed and if further changes needed

[jeromevdl]

As mentioned in the #650, I would merge the 2 PRs (add the 650 here).

Minor: line 787 says to re-run before line 793 says not to. Please adjust the order, something like: "set the following, delete the failed VPC origin (see …), then re-run."