feat(ci3): diversify build-instance spot via EC2 create-fleet

[charlielye]

Important

Draft until the IAM policy is applied. Depends on AztecProtocol/github-aws-oidc-factory#21 (grants the pipeline-exec role ec2:CreateFleet/CreateLaunchTemplate/DeleteLaunchTemplate + the describe/launch-template perms). Without it, provisioning hits UnauthorizedOperation. Marking ready once that policy is live.

Problem

Build instances almost never won their spot bid. Two root causes:

1. Diversification never ran. aws_request_instance looped m6a → m7a → m7i, but aws_request_instance_type fell back to on-demand inside the first pool's call, so m6a.32xlarge spot failed → m6a on-demand succeeded → returned. m7a/m7i spot was never tried (and m7i.32xlarge doesn't even exist).
2. A single pool is the worst case. AWS Spot Placement Score for m6a.32xlarge in us-east-2 is 1/10 in every AZ — one instance type, effectively one AZ, with a sub-on-demand per-cpu bid.

Change

Replace the legacy request-spot-instances call with a single aws ec2 create-fleet --type instant per request:

• capacity-optimized-prioritized across an expanded pool set spanning 192/128/64-vCPU sizes (48xl/32xl/16xl) and all AZs (subnet overrides in the SG's VPC). Priority prefers 128 → 192 → 64; 64 is still taken as spot before any on-demand.
• Memory-safe families only (m + r, ≥4 GB/vCPU). Measured build peak is ~1.9 GB/vCPU (driven by the parallel build fan-out), so c-series (2 GB/vCPU) is excluded by default and gated behind CI_SPOT_INCLUDE_C pending build-parallelism throttling.
• Spot retry loop within a CI_SPOT_TIMEOUT budget (default 60s, CI_SPOT_POLL interval), preserving the historical "try spot ~1 min, then on-demand" behaviour; each retry re-samples the whole diversified pool.
• On-demand fallback (lowest-price create-fleet) only after the spot budget expires.
• An ephemeral launch template carries the common config (AMI/SG/instance-profile/EBS + launch-time tags) and is deleted on exit. --type instant leaves no open spot request or persistent fleet, so teardown stays iid-based.

bootstrap_ec2 now requests cores=192,128,64.

Files

• ci3/aws_request_instance — rewritten as the override-list builder (size→priority, family pools, AWS_INSTANCE/CI_SPOT_INCLUDE_C handling).
• ci3/aws_request_instance_type — create-fleet provisioning (LT, subnet lookup, spot retry loop, on-demand fallback, markers, LT cleanup); IP-wait/SSM/SSH tail unchanged.
• ci3/bootstrap_ec2 — cores=192,128,64.

Validation

Exercised live against us-east-2:

• Builder emits correct tokens/priorities across amd64, arm64, CI_SPOT_INCLUDE_C, AWS_INSTANCE, and the uncurated cpus=4 path.
• create-fleet returned a spot m7i.48xlarge immediately (48 overrides, valid config).
• Real script end-to-end: spot acquired via the retry loop, markers written, SSM Online, launch template deleted, clean exit.
• On-demand fallback (NO_SPOT=1) acquires and marks ondemand.
• Teardown: no leaked instances, launch templates, or fleets.