feat(ci3): diversify build-instance spot via EC2 create-fleet (v5-next) - #23966
Merged
Conversation
Build instances almost never won their spot bid. Two causes: the request hit a single pool (m6a.32xlarge, one AZ — Spot Placement Score 1/10 in us-east-2), and the on-demand fallback fired inside the first pool's call, so the m6a/m7a/m7i loop never actually diversified. Replace the legacy request-spot-instances call with a single `aws ec2 create-fleet --type instant` per request: - capacity-optimized-prioritized across an expanded, memory-safe (>=4 GB/vCPU, m+r families) pool set spanning 192/128/64-vCPU sizes and all AZs. Priority prefers 128, then 192, then 64 (c-series is opt-in via CI_SPOT_INCLUDE_C pending build-parallelism throttling — measured peak ~1.9 GB/vCPU). - spot attempts retry within a CI_SPOT_TIMEOUT budget (default 60s), preserving the historical "try spot ~1 min then on-demand" behaviour. - on-demand create-fleet (lowest-price) only after the spot budget expires. - an ephemeral launch template carries the common config and is deleted on exit; --type instant leaves no spot request or fleet to clean up. Validated live: spot, spot-retry and on-demand paths all acquire and tear down cleanly with no leaked instances, templates or fleets. Requires the matching pipeline-exec IAM policy (AztecProtocol/github-aws-oidc-factory#21) to be applied first.
Cosmetic only — the override list was built by appending " $t:$prio", leaving a leading space that surfaced as a leading comma in the "(pools: …)" log line. Word-splitting already discarded it when parsing, so the fleet JSON was always correct (override count unchanged).
charlielye
marked this pull request as ready for review
June 9, 2026 14:36
ludamad
approved these changes
Jun 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Draft until the IAM policy is applied. Depends on AztecProtocol/github-aws-oidc-factory#21 (grants the
pipeline-execroleec2:CreateFleet/CreateLaunchTemplate/DeleteLaunchTemplate+ describe/launch-template perms). Without it, provisioning hitsUnauthorizedOperation. Marking ready once that policy is live.v5-next port of #23963 (identical ci3 changes; the three files match between
nextandv5-next).Problem
Build instances almost never won their spot bid. Two root causes:
aws_request_instanceloopedm6a → m7a → m7i, butaws_request_instance_typefell back to on-demand inside the first pool's call, so m6a.32xlarge spot failed → m6a on-demand succeeded → returned. m7a/m7i spot was never tried (andm7i.32xlargedoesn't even exist).m6a.32xlargein us-east-2 is 1/10 in every AZ.Change
Replace the legacy
request-spot-instancescall with a singleaws ec2 create-fleet --type instantper request:capacity-optimized-prioritizedacross an expanded pool set spanning 192/128/64-vCPU sizes and all AZs.Priorityprefers 128 → 192 → 64; 64 is still taken as spot before any on-demand.CI_SPOT_INCLUDE_C.CI_SPOT_TIMEOUTbudget (default 60s), preserving the historical "try spot ~1 min, then on-demand" behaviour.lowest-pricecreate-fleet) only after the spot budget expires.--type instantleaves no spot request or fleet to clean up.bootstrap_ec2now requestscores=192,128,64.Files
ci3/aws_request_instance— override-list builder (size→priority, family pools,AWS_INSTANCE/CI_SPOT_INCLUDE_C).ci3/aws_request_instance_type— create-fleet provisioning (LT, subnet lookup, spot retry loop, on-demand fallback, markers, LT cleanup); IP-wait/SSM/SSH tail unchanged.ci3/bootstrap_ec2—cores=192,128,64.Validation
Exercised live against us-east-2 (on the
nextbranch): create-fleet returned a spotm7i.48xlargeimmediately; real script end-to-end acquired spot via the retry loop, wrote markers, reached SSMOnline, deleted the launch template, clean exit; on-demand fallback (NO_SPOT=1) marksondemand; no leaked instances, templates, or fleets.