fix(ci3): cache DNS on build instances to dodge link-local PPS throttling - #24105
Merged
Merged
Conversation
…ling CI's parallel jobs (devbox + nested docker-in-docker) resolve the same hosts (S3, Docker Hub, npm, cargo, github) thousands of times, each lookup going straight to the VPC resolver. AWS caps traffic to the link-local resolver at ~1024 pps per ENI; over that, packets are silently dropped and show up as "could not resolve host" (e.g. the chonk-inputs S3 download failures). Route container DNS through the host's caching systemd-resolved: expose its stub on the instance's primary private IP (the one address reachable from the devbox container and nested dind) via DNSStubListenerExtra, and point containers at it with `docker run --dns`. Repeat lookups become cache hits and never reach the throttled resolver. Fails safe: if the IP can't be derived, systemd-resolved isn't active, or the stub doesn't come up, priv_ip is cleared and --dns is omitted, leaving DNS exactly as before. No counters/measurement — treating the throttling as the known cause.
ludamad
approved these changes
Jun 15, 2026
ludamad
added this pull request to the merge queue
Jun 15, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jun 15, 2026
alexghr
added this pull request to the merge queue
Jun 16, 2026
Collaborator
Flakey Tests🤖 says: This CI run detected 1 tests that failed, but were tolerated due to a .test_patterns.yml entry. |
AztecBot
pushed a commit
that referenced
this pull request
Jun 24, 2026
…ling (#24105) ## Problem CI DNS failures (`curl: (6) Could not resolve host …`, e.g. the `chonk_inputs.sh` S3 download) are consistent with AWS's **link-local PPS limit**: traffic to the Amazon resolver (the VPC `.2` address / `169.254.169.253`) is capped at **~1024 packets/sec per ENI**, and over that, packets are silently dropped (`linklocal_allowance_exceeded` in `ethtool -S`). We confirmed the build's DNS path makes this likely: the devbox container **and** nested docker-in-docker both get `nameserver 172.31.0.2` and query the VPC resolver directly — no caching. The host's `systemd-resolved` *is* caching (~48% hit on host-only traffic) but listens on loopback only (`127.0.0.53`), so containers can't use it. With the build's parallelism (and the larger spot instances), the aggregate DNS rate blows past 1024 pps. ## Fix Route container DNS through the host's caching `systemd-resolved`: - Expose its stub on the instance's **primary private IP** (derived from `ip route get`, no IMDS dependency) via `DNSStubListenerExtra` — that's the one address reachable from the devbox container *and* nested dind (unlike the docker0 gateway). - Point containers at it with `docker run --dns <priv_ip>`. Non-loopback nameservers propagate through the nested dockerd, so dind inherits it. Repeat lookups become cache hits and never reach the throttled resolver. ## Safety This can only help, never break resolution: if the IP can't be derived, `systemd-resolved` isn't active, or the stub doesn't come up on the IP (5×0.5s health check via `ss`), `priv_ip` is cleared and `--dns` is omitted — leaving DNS exactly as today. No counter instrumentation included — we're treating link-local throttling as the known cause. (PR #379's `linklocal_allowance_exceeded` logging can confirm before/after if desired.) ## Validation - `bash -n` on `ci3/bootstrap_ec2`; rendered+`bash -n` the injected host-script block; verified the `ip route get` parse and the `${priv_ip:+--dns …}` expansion locally. - Full validation is the PR's own CI run: a build instance that resolves through the cache and (ideally) flat `linklocal_allowance_exceeded`.
Collaborator
|
✅ Successfully backported to backport-to-v5-next-staging #24277. |
rangozd
pushed a commit
to rangozd/aztec-packages
that referenced
this pull request
Aug 5, 2026
…ling (AztecProtocol#383) Port of AztecProtocol#24105. ## Problem CI DNS failures (`curl: (6) Could not resolve host …`) are consistent with AWS's link-local PPS limit: traffic to the Amazon resolver (VPC `.2` / `169.254.169.253`) is capped at ~1024 pps per ENI; over that, packets are silently dropped (`linklocal_allowance_exceeded`). The devbox container and nested dind both query the VPC resolver directly with no caching, and the build's parallelism blows past the cap. ## Fix Route container DNS through the host's caching `systemd-resolved`: expose its stub on the instance's primary private IP (reachable from the devbox container and nested dind) via `DNSStubListenerExtra`, and point containers at it with `docker run --dns`. Repeat lookups become cache hits and never reach the throttled resolver. Fails safe: if the IP can't be derived, `systemd-resolved` isn't active, or the stub doesn't bind (health-checked via `ss`), `priv_ip` is cleared and `--dns` omitted — DNS unchanged. ## Validation `bash -n` on `ci3/bootstrap_ec2`; rendered+checked the injected host-script block; verified the `ip route get` parse and `--dns` expansion. Full validation is the CI run itself.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
CI DNS failures (
curl: (6) Could not resolve host …, e.g. thechonk_inputs.shS3 download) are consistent with AWS's link-local PPS limit: traffic to the Amazon resolver (the VPC.2address /169.254.169.253) is capped at ~1024 packets/sec per ENI, and over that, packets are silently dropped (linklocal_allowance_exceededinethtool -S).We confirmed the build's DNS path makes this likely: the devbox container and nested docker-in-docker both get
nameserver 172.31.0.2and query the VPC resolver directly — no caching. The host'ssystemd-resolvedis caching (~48% hit on host-only traffic) but listens on loopback only (127.0.0.53), so containers can't use it. With the build's parallelism (and the larger spot instances), the aggregate DNS rate blows past 1024 pps.Fix
Route container DNS through the host's caching
systemd-resolved:ip route get, no IMDS dependency) viaDNSStubListenerExtra— that's the one address reachable from the devbox container and nested dind (unlike the docker0 gateway).docker run --dns <priv_ip>. Non-loopback nameservers propagate through the nested dockerd, so dind inherits it.Repeat lookups become cache hits and never reach the throttled resolver.
Safety
This can only help, never break resolution: if the IP can't be derived,
systemd-resolvedisn't active, or the stub doesn't come up on the IP (5×0.5s health check viass),priv_ipis cleared and--dnsis omitted — leaving DNS exactly as today.No counter instrumentation included — we're treating link-local throttling as the known cause. (PR #379's
linklocal_allowance_exceededlogging can confirm before/after if desired.)Validation
bash -nonci3/bootstrap_ec2; rendered+bash -nthe injected host-script block; verified theip route getparse and the${priv_ip:+--dns …}expansion locally.linklocal_allowance_exceeded.