fix(ci): retry forge solc fetch on transient DNS failures - #24358
Merged
Conversation
PhilWindle
marked this pull request as ready for review
June 29, 2026 08:37
alexghr
approved these changes
Jun 29, 2026
PhilWindle
enabled auto-merge
June 29, 2026 08:39
AztecBot
pushed a commit
that referenced
this pull request
Jun 29, 2026
## Problem PR #24352 (and others) intermittently fail in the merge queue with a DNS error. The failing step is the `barretenberg/sol` test command: ``` Command: ...cd barretenberg/sol && forge test --no-match-contract Base Error: error sending request for url (https://binaries.soliditylang.org/linux-amd64/list.json) Context: - Error #0: client error (Connect) - Error #1: dns error - Error #2: failed to lookup address information: Temporary failure in name resolution ``` `forge` reaches out to `binaries.soliditylang.org` to resolve/fetch the solc binary. Under heavy parallel merge-queue load the VPC resolver intermittently drops lookups (`Temporary failure in name resolution`), and the operation fails with no retry, taking the whole CI run down with it. ## Fix Retry the solc-fetching `forge` operations **every 10s for ~5 min** (30 attempts), scoped to connection/DNS failures so genuine errors still fail fast. - **`ci3/retry`**: make the attempt count tunable via `RETRY_ATTEMPTS` (default unchanged at 3); `RETRY_SLEEP` was already honored (default 5s). Fully backward-compatible — existing callers are unaffected. - **`barretenberg/sol/bootstrap.sh`**: wrap the `forge test` test command in `RETRY_ATTEMPTS=30 RETRY_SLEEP=10 retry -p '<dns/connect regex>'`. This is the operation that fails in the merge queue. - **`l1-contracts/bootstrap.sh`**: the root svm download (the "single owner of the svm download") already had a short `retry`; extend it to the same 10s/5-min window and scope it to DNS/connection errors. (`2>/dev/null` removed so `retry`'s pattern matcher can see the DNS error on stderr.) The `-p` pattern (`dns error|Temporary failure in name resolution|error sending request|failed to lookup address|Connection refused|connection reset`) means a real `forge test` assertion failure or compile error is **not** retried — it exits immediately, so this never masks real failures or wastes 5 minutes on a genuinely broken test. ## Verification Exercised `ci3/retry` and the exact emitted `barretenberg/sol` command locally: - DNS-matching failure → retries up to `RETRY_ATTEMPTS`, sleeping `RETRY_SLEEP` between attempts. - Real test failure (no DNS text) → fails fast, no retry (0s). - Success → returns 0 on the first attempt (0s). - `RETRY_ATTEMPTS` unset → still 3 attempts (backward-compatible). --- *Created by [claudebox](https://claudebox.work/v2/sessions/5610bcb5aef4d0b9) · group: `slackbot`*
Collaborator
Author
|
✅ Successfully backported to backport-to-v5-next-staging #24360. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
PR #24352 (and others) intermittently fail in the merge queue with a DNS error. The failing step is the
barretenberg/soltest command:forgereaches out tobinaries.soliditylang.orgto resolve/fetch the solc binary. Under heavy parallel merge-queue load the VPC resolver intermittently drops lookups (Temporary failure in name resolution), and the operation fails with no retry, taking the whole CI run down with it.Fix
Retry the solc-fetching
forgeoperations every 10s for ~5 min (30 attempts), scoped to connection/DNS failures so genuine errors still fail fast.ci3/retry: make the attempt count tunable viaRETRY_ATTEMPTS(default unchanged at 3);RETRY_SLEEPwas already honored (default 5s). Fully backward-compatible — existing callers are unaffected.barretenberg/sol/bootstrap.sh: wrap theforge testtest command inRETRY_ATTEMPTS=30 RETRY_SLEEP=10 retry -p '<dns/connect regex>'. This is the operation that fails in the merge queue.l1-contracts/bootstrap.sh: the root svm download (the "single owner of the svm download") already had a shortretry; extend it to the same 10s/5-min window and scope it to DNS/connection errors. (2>/dev/nullremoved soretry's pattern matcher can see the DNS error on stderr.)The
-ppattern (dns error|Temporary failure in name resolution|error sending request|failed to lookup address|Connection refused|connection reset) means a realforge testassertion failure or compile error is not retried — it exits immediately, so this never masks real failures or wastes 5 minutes on a genuinely broken test.Verification
Exercised
ci3/retryand the exact emittedbarretenberg/solcommand locally:RETRY_ATTEMPTS, sleepingRETRY_SLEEPbetween attempts.RETRY_ATTEMPTSunset → still 3 attempts (backward-compatible).Created by claudebox · group:
slackbot