Skip to content

fix(ci): retry forge solc fetch on transient DNS failures - #24358

Merged
PhilWindle merged 1 commit into
nextfrom
cb/retry-solc-dns
Jun 29, 2026
Merged

fix(ci): retry forge solc fetch on transient DNS failures#24358
PhilWindle merged 1 commit into
nextfrom
cb/retry-solc-dns

Conversation

@AztecBot

Copy link
Copy Markdown
Collaborator

Problem

PR #24352 (and others) intermittently fail in the merge queue with a DNS error. The failing step is the barretenberg/sol test command:

Command: ...cd barretenberg/sol && forge test --no-match-contract Base
Error: error sending request for url (https://binaries.soliditylang.org/linux-amd64/list.json)
Context:
- Error #0: client error (Connect)
- Error #1: dns error
- Error #2: failed to lookup address information: Temporary failure in name resolution

forge reaches out to binaries.soliditylang.org to resolve/fetch the solc binary. Under heavy parallel merge-queue load the VPC resolver intermittently drops lookups (Temporary failure in name resolution), and the operation fails with no retry, taking the whole CI run down with it.

Fix

Retry the solc-fetching forge operations every 10s for ~5 min (30 attempts), scoped to connection/DNS failures so genuine errors still fail fast.

  • ci3/retry: make the attempt count tunable via RETRY_ATTEMPTS (default unchanged at 3); RETRY_SLEEP was already honored (default 5s). Fully backward-compatible — existing callers are unaffected.
  • barretenberg/sol/bootstrap.sh: wrap the forge test test command in RETRY_ATTEMPTS=30 RETRY_SLEEP=10 retry -p '<dns/connect regex>'. This is the operation that fails in the merge queue.
  • l1-contracts/bootstrap.sh: the root svm download (the "single owner of the svm download") already had a short retry; extend it to the same 10s/5-min window and scope it to DNS/connection errors. (2>/dev/null removed so retry's pattern matcher can see the DNS error on stderr.)

The -p pattern (dns error|Temporary failure in name resolution|error sending request|failed to lookup address|Connection refused|connection reset) means a real forge test assertion failure or compile error is not retried — it exits immediately, so this never masks real failures or wastes 5 minutes on a genuinely broken test.

Verification

Exercised ci3/retry and the exact emitted barretenberg/sol command locally:

  • DNS-matching failure → retries up to RETRY_ATTEMPTS, sleeping RETRY_SLEEP between attempts.
  • Real test failure (no DNS text) → fails fast, no retry (0s).
  • Success → returns 0 on the first attempt (0s).
  • RETRY_ATTEMPTS unset → still 3 attempts (backward-compatible).

Created by claudebox · group: slackbot

@AztecBot AztecBot added ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR. labels Jun 29, 2026
@PhilWindle
PhilWindle marked this pull request as ready for review June 29, 2026 08:37
@PhilWindle
PhilWindle requested a review from charlielye as a code owner June 29, 2026 08:37
@PhilWindle
PhilWindle enabled auto-merge June 29, 2026 08:39
@PhilWindle
PhilWindle added this pull request to the merge queue Jun 29, 2026
Merged via the queue into next with commit 872e2cf Jun 29, 2026
56 of 68 checks passed
@PhilWindle
PhilWindle deleted the cb/retry-solc-dns branch June 29, 2026 09:45
AztecBot pushed a commit that referenced this pull request Jun 29, 2026
## Problem

PR #24352 (and others) intermittently fail in the merge queue with a DNS
error. The failing step is the `barretenberg/sol` test command:

```
Command: ...cd barretenberg/sol && forge test --no-match-contract Base
Error: error sending request for url (https://binaries.soliditylang.org/linux-amd64/list.json)
Context:
- Error #0: client error (Connect)
- Error #1: dns error
- Error #2: failed to lookup address information: Temporary failure in name resolution
```

`forge` reaches out to `binaries.soliditylang.org` to resolve/fetch the
solc binary. Under heavy parallel merge-queue load the VPC resolver
intermittently drops lookups (`Temporary failure in name resolution`),
and the operation fails with no retry, taking the whole CI run down with
it.

## Fix

Retry the solc-fetching `forge` operations **every 10s for ~5 min** (30
attempts), scoped to connection/DNS failures so genuine errors still
fail fast.

- **`ci3/retry`**: make the attempt count tunable via `RETRY_ATTEMPTS`
(default unchanged at 3); `RETRY_SLEEP` was already honored (default
5s). Fully backward-compatible — existing callers are unaffected.
- **`barretenberg/sol/bootstrap.sh`**: wrap the `forge test` test
command in `RETRY_ATTEMPTS=30 RETRY_SLEEP=10 retry -p '<dns/connect
regex>'`. This is the operation that fails in the merge queue.
- **`l1-contracts/bootstrap.sh`**: the root svm download (the "single
owner of the svm download") already had a short `retry`; extend it to
the same 10s/5-min window and scope it to DNS/connection errors.
(`2>/dev/null` removed so `retry`'s pattern matcher can see the DNS
error on stderr.)

The `-p` pattern (`dns error|Temporary failure in name resolution|error
sending request|failed to lookup address|Connection refused|connection
reset`) means a real `forge test` assertion failure or compile error is
**not** retried — it exits immediately, so this never masks real
failures or wastes 5 minutes on a genuinely broken test.

## Verification

Exercised `ci3/retry` and the exact emitted `barretenberg/sol` command
locally:

- DNS-matching failure → retries up to `RETRY_ATTEMPTS`, sleeping
`RETRY_SLEEP` between attempts.
- Real test failure (no DNS text) → fails fast, no retry (0s).
- Success → returns 0 on the first attempt (0s).
- `RETRY_ATTEMPTS` unset → still 3 attempts (backward-compatible).

---
*Created by
[claudebox](https://claudebox.work/v2/sessions/5610bcb5aef4d0b9) ·
group: `slackbot`*
@AztecBot

Copy link
Copy Markdown
Collaborator Author

✅ Successfully backported to backport-to-v5-next-staging #24360.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-to-v5-next ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants