fix: don't time out bb socket startup while the bb process is alive - #24802
Merged
PhilWindle merged 7 commits intoJul 24, 2026
Merged
Conversation
Merges the current `v5-next` tree (`c11bc68`) back into `v5` and bumps the release-please manifest to `5.0.1` to cut the v5.0.1 patch release. - `.release-please-manifest.json`: `5.0.0` → `5.0.1` - Brings all commits on `v5-next` since `v5.0.0` onto `v5` (companion change: `v5-next` manifest bumped to `5.1.0` in `ee3716277a`). **Merge strategy:** this is a release-branch sync — merge so `v5` remains a true superset of the history (merge commit or fast-forward). A squash merge would collapse the whole tree into a single commit and rewrite the SHAs, so avoid it here. Tagging `v5.0.1` on the resulting release commit is a follow-up step.
The socket backend gave bb a hard 5s budget shared between socket file creation and connect, so a loaded prover spawning many bb processes at once (epoch top-tree) could fail startup with 'Timeout connecting to bb socket: unknown' before a single connect attempt was made, and the resulting plain Error was reported retry=false, costing the epoch. The backend now waits as long as the bb process is alive, failing fast with the real cause if it dies, with a single generous 60s backstop for a wedged process. Startup failures are wrapped as retryable ProvingErrors and the bb_prover wrap sites preserve the retry flag.
Unpinned, typescript now resolves to 7.x, which no longer ships lib/_tsc.js and crashes Yarn 4's builtin compat/typescript patch during install, failing docs/examples/bootstrap.sh before any example runs. Matches the existing pin in docs/examples/ts/bootstrap.sh.
Promotes `v5-next` onto `v5` for the **v5.1.0** release. - Merges `v5-next` into `v5`; the resulting tree is **identical to `v5-next`** and `.release-please-manifest.json` is set to **5.1.0** (the only merge conflict was the manifest — v5 held 5.0.1, resolved to v5-next's 5.1.0). - Merge via **merge commit** (v5 ruleset allows `merge` only). - After this merges, `v5` HEAD is tagged `v5.1.0`. Does not include #24840 (json-rpc result key), which is still on `merge-train/spartan-v5` and not in `v5-next`.
PhilWindle
approved these changes
Jul 24, 2026
Collaborator
|
❌ Failed to cherry-pick to |
This was referenced Jul 24, 2026
rangozd
pushed a commit
to rangozd/aztec-packages
that referenced
this pull request
Aug 5, 2026
…ztecProtocol#24802) Fixes the `Error: Timeout connecting to bb socket: unknown (retry=false)` failures reported by an operator since v5.0.0, which killed proving jobs during epoch top-tree and cost epochs. ## Root cause The NativeUnixSocket backend gave bb a hard 5s wall-clock budget that was **shared** between two phases: waiting for bb to create its socket file, and connecting to it. `connectWithRetry` reused the `startTime` captured before the file-wait poll loop, so when bb took close to 5s to create the socket (many bb processes spawning simultaneously during top-tree checkpoint/merge jobs, or the Node event loop starving the 50ms polls), the connect phase was entered with its budget already exhausted and threw without making a single connect attempt — that is what the `unknown` in the error message means (`lastErr` was never set). Two aggravating factors: - 5s is an arbitrary opinion about how fast a loaded machine should spawn a process. The pre-socket v4 CLI model had no such deadline — you waited on the child, and the only failures were real process events. - The resulting plain `Error` reached the proving agent, which only honours the retry flag on `ProvingError`s, so the failure was reported `retry=false` and the job failed permanently instead of being re-enqueued. ## Changes ### `barretenberg/ts` — socket backend restructure (`native_socket.ts`) - Replaced the constructor + deferred `connectionPromise` wiring with a `static async new()` factory, matching the shm and wasm backends. An instance can now only exist once connected, so `call()` no longer awaits a stashed connection promise (and is no longer `async` — its body has no awaits). - Startup is one flat wait-and-connect loop with the correct liveness condition: **retry for as long as the bb process is alive**. If bb dies, fail immediately with the real cause (exit code / signal); spawn failures are caught up front by awaiting the `spawn` event. Both 5s timers are deleted. - One generous 60s backstop remains for a bb that is alive but wedged before `listen()`. It kills the process (routing cleanup through the exit path) rather than leaving an orphan. It is a broken-process detector, not a performance expectation: the timed window ends at bb's `listen()`, which is reached after only exec + linking + minimal init (the expensive startup work comes after the socket is up), so firing it requires a machine degraded far beyond ordinary proving load. And if it ever does fire on a merely-distressed machine, the failure is retryable (see below), so the cost is a re-enqueue, not an epoch. - The four copy-pasted reject-pending-callbacks blocks (process error/exit, socket error/end) are consolidated into `failAllPending()`. Note one deliberate behavioural improvement: the socket `error`/`end` handlers now also destroy and null the socket, so subsequent `call()`s fail fast with `Socket not connected` instead of `write after destroy`. - New `native_socket.test.ts` covers: prompt startup, bb taking >5s to create its socket (the incident's failure mode — fails on the old code by construction, passes now), bb dying before the socket exists, and a nonexistent binary. ### `yarn-project/bb-prover` — make startup failures retryable - `BBJsInstance.create` wraps any `Barretenberg.new` failure as `ProvingError(..., retry: true)`: bb startup failures are environmental (machine load, wedged process), never a property of the proof inputs, so the job is always safe to retry. - The two catch sites in `bb_prover.ts` that re-wrap errors (`generateProof`, `verifyProof`) previously constructed a fresh `ProvingError` with the default `retry=false`, silently dropping the flag. They now propagate the inner error's retryability and attach it as `cause`. Errors that were non-retryable before remain non-retryable. The AVM paths don't wrap, so they needed no change. - New `bb_js_backend.test.ts` asserts a startup failure surfaces as a retryable `ProvingError`. ## Result for operators A loaded prover can take as long as it needs to spawn bb; a genuinely broken bb fails after 60s with a concrete, diagnosable cause instead of `Timeout ... unknown`; and if that ever happens the broker re-enqueues the job (up to its retry limit) instead of failing it permanently and costing the epoch. ## Testing - `barretenberg/ts`: new jest suite passes (including a 7s-delayed fake bb that the old implementation fails on). - `yarn-project`: full `yarn build`, bb-prover lint, and the new unit test pass. Root `./bootstrap.sh` green. ## Unrelated CI fix included `docs/examples/ts/aztecjs_runner/run.sh` installed `typescript` and `tsx` unpinned; the TypeScript 7.0.2 release (which drops `lib/_tsc.js`) crashes Yarn 4's builtin `compat/typescript` patch at install time, failing `docs/examples/bootstrap.sh execute` on every branch. Pinned to `typescript@^5.3.3` / `tsx@^4`, matching the existing pin in `docs/examples/ts/bootstrap.sh`. This was the only failure in this PR's first full CI run and needs forward-porting to the `next` line as well. # Conflicts: # barretenberg/ts/bb.js/src/bb_backends/node/index.ts # barretenberg/ts/bb.js/src/bb_backends/node/native_socket.test.ts # barretenberg/ts/bb.js/src/bb_backends/node/native_socket.ts
rangozd
pushed a commit
to rangozd/aztec-packages
that referenced
this pull request
Aug 5, 2026
## Summary Ports AztecProtocol#24802 to `next`. This keeps the same behavior change from the merged v5 PR: - bb.js native socket startup now waits while the bb process remains alive, with a 60s wedged-process backstop instead of the old shared 5s socket/connect deadline. - bb startup failures are wrapped as retryable `ProvingError`s in bb-prover. - retryability is preserved when bb-prover re-wraps proof generation and verification failures. ## Conflict resolution The automatic cherry-pick conflicted because `barretenberg/ts/src/...` on the source branch has moved to `barretenberg/ts/bb.js/src/...` on `next`. Resolved by adapting the socket backend changes into the `bb.js` path and preserving `next`'s `createAsyncBackend` API shape, which returns `IMsgpackBackendAsync` rather than constructing a `Barretenberg` wrapper there. Added a small follow-up commit to satisfy the `next` lint rule for `destroy()`. Refs AztecProtocol#24802 ## Verification - `git diff --check origin/next...HEAD` - `yarn formatting:fix` in `barretenberg/ts/bb.js` - `yarn test bb_backends/node/native_socket.test.ts --runInBand` in `barretenberg/ts/bb.js` Blocked locally: - `./bootstrap.sh build` in `barretenberg/ts/bb.js` passed formatting, then stopped at codegen because this checkout lacks `barretenberg/cpp/build/bin/bb`. - `JEST_MAX_WORKERS=1 yarn workspace @aztec/bb-prover test src/bb/bb_js_backend.test.ts` could not complete in this cold checkout without built workspace exports for `@aztec/bb.js`/`@aztec/foundation`. --- *Created by [claudebox](https://claudebox.work/v2/sessions/f03a66a010266d34/jobs/1) · group: `slackbot` · [Slack thread](https://aztecprotocol.slack.com/archives/C0AGN2WT3CP/p1784890226762699?thread_ts=1784890226.762699&cid=C0AGN2WT3CP)*
charlielye
added a commit
that referenced
this pull request
Aug 10, 2026
…db/bb-avm-sim services Hardens the generated IPC packages' spawn-and-connect path against the failure family behind #24802, and makes bb-avm-sim process failures invisible to everything above the AVM pool. Servers listen before heavy init: aztec-wsdb creates its socket before WorldState construction and bb-avm-sim before its upstream wsdb/CDB connects, so clients connect into the accept backlog immediately and the connect backstop only covers exec + linking + reaching listen(). bb-avm-sim installs SIGUSR1 and lifecycle handlers (incl. parent-death monitoring) before the socket is reachable; upstream connect budget 5s -> 60s. ipc-runtime gains SpawnedProcessBackend, extracted from the codegen template: liveness-based connect raced against child death, kill-on-expiry backstop, log capture (async fs throughout — sync fs here would stall the event loop on exactly the degraded machines this path runs on), SIGTERM -> SIGKILL teardown, and opt-in lazy respawn (next call after a death gets a fresh process; stable ipc path; no eager crash-loop). Errors are typed (IpcTransportError, IpcProcessExitedError, IpcSpawnError) with a retry flag distinguishing process death from configuration errors; call failures that race the child's exit event are attributed to the death after a short grace. The generated package shrinks to binary resolution + backend config and passes through a respawn option; the call surface gains nothing. The AVM pool spawns services with respawn enabled and is the only interpreter of the retry flag — callers keep exact pre-IPC semantics (result, tx failure, or their own deadline). Environmental spawn failures retry indefinitely on a flat 1s cadence (no backoff: sequencers live on ~6s slots), bounded by the caller's abort signal, which threads through checkout, the spawn-retry loop, and full-pool waits. A simulation whose process dies is re-issued once; a second death is attributed to the input so a simulator-crashing tx is evicted instead of retried forever. Cancellation escalates SIGUSR1 -> (5s) -> SIGKILL, reclaiming the pool slot from a wedged simulation. Configuration errors surface fatally at the boot-time prewarm. wsdb keeps fail-fast semantics: a respawned wsdb would lose all forks and uncommitted state. New tests: SpawnedProcessBackend unit tests (script-based fake servers), echo ts_package reliability tests (slow-listen, die-before-listen, wedged-then-killed, missing-binary classification, respawn), AVM pool tests (indefinite abort-aware spawn retry, config fast-fail, re-issue-once, input attribution), and wsdb/avm server compile + startup-order checks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the
Error: Timeout connecting to bb socket: unknown (retry=false)failures reported by an operator since v5.0.0, which killed proving jobs during epoch top-tree and cost epochs.Root cause
The NativeUnixSocket backend gave bb a hard 5s wall-clock budget that was shared between two phases: waiting for bb to create its socket file, and connecting to it.
connectWithRetryreused thestartTimecaptured before the file-wait poll loop, so when bb took close to 5s to create the socket (many bb processes spawning simultaneously during top-tree checkpoint/merge jobs, or the Node event loop starving the 50ms polls), the connect phase was entered with its budget already exhausted and threw without making a single connect attempt — that is what theunknownin the error message means (lastErrwas never set).Two aggravating factors:
Errorreached the proving agent, which only honours the retry flag onProvingErrors, so the failure was reportedretry=falseand the job failed permanently instead of being re-enqueued.Changes
barretenberg/ts— socket backend restructure (native_socket.ts)connectionPromisewiring with astatic async new()factory, matching the shm and wasm backends. An instance can now only exist once connected, socall()no longer awaits a stashed connection promise (and is no longerasync— its body has no awaits).spawnevent. Both 5s timers are deleted.listen(). It kills the process (routing cleanup through the exit path) rather than leaving an orphan. It is a broken-process detector, not a performance expectation: the timed window ends at bb'slisten(), which is reached after only exec + linking + minimal init (the expensive startup work comes after the socket is up), so firing it requires a machine degraded far beyond ordinary proving load. And if it ever does fire on a merely-distressed machine, the failure is retryable (see below), so the cost is a re-enqueue, not an epoch.failAllPending(). Note one deliberate behavioural improvement: the socketerror/endhandlers now also destroy and null the socket, so subsequentcall()s fail fast withSocket not connectedinstead ofwrite after destroy.native_socket.test.tscovers: prompt startup, bb taking >5s to create its socket (the incident's failure mode — fails on the old code by construction, passes now), bb dying before the socket exists, and a nonexistent binary.yarn-project/bb-prover— make startup failures retryableBBJsInstance.createwraps anyBarretenberg.newfailure asProvingError(..., retry: true): bb startup failures are environmental (machine load, wedged process), never a property of the proof inputs, so the job is always safe to retry.bb_prover.tsthat re-wrap errors (generateProof,verifyProof) previously constructed a freshProvingErrorwith the defaultretry=false, silently dropping the flag. They now propagate the inner error's retryability and attach it ascause. Errors that were non-retryable before remain non-retryable. The AVM paths don't wrap, so they needed no change.bb_js_backend.test.tsasserts a startup failure surfaces as a retryableProvingError.Result for operators
A loaded prover can take as long as it needs to spawn bb; a genuinely broken bb fails after 60s with a concrete, diagnosable cause instead of
Timeout ... unknown; and if that ever happens the broker re-enqueues the job (up to its retry limit) instead of failing it permanently and costing the epoch.Testing
barretenberg/ts: new jest suite passes (including a 7s-delayed fake bb that the old implementation fails on).yarn-project: fullyarn build, bb-prover lint, and the new unit test pass. Root./bootstrap.shgreen.Unrelated CI fix included
docs/examples/ts/aztecjs_runner/run.shinstalledtypescriptandtsxunpinned; the TypeScript 7.0.2 release (which dropslib/_tsc.js) crashes Yarn 4's builtincompat/typescriptpatch at install time, failingdocs/examples/bootstrap.sh executeon every branch. Pinned totypescript@^5.3.3/tsx@^4, matching the existing pin indocs/examples/ts/bootstrap.sh. This was the only failure in this PR's first full CI run and needs forward-porting to thenextline as well.