fix: port bb socket startup retry to next - #24960
Merged
Merged
Conversation
…24802) Fixes the `Error: Timeout connecting to bb socket: unknown (retry=false)` failures reported by an operator since v5.0.0, which killed proving jobs during epoch top-tree and cost epochs. ## Root cause The NativeUnixSocket backend gave bb a hard 5s wall-clock budget that was **shared** between two phases: waiting for bb to create its socket file, and connecting to it. `connectWithRetry` reused the `startTime` captured before the file-wait poll loop, so when bb took close to 5s to create the socket (many bb processes spawning simultaneously during top-tree checkpoint/merge jobs, or the Node event loop starving the 50ms polls), the connect phase was entered with its budget already exhausted and threw without making a single connect attempt — that is what the `unknown` in the error message means (`lastErr` was never set). Two aggravating factors: - 5s is an arbitrary opinion about how fast a loaded machine should spawn a process. The pre-socket v4 CLI model had no such deadline — you waited on the child, and the only failures were real process events. - The resulting plain `Error` reached the proving agent, which only honours the retry flag on `ProvingError`s, so the failure was reported `retry=false` and the job failed permanently instead of being re-enqueued. ## Changes ### `barretenberg/ts` — socket backend restructure (`native_socket.ts`) - Replaced the constructor + deferred `connectionPromise` wiring with a `static async new()` factory, matching the shm and wasm backends. An instance can now only exist once connected, so `call()` no longer awaits a stashed connection promise (and is no longer `async` — its body has no awaits). - Startup is one flat wait-and-connect loop with the correct liveness condition: **retry for as long as the bb process is alive**. If bb dies, fail immediately with the real cause (exit code / signal); spawn failures are caught up front by awaiting the `spawn` event. Both 5s timers are deleted. - One generous 60s backstop remains for a bb that is alive but wedged before `listen()`. It kills the process (routing cleanup through the exit path) rather than leaving an orphan. It is a broken-process detector, not a performance expectation: the timed window ends at bb's `listen()`, which is reached after only exec + linking + minimal init (the expensive startup work comes after the socket is up), so firing it requires a machine degraded far beyond ordinary proving load. And if it ever does fire on a merely-distressed machine, the failure is retryable (see below), so the cost is a re-enqueue, not an epoch. - The four copy-pasted reject-pending-callbacks blocks (process error/exit, socket error/end) are consolidated into `failAllPending()`. Note one deliberate behavioural improvement: the socket `error`/`end` handlers now also destroy and null the socket, so subsequent `call()`s fail fast with `Socket not connected` instead of `write after destroy`. - New `native_socket.test.ts` covers: prompt startup, bb taking >5s to create its socket (the incident's failure mode — fails on the old code by construction, passes now), bb dying before the socket exists, and a nonexistent binary. ### `yarn-project/bb-prover` — make startup failures retryable - `BBJsInstance.create` wraps any `Barretenberg.new` failure as `ProvingError(..., retry: true)`: bb startup failures are environmental (machine load, wedged process), never a property of the proof inputs, so the job is always safe to retry. - The two catch sites in `bb_prover.ts` that re-wrap errors (`generateProof`, `verifyProof`) previously constructed a fresh `ProvingError` with the default `retry=false`, silently dropping the flag. They now propagate the inner error's retryability and attach it as `cause`. Errors that were non-retryable before remain non-retryable. The AVM paths don't wrap, so they needed no change. - New `bb_js_backend.test.ts` asserts a startup failure surfaces as a retryable `ProvingError`. ## Result for operators A loaded prover can take as long as it needs to spawn bb; a genuinely broken bb fails after 60s with a concrete, diagnosable cause instead of `Timeout ... unknown`; and if that ever happens the broker re-enqueues the job (up to its retry limit) instead of failing it permanently and costing the epoch. ## Testing - `barretenberg/ts`: new jest suite passes (including a 7s-delayed fake bb that the old implementation fails on). - `yarn-project`: full `yarn build`, bb-prover lint, and the new unit test pass. Root `./bootstrap.sh` green. ## Unrelated CI fix included `docs/examples/ts/aztecjs_runner/run.sh` installed `typescript` and `tsx` unpinned; the TypeScript 7.0.2 release (which drops `lib/_tsc.js`) crashes Yarn 4's builtin `compat/typescript` patch at install time, failing `docs/examples/bootstrap.sh execute` on every branch. Pinned to `typescript@^5.3.3` / `tsx@^4`, matching the existing pin in `docs/examples/ts/bootstrap.sh`. This was the only failure in this PR's first full CI run and needs forward-porting to the `next` line as well. # Conflicts: # barretenberg/ts/bb.js/src/bb_backends/node/index.ts # barretenberg/ts/bb.js/src/bb_backends/node/native_socket.test.ts # barretenberg/ts/bb.js/src/bb_backends/node/native_socket.ts
AztecBot
added
ci-draft
PhilWindle
approved these changes
Jul 24, 2026
PhilWindle
marked this pull request as ready for review
July 24, 2026 11:10
PhilWindle
enabled auto-merge
July 24, 2026 11:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports #24802 to
next.This keeps the same behavior change from the merged v5 PR:
ProvingErrors in bb-prover.Conflict resolution
The automatic cherry-pick conflicted because
barretenberg/ts/src/...on the source branch has moved tobarretenberg/ts/bb.js/src/...onnext.Resolved by adapting the socket backend changes into the
bb.jspath and preservingnext'screateAsyncBackendAPI shape, which returnsIMsgpackBackendAsyncrather than constructing aBarretenbergwrapper there. Added a small follow-up commit to satisfy thenextlint rule fordestroy().Refs #24802
Verification
git diff --check origin/next...HEADyarn formatting:fixinbarretenberg/ts/bb.jsyarn test bb_backends/node/native_socket.test.ts --runInBandinbarretenberg/ts/bb.jsBlocked locally:
./bootstrap.sh buildinbarretenberg/ts/bb.jspassed formatting, then stopped at codegen because this checkout lacksbarretenberg/cpp/build/bin/bb.JEST_MAX_WORKERS=1 yarn workspace @aztec/bb-prover test src/bb/bb_js_backend.test.tscould not complete in this cold checkout without built workspace exports for@aztec/bb.js/@aztec/foundation.Created by claudebox · group:
slackbot· Slack thread