Skip to content

fix: don't time out bb socket startup while the bb process is alive - #24802

Merged
PhilWindle merged 7 commits into
merge-train/spartan-v5from
charlie/fix-bb-socket-startup-timeout
Jul 24, 2026
Merged

fix: don't time out bb socket startup while the bb process is alive#24802
PhilWindle merged 7 commits into
merge-train/spartan-v5from
charlie/fix-bb-socket-startup-timeout

Conversation

@charlielye

@charlielye charlielye commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Fixes the Error: Timeout connecting to bb socket: unknown (retry=false) failures reported by an operator since v5.0.0, which killed proving jobs during epoch top-tree and cost epochs.

Root cause

The NativeUnixSocket backend gave bb a hard 5s wall-clock budget that was shared between two phases: waiting for bb to create its socket file, and connecting to it. connectWithRetry reused the startTime captured before the file-wait poll loop, so when bb took close to 5s to create the socket (many bb processes spawning simultaneously during top-tree checkpoint/merge jobs, or the Node event loop starving the 50ms polls), the connect phase was entered with its budget already exhausted and threw without making a single connect attempt — that is what the unknown in the error message means (lastErr was never set).

Two aggravating factors:

  • 5s is an arbitrary opinion about how fast a loaded machine should spawn a process. The pre-socket v4 CLI model had no such deadline — you waited on the child, and the only failures were real process events.
  • The resulting plain Error reached the proving agent, which only honours the retry flag on ProvingErrors, so the failure was reported retry=false and the job failed permanently instead of being re-enqueued.

Changes

barretenberg/ts — socket backend restructure (native_socket.ts)

  • Replaced the constructor + deferred connectionPromise wiring with a static async new() factory, matching the shm and wasm backends. An instance can now only exist once connected, so call() no longer awaits a stashed connection promise (and is no longer async — its body has no awaits).
  • Startup is one flat wait-and-connect loop with the correct liveness condition: retry for as long as the bb process is alive. If bb dies, fail immediately with the real cause (exit code / signal); spawn failures are caught up front by awaiting the spawn event. Both 5s timers are deleted.
  • One generous 60s backstop remains for a bb that is alive but wedged before listen(). It kills the process (routing cleanup through the exit path) rather than leaving an orphan. It is a broken-process detector, not a performance expectation: the timed window ends at bb's listen(), which is reached after only exec + linking + minimal init (the expensive startup work comes after the socket is up), so firing it requires a machine degraded far beyond ordinary proving load. And if it ever does fire on a merely-distressed machine, the failure is retryable (see below), so the cost is a re-enqueue, not an epoch.
  • The four copy-pasted reject-pending-callbacks blocks (process error/exit, socket error/end) are consolidated into failAllPending(). Note one deliberate behavioural improvement: the socket error/end handlers now also destroy and null the socket, so subsequent call()s fail fast with Socket not connected instead of write after destroy.
  • New native_socket.test.ts covers: prompt startup, bb taking >5s to create its socket (the incident's failure mode — fails on the old code by construction, passes now), bb dying before the socket exists, and a nonexistent binary.

yarn-project/bb-prover — make startup failures retryable

  • BBJsInstance.create wraps any Barretenberg.new failure as ProvingError(..., retry: true): bb startup failures are environmental (machine load, wedged process), never a property of the proof inputs, so the job is always safe to retry.
  • The two catch sites in bb_prover.ts that re-wrap errors (generateProof, verifyProof) previously constructed a fresh ProvingError with the default retry=false, silently dropping the flag. They now propagate the inner error's retryability and attach it as cause. Errors that were non-retryable before remain non-retryable. The AVM paths don't wrap, so they needed no change.
  • New bb_js_backend.test.ts asserts a startup failure surfaces as a retryable ProvingError.

Result for operators

A loaded prover can take as long as it needs to spawn bb; a genuinely broken bb fails after 60s with a concrete, diagnosable cause instead of Timeout ... unknown; and if that ever happens the broker re-enqueues the job (up to its retry limit) instead of failing it permanently and costing the epoch.

Testing

  • barretenberg/ts: new jest suite passes (including a 7s-delayed fake bb that the old implementation fails on).
  • yarn-project: full yarn build, bb-prover lint, and the new unit test pass. Root ./bootstrap.sh green.

Unrelated CI fix included

docs/examples/ts/aztecjs_runner/run.sh installed typescript and tsx unpinned; the TypeScript 7.0.2 release (which drops lib/_tsc.js) crashes Yarn 4's builtin compat/typescript patch at install time, failing docs/examples/bootstrap.sh execute on every branch. Pinned to typescript@^5.3.3 / tsx@^4, matching the existing pin in docs/examples/ts/bootstrap.sh. This was the only failure in this PR's first full CI run and needs forward-porting to the next line as well.

aminsammara and others added 3 commits July 15, 2026 18:01
Merges the current `v5-next` tree (`c11bc68`) back into `v5` and bumps
the release-please manifest to `5.0.1` to cut the v5.0.1 patch release.

- `.release-please-manifest.json`: `5.0.0` → `5.0.1`
- Brings all commits on `v5-next` since `v5.0.0` onto `v5` (companion
change: `v5-next` manifest bumped to `5.1.0` in `ee3716277a`).

**Merge strategy:** this is a release-branch sync — merge so `v5`
remains a true superset of the history (merge commit or fast-forward). A
squash merge would collapse the whole tree into a single commit and
rewrite the SHAs, so avoid it here.

Tagging `v5.0.1` on the resulting release commit is a follow-up step.
The socket backend gave bb a hard 5s budget shared between socket file
creation and connect, so a loaded prover spawning many bb processes at
once (epoch top-tree) could fail startup with 'Timeout connecting to bb
socket: unknown' before a single connect attempt was made, and the
resulting plain Error was reported retry=false, costing the epoch.

The backend now waits as long as the bb process is alive, failing fast
with the real cause if it dies, with a single generous 60s backstop for
a wedged process. Startup failures are wrapped as retryable
ProvingErrors and the bb_prover wrap sites preserve the retry flag.
@charlielye
charlielye requested a review from a team July 20, 2026 09:31
Unpinned, typescript now resolves to 7.x, which no longer ships
lib/_tsc.js and crashes Yarn 4's builtin compat/typescript patch during
install, failing docs/examples/bootstrap.sh before any example runs.
Matches the existing pin in docs/examples/ts/bootstrap.sh.
@charlielye
charlielye requested a review from a team as a code owner July 20, 2026 10:19
@charlielye charlielye added ci-release-pr Creates a development tag and runs the release suite and removed ci-full-no-test-cache labels Jul 20, 2026
@AztecBot AztecBot removed the ci-release-pr Creates a development tag and runs the release suite label Jul 20, 2026
aminsammara and others added 3 commits July 21, 2026 23:04
Promotes `v5-next` onto `v5` for the **v5.1.0** release.

- Merges `v5-next` into `v5`; the resulting tree is **identical to
`v5-next`** and `.release-please-manifest.json` is set to **5.1.0** (the
only merge conflict was the manifest — v5 held 5.0.1, resolved to
v5-next's 5.1.0).
- Merge via **merge commit** (v5 ruleset allows `merge` only).
- After this merges, `v5` HEAD is tagged `v5.1.0`.

Does not include #24840 (json-rpc result key), which is still on
`merge-train/spartan-v5` and not in `v5-next`.
@PhilWindle
PhilWindle changed the base branch from v5 to merge-train/spartan-v5 July 24, 2026 10:48
@PhilWindle PhilWindle added the port-to-next Forward-port this merged PR into next label Jul 24, 2026
@PhilWindle
PhilWindle enabled auto-merge July 24, 2026 10:48
@PhilWindle
PhilWindle merged commit 2413b09 into merge-train/spartan-v5 Jul 24, 2026
28 checks passed
@PhilWindle
PhilWindle deleted the charlie/fix-bb-socket-startup-timeout branch July 24, 2026 10:49
@AztecBot

Copy link
Copy Markdown
Collaborator

❌ Failed to cherry-pick to next due to conflicts. (🤖) View backport run.

rangozd pushed a commit to rangozd/aztec-packages that referenced this pull request Aug 5, 2026
…ztecProtocol#24802)

Fixes the `Error: Timeout connecting to bb socket: unknown
(retry=false)` failures reported by an operator since v5.0.0, which
killed proving jobs during epoch top-tree and cost epochs.

## Root cause

The NativeUnixSocket backend gave bb a hard 5s wall-clock budget that
was **shared** between two phases: waiting for bb to create its socket
file, and connecting to it. `connectWithRetry` reused the `startTime`
captured before the file-wait poll loop, so when bb took close to 5s to
create the socket (many bb processes spawning simultaneously during
top-tree checkpoint/merge jobs, or the Node event loop starving the 50ms
polls), the connect phase was entered with its budget already exhausted
and threw without making a single connect attempt — that is what the
`unknown` in the error message means (`lastErr` was never set).

Two aggravating factors:

- 5s is an arbitrary opinion about how fast a loaded machine should
spawn a process. The pre-socket v4 CLI model had no such deadline — you
waited on the child, and the only failures were real process events.
- The resulting plain `Error` reached the proving agent, which only
honours the retry flag on `ProvingError`s, so the failure was reported
`retry=false` and the job failed permanently instead of being
re-enqueued.

## Changes

### `barretenberg/ts` — socket backend restructure (`native_socket.ts`)

- Replaced the constructor + deferred `connectionPromise` wiring with a
`static async new()` factory, matching the shm and wasm backends. An
instance can now only exist once connected, so `call()` no longer awaits
a stashed connection promise (and is no longer `async` — its body has no
awaits).
- Startup is one flat wait-and-connect loop with the correct liveness
condition: **retry for as long as the bb process is alive**. If bb dies,
fail immediately with the real cause (exit code / signal); spawn
failures are caught up front by awaiting the `spawn` event. Both 5s
timers are deleted.
- One generous 60s backstop remains for a bb that is alive but wedged
before `listen()`. It kills the process (routing cleanup through the
exit path) rather than leaving an orphan. It is a broken-process
detector, not a performance expectation: the timed window ends at bb's
`listen()`, which is reached after only exec + linking + minimal init
(the expensive startup work comes after the socket is up), so firing it
requires a machine degraded far beyond ordinary proving load. And if it
ever does fire on a merely-distressed machine, the failure is retryable
(see below), so the cost is a re-enqueue, not an epoch.
- The four copy-pasted reject-pending-callbacks blocks (process
error/exit, socket error/end) are consolidated into `failAllPending()`.
Note one deliberate behavioural improvement: the socket `error`/`end`
handlers now also destroy and null the socket, so subsequent `call()`s
fail fast with `Socket not connected` instead of `write after destroy`.
- New `native_socket.test.ts` covers: prompt startup, bb taking >5s to
create its socket (the incident's failure mode — fails on the old code
by construction, passes now), bb dying before the socket exists, and a
nonexistent binary.

### `yarn-project/bb-prover` — make startup failures retryable

- `BBJsInstance.create` wraps any `Barretenberg.new` failure as
`ProvingError(..., retry: true)`: bb startup failures are environmental
(machine load, wedged process), never a property of the proof inputs, so
the job is always safe to retry.
- The two catch sites in `bb_prover.ts` that re-wrap errors
(`generateProof`, `verifyProof`) previously constructed a fresh
`ProvingError` with the default `retry=false`, silently dropping the
flag. They now propagate the inner error's retryability and attach it as
`cause`. Errors that were non-retryable before remain non-retryable. The
AVM paths don't wrap, so they needed no change.
- New `bb_js_backend.test.ts` asserts a startup failure surfaces as a
retryable `ProvingError`.

## Result for operators

A loaded prover can take as long as it needs to spawn bb; a genuinely
broken bb fails after 60s with a concrete, diagnosable cause instead of
`Timeout ... unknown`; and if that ever happens the broker re-enqueues
the job (up to its retry limit) instead of failing it permanently and
costing the epoch.

## Testing

- `barretenberg/ts`: new jest suite passes (including a 7s-delayed fake
bb that the old implementation fails on).
- `yarn-project`: full `yarn build`, bb-prover lint, and the new unit
test pass. Root `./bootstrap.sh` green.

## Unrelated CI fix included

`docs/examples/ts/aztecjs_runner/run.sh` installed `typescript` and
`tsx` unpinned; the TypeScript 7.0.2 release (which drops `lib/_tsc.js`)
crashes Yarn 4's builtin `compat/typescript` patch at install time,
failing `docs/examples/bootstrap.sh execute` on every branch. Pinned to
`typescript@^5.3.3` / `tsx@^4`, matching the existing pin in
`docs/examples/ts/bootstrap.sh`. This was the only failure in this PR's
first full CI run and needs forward-porting to the `next` line as well.
# Conflicts:
#	barretenberg/ts/bb.js/src/bb_backends/node/index.ts
#	barretenberg/ts/bb.js/src/bb_backends/node/native_socket.test.ts
#	barretenberg/ts/bb.js/src/bb_backends/node/native_socket.ts
rangozd pushed a commit to rangozd/aztec-packages that referenced this pull request Aug 5, 2026
## Summary

Ports AztecProtocol#24802 to
`next`.

This keeps the same behavior change from the merged v5 PR:
- bb.js native socket startup now waits while the bb process remains
alive, with a 60s wedged-process backstop instead of the old shared 5s
socket/connect deadline.
- bb startup failures are wrapped as retryable `ProvingError`s in
bb-prover.
- retryability is preserved when bb-prover re-wraps proof generation and
verification failures.

## Conflict resolution

The automatic cherry-pick conflicted because `barretenberg/ts/src/...`
on the source branch has moved to `barretenberg/ts/bb.js/src/...` on
`next`.

Resolved by adapting the socket backend changes into the `bb.js` path
and preserving `next`'s `createAsyncBackend` API shape, which returns
`IMsgpackBackendAsync` rather than constructing a `Barretenberg` wrapper
there. Added a small follow-up commit to satisfy the `next` lint rule
for `destroy()`.

Refs AztecProtocol#24802

## Verification

- `git diff --check origin/next...HEAD`
- `yarn formatting:fix` in `barretenberg/ts/bb.js`
- `yarn test bb_backends/node/native_socket.test.ts --runInBand` in
`barretenberg/ts/bb.js`

Blocked locally:
- `./bootstrap.sh build` in `barretenberg/ts/bb.js` passed formatting,
then stopped at codegen because this checkout lacks
`barretenberg/cpp/build/bin/bb`.
- `JEST_MAX_WORKERS=1 yarn workspace @aztec/bb-prover test
src/bb/bb_js_backend.test.ts` could not complete in this cold checkout
without built workspace exports for `@aztec/bb.js`/`@aztec/foundation`.

---
*Created by
[claudebox](https://claudebox.work/v2/sessions/f03a66a010266d34/jobs/1)
· group: `slackbot` · [Slack
thread](https://aztecprotocol.slack.com/archives/C0AGN2WT3CP/p1784890226762699?thread_ts=1784890226.762699&cid=C0AGN2WT3CP)*
charlielye added a commit that referenced this pull request Aug 10, 2026
…db/bb-avm-sim services

Hardens the generated IPC packages' spawn-and-connect path against the
failure family behind #24802, and makes bb-avm-sim process failures
invisible to everything above the AVM pool.

Servers listen before heavy init: aztec-wsdb creates its socket before
WorldState construction and bb-avm-sim before its upstream wsdb/CDB
connects, so clients connect into the accept backlog immediately and the
connect backstop only covers exec + linking + reaching listen(). bb-avm-sim
installs SIGUSR1 and lifecycle handlers (incl. parent-death monitoring)
before the socket is reachable; upstream connect budget 5s -> 60s.

ipc-runtime gains SpawnedProcessBackend, extracted from the codegen
template: liveness-based connect raced against child death, kill-on-expiry
backstop, log capture (async fs throughout — sync fs here would stall the
event loop on exactly the degraded machines this path runs on), SIGTERM ->
SIGKILL teardown, and opt-in lazy respawn (next call after a death gets a
fresh process; stable ipc path; no eager crash-loop). Errors are typed
(IpcTransportError, IpcProcessExitedError, IpcSpawnError) with a retry flag
distinguishing process death from configuration errors; call failures that
race the child's exit event are attributed to the death after a short
grace. The generated package shrinks to binary resolution + backend config
and passes through a respawn option; the call surface gains nothing.

The AVM pool spawns services with respawn enabled and is the only
interpreter of the retry flag — callers keep exact pre-IPC semantics
(result, tx failure, or their own deadline). Environmental spawn failures
retry indefinitely on a flat 1s cadence (no backoff: sequencers live on
~6s slots), bounded by the caller's abort signal, which threads through
checkout, the spawn-retry loop, and full-pool waits. A simulation whose
process dies is re-issued once; a second death is attributed to the input
so a simulator-crashing tx is evicted instead of retried forever.
Cancellation escalates SIGUSR1 -> (5s) -> SIGKILL, reclaiming the pool
slot from a wedged simulation. Configuration errors surface fatally at the
boot-time prewarm. wsdb keeps fail-fast semantics: a respawned wsdb would
lose all forks and uncommitted state.

New tests: SpawnedProcessBackend unit tests (script-based fake servers),
echo ts_package reliability tests (slow-listen, die-before-listen,
wedged-then-killed, missing-binary classification, respawn), AVM pool
tests (indefinite abort-aware spawn retry, config fast-fail, re-issue-once,
input attribution), and wsdb/avm server compile + startup-order checks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

port-to-next Forward-port this merged PR into next

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants