Skip to content

daemon: concurrent fbuild invocations stampede daemon spawn — add single-flight + bind before heavy init #1010

Description

@zackees

Context

When N fbuild invocations start concurrently with no daemon running (a common CI / parallel-build pattern), they all spawn a full fbuild-daemon. The direct spawn path has no spawn lock, no single-flight, and no pre-bind probe: ensure_direct_daemon_running (crates/fbuild-cli/src/daemon_client.rs:818-861, entered via ensure_daemon_running() from cli/build.rs:59, cli/deploy.rs:164, device, mcp, …) does health-check → spawn → poll 10 s, retried 3×. The broker path (try_acquire_broker_daemon, daemon_client.rs:699-772) would centralize spawn but silently falls back to the direct path whenever no broker is running — the common case.

Arbitration happens only at TCP bind (SO_EXCLUSIVEADDRUSE on Windows; losers retry 3×500 ms then exit(1), crates/fbuild-daemon/src/main.rs:557-638). The expensive problem: each loser fully initializes before it loses the bind race — containment-group install (main.rs:48), USB overlay network fetch (main.rs:86), and the embedded zccache service start (main.rs:93, CompileBackend::start()) all run before bind_listener_with_retry at main.rs:188. So N-1 doomed daemons each pay full startup and concurrently hammer ~/.fbuild/<mode>/zccache/. And a bind-race loser's exit(1) after ~1.5 s can be misread by the retrying CLI as a spawn failure rather than "someone else won."

Proposal

  • Single-flight the spawn: take an exclusive spawn-slot lock file in get_daemon_dir() (non-blocking create_new); the winner spawns, losers defer and poll health — the pattern soldr uses (.spawn.lock in soldr-cli/src/daemon/lifecycle.rs) and zccache uses (perf(build): framework libs (~150s) recompiled per project; core cache never hydrates #952 spawn slot).
  • Bind before heavy init: reorder fbuild-daemon startup so bind_listener_with_retry runs first; a bind-race loser then exits cheaply without starting containment, USB overlay, or the embedded zccache service (main.rs:48/86/93 moved after main.rs:188).
  • Loser defers, not fails: a bind-race loser should signal "winner exists" so the retrying CLI polls the winner's health instead of treating exit(1) as a spawn failure.

Acceptance criteria

  • N concurrent fbuild invocations produce exactly one fbuild-daemon (spawn-slot single-flight); losers adopt the winner.
  • A bind-race loser does not start the embedded zccache service / containment / USB overlay before exiting.
  • A concurrency test (≥16 threads) asserts one daemon and no CLI misreads a loser exit as failure.

Decisions

  • Severity: MEDIUM-HIGH — not a correctness bug, but under parallel CI it wastes N× daemon startups (each starting an embedded zccache) and can flap the daemon; the fix is well-precedented (soldr .spawn.lock, zccache perf(build): framework libs (~150s) recompiled per project; core cache never hydrates #952).
  • Separate from the endpoint-identity issue (filed alongside) — that one decides which daemon owns the endpoint; this one ensures only one is spawned.

Related

Cross-repo sibling of zccache #1007 (uses the #952 spawn-slot single-flight) and soldr #1493. Filed after a cross-repo daemon-deployment audit; soldr's .spawn.lock and zccache's spawn slot are the reference implementations.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    Triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions