feat: route fbuild through a versioned running-process daemon broker

[zackees]

Context

fbuild has the pieces for running-process broker adoption, but the runtime path is still effectively a single direct loopback daemon:

• crates/fbuild-cli/src/daemon_client.rs still constructs DaemonClient against fbuild_paths::get_daemon_url() and ensure_daemon_running() still spawns a bare fbuild-daemon process directly.
• crates/fbuild-python/src/daemon.rs mirrors that direct spawn / direct HTTP health-check behavior for PyO3 callers.
• crates/fbuild-daemon/src/broker/ now contains the v1 protocol/service/session modules, the FBUILD_PAYLOAD_PROTOCOL = 0x7EB1 pin, parity-tested JSON/prost payload model, service-definition installer, and FbuildBrokerSession::adopt with RUNNING_PROCESS_DISABLE=1 fallback.
• docs/running-process/inventory.md records the intended target: JSON direct path retained as fallback, prost broker path for broker-routed requests, SHARED_BROKER locally, EXPLICIT_INSTANCE ci-trusted in CI, and version negotiation through running-process.
• fbuild daemon running-process still reports a direct-fallback mode from fbuild-paths::running_process, and RunningProcessDaemonMode::uses_direct_fallback() is hard-coded true.

That means the current system has broker metadata and service registration, but clients are not yet delegated through a daemon broker that selects a versioned fbuild-daemon backend. The next iteration should make the broker path operational without attempting the broader serial-port custody redesign in #592.

Proposal

Implement the next migration slice: route fbuild CLI/Python daemon acquisition through running-process, negotiate a versioned fbuild-daemon backend, and keep the current direct loopback HTTP daemon as the rollback path.

Concretely:

1. Add a shared daemon acquisition path used by CLI and Python bindings.

   • Prefer running-process adoption when RUNNING_PROCESS_DISABLE is not set.
   • Negotiate service fbuild and wanted version env!(CARGO_PKG_VERSION) / packaged fbuild version.
   • Cache/use the broker-returned backend endpoint for subsequent requests where the running-process API supports it.

2. Make the broker-selected backend the source of truth for versioned daemon routing.

   • A client installed from version N should connect through the broker to the negotiated compatible/highest registered fbuild-daemon backend rather than spawning an arbitrary fbuild-daemon from PATH.
   • Preserve venv-adjacent daemon resolution as the binary registered in the service definition, not as a parallel direct-spawn path.
   • Keep dev/prod isolation: FBUILD_DEV_MODE=1 still isolates port/root from prod.

3. Bridge broker requests to the existing HTTP handlers without forking schemas.

   • Reuse BrokerRequest / BrokerResponse and DaemonOp from crates/fbuild-daemon/src/broker/protocol.rs.
   • Keep the existing JSON HTTP endpoints as the compatibility/fallback surface.
   • Ensure build_streaming/NDJSON behavior is either supported over broker or explicitly falls back with a tracked limitation and test coverage.

4. Update diagnostics and docs.

   • fbuild daemon running-process --json should distinguish broker-negotiated, direct-fallback, and disabled states.
   • Runtime diagnostics should show the negotiated daemon version and endpoint when the broker path is active.
   • Update stale crates/fbuild-daemon/README.md text that still says broker adoption was declined.

Acceptance criteria

• With RUNNING_PROCESS_DISABLE unset and a valid fbuild.servicedef, a normal CLI command reaches a running-process-negotiated fbuild-daemon backend instead of directly spawning/talking to the fixed loopback daemon first.
• RUNNING_PROCESS_DISABLE=1 reliably preserves the legacy direct loopback HTTP path and existing spawn/backoff behavior.
• CLI and PyO3 daemon startup/status paths share the same routing policy so Python callers do not bypass the broker when CLI callers use it.
• Multiple installed fbuild versions register usable service definitions, and the client reports the negotiated daemon version in diagnostics.
• The existing HTTP JSON endpoints and schemas remain compatible with FastLED/Python callers.
• Tests cover broker-negotiated success, broker unavailable/refused fallback or error behavior, disable-env direct fallback, service-definition version metadata, and at least one end-to-end operation (health or daemon info) through the negotiated path.
• Docs under docs/running-process/ and crates/fbuild-daemon/README.md describe the active state accurately.

Open questions

• Should streaming build output be broker-framed in this slice, or should build_streaming intentionally use direct HTTP until a follow-up issue handles streaming frames?
• Does running-process currently expose the exact version-selection policy fbuild wants (highest compatible registered backend), or does fbuild need a small compatibility layer around the current API?
• Should CI use EXPLICIT_INSTANCE ci-trusted immediately in this slice, or should local broker routing land first with CI isolation as a follow-up?

Related issues

• running-process broker: fbuild adoption tracker #510 tracked the original fbuild running-process adoption and is now closed with the runtime broker path still stubbed/direct-fallback.
• Don't add a separate fbuild-broker crate — keep fbuild close to a monocrate #560 closed the crate-layout decision: keep broker adoption inside fbuild-daemon/src/broker/, not a new workspace crate.
• design: restore broker-mediated serial proxy + finish running-process broker (port custody, disconnect events, multi-client singleton, cross-version device lease) #592 is strongly related but broader: serial-port custody, proxy lifecycle, disconnect events, multi-client singleton behavior, and cross-version device leases. This issue is the narrower process/versioned-daemon routing prerequisite.