fix(desktop): supervise and re-arm relay-mesh runtime - #2823
Conversation
Relay-mesh agents went silent with no error if the embedded mesh runtime exited or wedged after launch: `mesh_llm_runtime` stayed `Some` while the local OpenAI ingress on :9337 was dead, so `ensure_relay_mesh_for_record` took the "already running" fast path and `wait_for_mesh_inference` just timed out against a dead endpoint (#2062). Add a fast liveness probe (`mesh_ingress_is_live`: single 3s `GET /v1/models`, the same call the issue used to confirm the ingress was down). When a runtime handle is present but the ingress is unreachable, drop the stale runtime (best-effort stop; never block re-arm on a wedged runtime — the zombie-guard motivation) and fall through to re-arm it via the normal bootstrap path. Resolves the reported symptom (option 1 + zombie guard from the issue): long-lived sessions recover shared-compute agents on the next dispatch instead of staying silent until a manual restart. Additive; the healthy fast path (live ingress) is unchanged. Refs #2062 Signed-off-by: Bartok9 <danielrpike9@gmail.com>
Brad #2304: mesh_ingress_is_live inside ensure_relay_mesh_for_record only ran on start/restore — not after launch. Local agents hit :9337 themselves, so there is no desktop turn-dispatch hook; add a bounded coordinator watchdog (15s base, double on failure to 120s) that probes ingress, drops a zombie runtime handle, and re-arms via ensure_relay_mesh_for_record for relay-mesh agents. Failed re-arm writes an actionable \"Buzz shared compute offline\" last_error. Share drop_stale_mesh_runtime_if_ingress_dead with ensure path. Unit tests cover dead-port probe, noop drop without handle, and failure copy. Refs #2062 Signed-off-by: Bartok9 <danielrpike9@gmail.com>
…opped agents Addresses Brad's #2304 review of head 6d20d20: 1. Bounded stale stop() — a wedged runtime could hang stop().await and block the watchdog forever, defeating the never-block-re-arm intent. Wrap in a 3s tokio::time::timeout (matches probe budget); log and drop the handle on timeout/error so the watchdog keeps making progress. 2. Probe/evict race — capture the runtime identity (new monotonic DesktopMeshRuntime::id) before the ingress probe .await and only evict if the same handle is still installed on lock reacquire, so a concurrent stop/start replacement is never evicted. 3. Stopped/manual records — re-arm now filters to local relay-mesh agents whose own process is actually running (runtime_pid + process_is_running), so deliberately stopped agents are not resurrected. Adds unit tests for the running/stopped/non-mesh re-arm target filter. Signed-off-by: Bartok9 <danielrpike9@gmail.com>
Address remaining review gaps on the watchdog re-arm path: 1. Keep bounded stop timeout (already present) and document the never-block-re-arm invariant. 2. Extract identity-compare helper + injectable ingress probe so probe/evict never drops a concurrent replacement runtime. 3. Intersect relay-mesh records with live managed_agent_processes pubkeys (not every configured mesh record / pid-only heuristic). 4. persist/clear mesh last_error under managed_agents_store_lock, bump updated_at, preserve unrelated errors, surface save failures. Tests: process-map filter, identity skip, stop budget, error classifier; hardware-gated kill-:9337 recovery documented as Signed-off-by: Bartok9 <danielrpike9@gmail.com> #[ignore] for manual mesh machines.
…micspiral #2304) Address the mesh-applicability review on the re-arm watchdog: 1. Consecutive-failure debounce before eviction (merge-blocker). A single dead /v1/models probe past the 3s budget (transient stall: model load/reload, VRAM alloc, GC/mmap pause, inference saturation) no longer cold-restarts a healthy runtime. Require DEAD_PROBE_EVICT_THRESHOLD (2) consecutive dead probes on the watchdog cadence; a live probe or an identity-mismatch replacement resets the streak. Counter lives on AppState (AtomicU32). 2. clear_mesh_last_error_if_set now matches a sentinel prefix (MESH_REARM_ERROR_SENTINEL) instead of the loose "shared compute" substring, so recovery only clears errors this watchdog set. 3. Documented the serve->client re-arm mode transition as intentional-by-design (safe fail-safe; serve restoration stays restore_mesh_sharing's job). Tests: added eviction_debounces_transient_dead_probe; updated error classifier + actionable-copy tests for the sentinel. 17 passed, 2 ignored. Signed-off-by: Bartok9 <danielrpike9@gmail.com>
CI on #2304 failed: cargo fmt --check on mesh_llm paths, and desktop file-size gate (app_state + mesh_llm growth for watchdog). Signed-off-by: Bartok9 <danielrpike9@gmail.com>
The AppState field is #[cfg(feature = "mesh-llm")] but its initializer was unconditional, breaking non-mesh builds under -D warnings. Gate the initializer and fully-qualify AtomicU32 so the import is not left unused (micspiral, #2304). Signed-off-by: Bartok9 <danielrpike9@gmail.com>
Signed-off-by: Michael Neale <michael.neale@gmail.com>
Signed-off-by: Michael Neale <michael.neale@gmail.com>
Contributor note from #2304 (Bartok9)@micspiral — thank you for taking this over after validating against a real two-machine setup. Moving here as a fellow contributor (happy for you to keep driving the code). Credit: Clean and appreciated. PR body credits #2304; 7 commits remain authored/signed as Bartok9 ( What I dual-checked against the #2304 intent + your.additions
Checks: Desktop/E2E/DCO/lint green from here; MERGEABLE, waiting on review protections. Optional polish (your call — zero pressure; offering eyes only):
I will close #2304 as superseded by this PR (full cherry-pick + your ops layer). Happy to review further revisions, reproduce dogfood nits, or ship a small follow-up PR if you want a fix coded passively — or stay in review-only mode if you prefer to implement everything yourself. |
…unity-policy-fetch * origin/main: Fix formatting in README.md diagram (block#2284) fix(desktop): make Linux AppImage GStreamer work on non-Debian distros (block#2176) refactor(desktop): remove Agent directory section from Agents page (block#2290) fix(desktop): enable arboard Wayland backend so Linux copies reach the Wayland clipboard (block#2904) fix(desktop): supervise and re-arm relay-mesh runtime (block#2823) fix(agents): run live Databricks discovery instead of the fallback list (block#2890) fix(desktop): retire prepend mode on every reader wheel (block#2913) fix(desktop): consolidate prepend scroll correction (block#2855) docs(buzz-acp): correct agent key generation instructions (block#2875) fix(desktop): track concurrent agent turns up to the harness maximum (block#2882) docs(contributing): trim to goose-scale minimal intake surface (block#2780) fix(relay): preserve reconnect backoff (block#2759)
…ding-harnesses * origin/main: (47 commits) fix(desktop): remove bundled libsystemd from AppImage (#2353) docs: document required DCO sign-off and add commit-msg sign-off hook (#2993) fix(desktop): make agent definition authoritative for model/provider/prompt (#1968) chore(desktop): delete dead persona catalog UI cluster (#2886) fix(desktop): surface install failures hidden by curl-pipe exit codes (#2892) fix(mobile): validate invite relay destinations (#2986) Refactor managed-agent runtime into cohesive modules (#2974) Refine mobile settings and themes (#2844) Fix formatting in README.md diagram (#2284) fix(desktop): make Linux AppImage GStreamer work on non-Debian distros (#2176) refactor(desktop): remove Agent directory section from Agents page (#2290) fix(desktop): enable arboard Wayland backend so Linux copies reach the Wayland clipboard (#2904) fix(desktop): supervise and re-arm relay-mesh runtime (#2823) fix(agents): run live Databricks discovery instead of the fallback list (#2890) fix(desktop): retire prepend mode on every reader wheel (#2913) fix(desktop): consolidate prepend scroll correction (#2855) docs(buzz-acp): correct agent key generation instructions (#2875) fix(desktop): track concurrent agent turns up to the harness maximum (#2882) docs(contributing): trim to goose-scale minimal intake surface (#2780) fix(relay): preserve reconnect backoff (#2759) ... Signed-off-by: morgmart <98432065+morgmart@users.noreply.github.com> # Conflicts: # desktop/src/features/onboarding/ui/SetupStep.tsx
Uses MeshLLM built-in `mesh` collective intelligence / Mixture of Agents when Buzz Auto sees two or more distinct physical models. With zero or one model, Auto remains ordinary `auto`. This is an alternative way to improve tool responses, accuracy, and resistance to hallucination when a high-latency distributed mesh contains diverse models. Models and members may come and go: collective routing enables only after stable capacity, drops on confirmed contraction, and can recover later. Mesh-specific failures retry once through ordinary Auto. This update also pins MeshLLM to a v0.73.1-compatible backport of [MeshLLM #1074](Mesh-LLM/mesh-llm#1074), so client-only Buzz nodes cannot enter model election or download a remote provider model. Buzz preserves the selected local sharing model and switches an existing client to sharing across a controlled app restart, retaining one runtime and one `:9337` / `:3131` pair per machine. Validation: - Full local `just ci` passes on the cleaned branch. - MeshLLM host-runtime suite: 1,568 passed, 0 failed; strict Clippy passes. - Buzz desktop Tauri suite with `mesh-llm`: 1,721 passed, 0 failed; strict feature Clippy passes. - Playwright covers client-to-share using the saved local model and no destructive stop. - Packaged two-machine testing proved single-model routing, dual-model collective routing, tool-markup fallback, and runtime reuse. - Packaged client-only recheck routed a real Mini Buzz turn through M5 while Mini stayed `is_client=true`, `is_host=false`, hosted no models, and created no Gemma cache. Builds on the recovery work merged in #2823; this PR does not duplicate it. --------- Signed-off-by: Michael Neale <michael.neale@gmail.com> Signed-off-by: Tyler Longwell <tlongwell@block.xyz> Co-authored-by: Michael Neale <michael.neale@gmail.com> Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
## Buzz Desktop release v0.5.0 ### Changes since v0.4.26: - feat(invites): add use-limited invite links ([#3141](#3141)) ([`d500c2d5c`](d500c2d)) - fix(node): bump Buzz-supplied Node runtimes past OpenClaw's >=24.15.0 floor ([#3218](#3218)) ([`98a7b1334`](98a7b13)) - fix(desktop): preserve thread anchor through layout reflow ([#3212](#3212)) ([`9810d8545`](9810d85)) - feat(search): parse from:/in:/after:/before: and pass them in the filter ([#2871](#2871)) ([`cb2a265b5`](cb2a265)) - fix(desktop): fetch join policies through native networking ([#2862](#2862)) ([`0019f8076`](0019f80)) - fix(desktop): republish agent identity records when a persona rename propagates ([#2607](#2607)) ([`7ca0bbd94`](7ca0bbd)) - fix(desktop): keep project Inbox previews compact ([#3193](#3193)) ([`de1396050`](de13960)) - Inbox refactor ([#2045](#2045)) ([`2bd4c24b7`](2bd4c24)) - Fix composer selection formatting and drop overlay ([#3172](#3172)) ([`99da5b7eb`](99da5b7)) - Refine pending message status ([#3153](#3153)) ([`75588eaff`](75588ea)) - fix(desktop): recover full local storage on startup ([#3182](#3182)) ([`174c38e4b`](174c38e)) - fix(desktop): keep collapsed table separators out of spoilers ([#3169](#3169)) ([`4d8b676bb`](4d8b676)) - feat(desktop): redesign agent runtime settings ([#3093](#3093)) ([`d98da7389`](d98da73)) - fix(desktop): use forward slashes for git credential.helper on Windows ([#3023](#3023)) ([`899531684`](8995316)) - chore(desktop): add AgentCreationPreview file-size override to unblock main CI ([#3154](#3154)) ([`b92a1f4bf`](b92a1f4)) - fix(desktop): make the test loader work on Windows ([#2758](#2758)) ([`8bb43d519`](8bb43d5)) - fix(desktop): make lint and unit-test gates work on Windows ([#2943](#2943)) ([`545bb46b8`](545bb46)) - feat(desktop): add search to agent emoji picker ([#2630](#2630)) ([`313f793c8`](313f793)) - fix(desktop): keep identity key help dialog readable in dark mode ([#2854](#2854)) ([`be275cfc6`](be275cf)) - feat(acp): title agent sessions from the agent and channel name ([#3028](#3028)) ([`f2fe3b63c`](f2fe3b6)) - feat(git): use agent display name as git author name ([#3040](#3040)) ([`18eef633d`](18eef63)) - fix(deps): bump nostr to 0.44.6 for RUSTSEC-2026-0216 (NIP-44 remote DoS) ([#3135](#3135)) ([`31e2de196`](31e2de1)) - fix(desktop): read the newest pair-scoped harness log ([#3134](#3134)) ([`654f38490`](654f384)) - feat(desktop): handle project work from Inbox ([#3117](#3117)) ([`c5c4f390b`](c5c4f39)) - fix(desktop): clarify identity key button when key exists ([#2357](#2357)) ([`87b3fcd3c`](87b3fcd)) - Restore Goose and Buzz Agent to onboarding harness selection ([#2731](#2731)) ([`7fc0cc82d`](7fc0cc8)) - fix(desktop): render rich project work item content ([#3100](#3100)) ([`afb272bb7`](afb272b)) - feat(acp): bring your own harness (BYOH) — generic ACP runtime seam + settings gallery ([#2773](#2773)) ([`95fdf9788`](95fdf97)) - feat(desktop): use collective mesh routing for Auto ([#2825](#2825)) ([`16d4ec335`](16d4ec3)) - fix(desktop): strip legacy baked team instructions from stored prompts ([#3035](#3035)) ([`aee631448`](aee6314)) - feat(agents): lower default agent parallelism from 24 to 10 ([#3038](#3038)) ([`5d8ede446`](5d8ede4)) - Polish community rail and mobile pairing ([#2972](#2972)) ([`e6c90bb7c`](e6c90bb)) - fix(desktop): remove bundled libsystemd from AppImage ([#2353](#2353)) ([`a31fc4d2f`](a31fc4d)) - fix(desktop): make agent definition authoritative for model/provider/prompt ([#1968](#1968)) ([`8c0e8cb16`](8c0e8cb)) - chore(desktop): delete dead persona catalog UI cluster ([#2886](#2886)) ([`8e67cf399`](8e67cf3)) - fix(desktop): surface install failures hidden by curl-pipe exit codes ([#2892](#2892)) ([`166c6655e`](166c665)) - Refactor managed-agent runtime into cohesive modules ([#2974](#2974)) ([`74b63e184`](74b63e1)) - fix(desktop): make Linux AppImage GStreamer work on non-Debian distros ([#2176](#2176)) ([`cc6c4d347`](cc6c4d3)) - refactor(desktop): remove Agent directory section from Agents page ([#2290](#2290)) ([`5d1233e84`](5d1233e)) - fix(desktop): enable arboard Wayland backend so Linux copies reach the Wayland clipboard ([#2904](#2904)) ([`ab7aa8b12`](ab7aa8b)) - fix(desktop): supervise and re-arm relay-mesh runtime ([#2823](#2823)) ([`aa51dab9d`](aa51dab)) - fix(agents): run live Databricks discovery instead of the fallback list ([#2890](#2890)) ([`8eb6e3eb6`](8eb6e3e)) - fix(desktop): retire prepend mode on every reader wheel ([#2913](#2913)) ([`07d0265cf`](07d0265)) - fix(desktop): consolidate prepend scroll correction ([#2855](#2855)) ([`25e7864b3`](25e7864)) - fix(desktop): track concurrent agent turns up to the harness maximum ([#2882](#2882)) ([`20bff5910`](20bff59)) - fix(relay): preserve reconnect backoff ([#2759](#2759)) ([`499c5d349`](499c5d3)) - refactor(relay): expose reconnect timing policy ([#2310](#2310)) ([`2f0041595`](2f00415)) - fix(desktop): clear stale working badges on agent stop/restart ([#2803](#2803)) ([`a64cc71f6`](a64cc71)) - fix(desktop): surface agent rename relay profile sync failure as a warning toast ([#2279](#2279)) ([`5e3d2e484`](5e3d2e4)) - fix(discovery): inject PATH into Codex adapter planning ([#2767](#2767)) ([`6ab3835f3`](6ab3835)) **To release:** merge this PR. The tag and build will happen automatically. Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Refs #2062
This carries forward the relay-mesh recovery work from #2304 by @Bartok9 (cherry-picked with original authorship/sign-offs) and adds the startup/readiness supervision and packaging fixes found while validating it against a real two-machine Buzz setup.
From #2304
:9337) after launch and re-arm a stale relay-mesh runtime.Added here
:9337), rather than management port:3131, as Buzz's client-readiness boundary. A usable client no longer fails or holds agent-save open merely because management startup is still pending.buzz-acp,buzz-agent, and tool sidecars to be reported as missing.Validation
Automated:
just ci— passed, including formatting, Clippy with warnings denied, desktop/web/mobile checks and tests, and builds.-D warnings— passed.mesh-llm— built successfully; every bundled sidecar passed executable-mode and deep code-signature verification.Live two-machine E2E:
unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M.buzz-agent/relay-mesh agent.:9337accepted inference and the saved ACP harness started with no error while:3131was still unavailable.CORRECTED-MINI-E2E-OK).:3131was still unavailable.AFTER-RESTART-E2E-OK).@C55message traversed desktop →buzz-acp→buzz-agent→ mini:9337→ M5 compute and published the requested reply successfully.