fix(packages,daemon): reclaim dead-owner install locks; stop leaving stale endpoint records (#1213)

[zackees]

Addresses #1213 parts 1, 2 and 3.

Part 1 — the deadlock (the one that cost 40 minutes)

The headline finding: owner.txt was already recording the holder's pid=. lock_is_stale just never read it. It checked only the directory mtime against a 2 h ceiling, so a crashed install wedged every later build for up to two hours — matching the reported ~40 min of waiting for another process to install ... with no fbuild process alive. The data needed to fix this was already on disk.

lock_is_stale now reclaims a lock when its owner is gone. Three ordered reasons, deliberately ordered by confidence:

1. the directory vanished;
2. the owner process is dead (new);
3. the age ceiling — kept as the backstop for what liveness can't answer (a hung-but-running peer, a missing owner record).

Because the failure mode of getting this wrong is deleting a live install's lock, every probe fails safe toward "alive":

• PID recycling is covered by also recording exe_stem=. A recycled PID now running some other program means the original owner is gone; a genuinely-alive fbuild keeps its lock.
• pid_executable_path returning None (permission denied, race) counts as alive, never dead.
• Records written before exe_stem existed fall back to liveness alone.
• create_dir and write_lock_owner are two steps, so a waiter can observe the gap. A lock with no owner record is left alone for a 30 s grace, then reclaimed — otherwise a crash between those two calls still wedged for the full 2 h.

The issue also notes the diagnostic trap (a directory-as-mutex makes File.Open return "Access is denied", reading as "held by a live process"). The reclaim path now logs the owning PID when it reclaims, so the state is legible.

Part 2 — stale endpoint records

• daemon stop now clears the port/pid/status/owner-claim records when it finds no daemon. The early return was the bug: "daemon is not running" is exactly when those records are known to be garbage, and it was the one path that left them untouched.
• Found an omission the issue didn't mention: graceful shutdown removed pid, port and owner claim but not daemon_status.json. So daemon status kept describing a dead PID after a clean exit, not just a crash.
• get_daemon_port() no longer trusts the port file verbatim — it's ignored when the recorded owner is provably gone, with the same fail-safe rules as above (a missing claim, including the startup window where the port file exists but the claim doesn't yet, counts as alive).

What part 2 does not fix — stated plainly

This does not explain "clients keep dialing a dead port". The port is derived deterministically from (version, cache identity) per #1009, so discarding the port file yields the same number — which is precisely why deleting the files by hand didn't help you either. That symptom is a daemon-respawn failure, and pinning it down needs a repro I don't have.

So: this PR makes the recorded state stop lying about a dead daemon, which is what the issue asks for. If you want #1213 kept open for the respawn behaviour, say so and I'll re-open or split it rather than let it close silently.

Part 3 — undiagnosable esptool failure

cached esptool executable ... exited with status 2 carried no output, which is what you hit on Windows with a freshly-installed 5.3.0. The error now includes captured stderr/stdout, and says so explicitly when both are empty rather than leaving the reader guessing which it was.

Tests

9 new, all passing:

• dead PID reclaimed immediately; live PID never reclaimed; recycled PID treated as dead; legacy record without exe_stem; the create/write race window left alone; abandoned-mid-creation lock reclaimed after grace; owner record round-trip
• the endpoint liveness gate fails safe with no claim; the test binary is not mistaken for fbuild-daemon

Verified locally: -p fbuild-packages-fetch -p fbuild-library -p fbuild-paths green, a full workspace check clean, clippy clean with -D warnings.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

@zackees, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 56 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: be674fc2-1e08-4dd3-93dc-13b08c680e8b

📥 Commits

Reviewing files that changed from the base of the PR and between dd1ddd4 and 93ad12a.

📒 Files selected for processing (5)

• crates/fbuild-cli/src/cli/daemon_cmd.rs
• crates/fbuild-daemon/src/main.rs
• crates/fbuild-library/src/library/esptool.rs
• crates/fbuild-packages-fetch/src/install_lock.rs
• crates/fbuild-paths/src/lib.rs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}