Skip to content

test(cli): pin daemon crash-recovery with a real-process regression test (#1228) - #1232

Merged
zackees merged 1 commit into
mainfrom
fix/1228-crash-recovery-test
Aug 2, 2026
Merged

test(cli): pin daemon crash-recovery with a real-process regression test (#1228)#1232
zackees merged 1 commit into
mainfrom
fix/1228-crash-recovery-test

Conversation

@zackees

@zackees zackees commented Aug 1, 2026

Copy link
Copy Markdown
Member

Closes #1228

Trace results (issue ask 1)

Reproduced the crash scenario empirically on current main (Windows, dev mode, real binaries): started a daemon, hard-killed its PID, ran the next client invocation with tracing.

The residual respawn failure no longer reproduces. The client logged daemon not running, starting..., respawned a sibling fbuild-daemon on the same deterministic port, and the request succeeded ~5.4 s after the kill (old PID 34184 → new PID 38452, port 58866). The chain that fixed it since the 2.5.2 incident: the #1227 owner-liveness gate (stale records no longer masquerade as a live daemon), #830 sibling-binary spawn, #1010/#1159 spawn single-flight, and the #1208 wall-clock readiness budget.

Fix (issue ask 2)

No code defect remains on the traced path — the deliverable is the regression pin (below). Two scoping notes recorded for follow-up rather than silently dropped:

  • Broker path untested: ensure_daemon_running tries running-process broker adoption before the direct path. If a stale broker lease ever hands out a dead endpoint, the client errors without falling back to direct spawn (daemon_client.rs:762 propagates). The default mode is direct fallback, so this is theoretical until the broker ships enabled; the test pins the direct path explicitly via RUNNING_PROCESS_DISABLE=1.
  • Spawn-path environment discovery: the daemon grandchild's env is rebuilt from the OS user baseline (user_baseline_environment), which discards any test/caller-provided HOME/USERPROFILE. This is why the test cannot HOME-isolate the daemon and instead skips when a live dev daemon owns the real root. It also implies a user with an overridden HOME gets CLI and daemon disagreeing on the fbuild root — worth a look if identity-derived-port mismatch reports ever show up.

Regression test (issue ask 3)

crates/fbuild-cli/tests/daemon_crash_recovery.rs — real-process, #[ignore]-gated like the #1159 daemon test (runs under bash test --full / --include-ignored):

  1. Brings a daemon up through the production acquisition path (fbuild daemon restart with FBUILD_DAEMON_PORT=free port, FBUILD_CACHE_DIR=tempdir).
  2. Kills it uncleanly (terminate_pid), leaving stale records.
  3. Asserts the very next CLI invocation respawns and reaches a new daemon PID on the same port.
  4. Stops the daemon (also exercising fix(packages,daemon): reclaim dead-owner install locks; stop leaving stale endpoint records (#1213) #1227's record clearing).

Safety: skips (never contends) when a live dev daemon holds the real ~/.fbuild/dev root-owner lock, and when no sibling fbuild-daemon binary exists.

Validation

  • soldr cargo test -p fbuild-cli --test daemon_crash_recovery -- --include-ignored1 passed (16.7 s, real binaries, Windows).
  • soldr cargo clippy -p fbuild-cli --tests -- -D warnings — clean.

🤖 Generated with Claude Code

…est (#1228)

The #1228 trace on current main shows the residual respawn failure no
longer reproduces: after an unclean daemon kill, the next client
invocation detects the dead endpoint (via the #1227 owner-liveness gate),
respawns a sibling fbuild-daemon on the same deterministic port, and the
request succeeds (~5s observed). Pin that behavior end-to-end: spawn the
real binaries, terminate the daemon hard, and assert the very next CLI
invocation reaches a NEW daemon PID. Ignore-gated like the #1159
real-daemon test; skips rather than contending when a live dev daemon
owns the real ~/.fbuild/dev root, since the production spawn path
rebuilds the daemon env from the OS user baseline and cannot be
HOME-isolated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Aug 1, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@zackees, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 54 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 2342fed7-8a58-4123-8f55-a0b1a228a841

📥 Commits

Reviewing files that changed from the base of the PR and between 3e9dc5c and baf48c6.

📒 Files selected for processing (2)
  • crates/fbuild-cli/tests/README.md
  • crates/fbuild-cli/tests/daemon_crash_recovery.rs

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@zackees
zackees merged commit 7a4cd2f into main Aug 2, 2026
91 of 93 checks passed
@zackees
zackees deleted the fix/1228-crash-recovery-test branch August 2, 2026 00:20
zackees added a commit that referenced this pull request Aug 2, 2026
…erited from #1232)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
zackees added a commit that referenced this pull request Aug 2, 2026
…break inherited from #1232)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
zackees added a commit that referenced this pull request Aug 2, 2026
The ban_std_pathbuf dylint denies new PathBuf usage and the allowlist
forbids new entries; the #1232 test slipped in with four explicit
mentions and broke the Dylint gate on this branch. Use NormalizedPath.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@fastled-project-sync fastled-project-sync Bot moved this to Triage in FastLED Tracker Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Triage

Development

Successfully merging this pull request may close these issues.

fix(daemon): client redials a dead deterministic port — respawn after crash never recovers (split from #1213)

1 participant