handoff(#1154): finish cold-cache reset with soldr-style daemon ownership

[zackees]

Handoff objective

Continue and finish #1154: make every benchmark “cold” trial actually clear compiler-object caches for Arduino CLI, PlatformIO, and fbuild, then ship and validate the production benchmark run.

This handoff records the requester’s latest architecture direction: copy soldr’s daemon ownership/start strategy. Do not put cross-process locks in zccache’s compile/object hot path.

Requester decisions

• fbuild clean cache ... remains the desired public operation.
• Zccache object access must keep using zccache’s internal in-memory synchronization.
• A cross-process lock is acceptable only for fbuild-daemon startup/lifetime ownership.
• Prevent a thundering herd when many callers observe no daemon.
• A live-but-wedged owner must not create an immortal “zombie lock”: after a bounded grace period, verify the owner’s PID/process identity, terminate/displace that owner, and let the OS release the lock. Never blindly delete/break an advisory lock file.
• Use soldr’s implementation as the reference strategy.

Current local state

An uncommitted implementation exists in:

• worktree: .claude/worktrees/issue-1154-cold-cache
• branch: fix/issue-1154-cold-cache
• base: 71096e2e; it is currently 2 commits behind origin/main (9284e60a at handoff time)

The useful benchmark/CLI/docs work should be retained. The first lifecycle-lock implementation must be redesigned before commit.

Changed areas:

• benchmark sequencing and metadata: bench/fastled-examples/src/build_comparison.rs
• split tests: bench/fastled-examples/src/build_comparison_tests.rs
• benchmark docs/workflow trigger
• CleanScope::Cache, parser test, CLI docs
• current clean.rs stop/delete/restart orchestration
• current generic fbuild-core::file_lock and two-lock daemon protocol (rework/remove as described below)

Generated local Docker harness files under .clud/docker-build/ are untracked and must not be committed.

An ignored soldr checkout is available at .extern-repos/soldr, inspected at soldr commit 86377878efa22536f08b31305f5492b06d8bb6da with vendored zccache 74f59cbfe86f5b7d9bbb734bc4c661c42c0c7988.

What already works

The benchmark runner now performs untimed preparation before every cold trial and no cleanup before the immediate warm trial:

• Arduino CLI: arduino-cli cache clean, then remove configured build dir.
• PlatformIO: pio system prune --cache --force, then pio run ... --target clean.
• fbuild: fbuild clean cache ....

Published JSON/HTML/docs now accurately say that Arduino/PlatformIO download/HTTP caches are removed while fbuild package archives and installed packages/toolchains remain.

Tests previously passed in the isolated Linux build container:

• benchmark runner: 6 tests
• CLI parser: clean_cache_scope_is_accepted
• initial clean module tests
• benchmark clippy with -D warnings
• generic shared/exclusive lock tests: 3 tests

The last full CLI compile was interrupted/restarted during redesign and is not final validation.

Review results

Docs re-review: no findings.

Rust review found two blockers in the first implementation:

1. New locks protect only new daemons. Older version-keyed fbuild daemons share ~/.fbuild/<mode>/zccache but do not hold those locks, so deletion can still race a legacy daemon.
2. CLAUDE.md currently says “No file-based locks — all locking through daemon's in-memory managers.” The requester has now explicitly narrowed the exception to fbuild-daemon startup/lifetime and asked to copy soldr’s strategy. Update the durable architecture rule/documentation so the exception is explicit; do not leave a silent contradiction.

Earlier review findings already addressed in the working diff and must remain covered:

• positively prove the daemon/cache owner exited before raw deletion;
• validate cache identity before destructive action (FBUILD_DAEMON_PORT must not target a different mode/root);
• distinguish explicit busy refusal from lost shutdown response and restore availability on failure.

Soldr strategy to copy

Reference checkout: .extern-repos/soldr.

Version-blind lifetime ownership

crates/soldr-daemon/src/daemon/lifecycle.rs:

• RootOwnershipGuard takes an exclusive fs2 lock on a stable, version-independent root-owner.lock.
• The daemon acquires it before starting the embedded zccache service and holds it for the daemon’s complete lifetime.
• Competing daemon versions therefore cannot own the same cache root simultaneously.
• OS process death releases the lock automatically; the pathname can remain without being authoritative.

Spawn-herd single flight

Same file, acquire_spawn_lock / try_spawn_detached*:

• a short-lived nonblocking exclusive .spawn.lock elects exactly one spawner;
• losers re-check liveness and do not spawn another daemon;
• the winning caller holds the spawn lock through PID publication and readiness (bounded to 5 seconds);
• daemon bind/startup still independently fails closed if root ownership is busy.

Stale/zombie recovery

Same file, displace_stale_daemon*:

• a version-blind PID claim records PID plus executable path;
• liveness and executable identity are verified before any signal;
• graceful IPC shutdown gets a bounded wait (5 seconds in soldr);
• a verified stale/hung soldr daemon then receives TERM/forced termination and another bounded wait;
• the OS releases root/spawn locks when the process exits; code never “unlocks” another process’s live file descriptor.

Copy the semantics, including version arbitration: an older client must not evict a newer healthy daemon merely because its local binary is newer on disk; a newer client may displace an older owner after the grace/identity checks.

Do not copy soldr’s current soldr clean raw-directory deletion (crates/soldr-cli/src/cache/mod.rs) as the fbuild safety model. The relevant reference is soldr’s lifecycle/root-owner/spawn-herd code.

Zccache API finding

Fbuild pins zccache 1.12.14 at commit 4b13676bb3a76c87d2272b2d9ccb76e821734762.

Zccache already has a synchronized internal Request::Clear implementation in daemon/server/handle_clear.rs. It clears in-memory indexes, on-disk artifacts, metadata/depgraph state, request caches, and statistics. Newer zccache also has clear-publication barrier tests.

However, the public embedded ZccacheService API exposes only start/compile/stats/flush/shutdown (plus maintenance in newer soldr’s vendored revision); it does not expose clear() at either the fbuild pin or soldr’s inspected vendored revision.

A small upstream zccache ZccacheService::clear() -> ClearReport API would enable the best reset path: fbuild-daemon refuses while operation_in_progress, invokes zccache’s internal clear through an HTTP endpoint, and never restarts. If that dependent change is pursued, use .extern-repos/zccache, open/link the prerequisite zccache PR both ways, then pin fbuild to the merged commit. Version-blind fbuild root ownership is still needed because different fbuild versions currently use different HTTP endpoints while sharing the same zccache root.

Recommended implementation order

1. Rebase the worktree onto current origin/main; preserve unrelated user/agent changes.
2. Replace the generic reset-gate/activity-lock design with soldr-style, version-blind per-cache-root daemon ownership acquired before listener/heavy zccache initialization and held for daemon lifetime.
3. Add soldr-style short-lived spawn-herd election to the direct launch path. Confirm running-process broker adoption is already single-flight; root ownership remains the final safety gate for broker/direct races.
4. Upgrade the fixed daemon owner claim so it contains PID + executable identity + mode/cache identity + endpoint/version needed for safe displacement. Verify the live process identity before signaling.
5. Add bounded graceful shutdown, verified forced displacement, and bounded readiness. The timeout handles a live hung owner; dead owners release OS locks automatically.
6. Handle rollout from legacy daemons that do not hold root ownership. Before the first destructive reset, discover/refuse or safely displace every legacy fbuild-daemon that could own the active mode/root. Do not assume the current version-keyed HTTP endpoint is the only owner. Add a real-child-process regression for this transition.
7. Prefer adding/exposing zccache embedded clear() and a daemon-owned fbuild reset endpoint. If that is not feasible in this change, the fallback stop/delete/restart flow must hold spawn election across the full transition and acquire exclusive root ownership before deletion.
8. Make fbuild clean cache always continue with existing clean all target cleanup, retain packages/toolchains/fbuild archives, remain idempotent, and restore a usable daemon even when deletion fails.
9. Run formatter, focused tests, clippy/review gate, push PR, merge, verify issue closure.
10. After merge, wait for benchmark-build-comparison.yml; inspect the three-trial log and published JSON/HTML to prove cleanup precedes every cold trial and not the warm build.

Required tests

• spawn herd: many concurrent starters produce one daemon owner/spawn;
• root ownership is version-blind and exclusive across real processes;
• abrupt owner death releases ownership;
• live hung verified owner is displaced only after timeout;
• unrelated/recycled PID is never signaled;
• older client does not evict newer healthy daemon;
• legacy daemon without the new lock prevents raw deletion or is safely displaced;
• busy fbuild operation refuses reset without deletion;
• identity/mode mismatch refuses deletion;
• cleanup failure still restores daemon availability;
• cache cleanup is idempotent and leaves sibling fbuild roots/package archives intact;
• every benchmark cold trial gets the exact cleanup sequence outside timing; warm gets none.

Acceptance / closeout

This issue is a handoff companion to #1154, not a replacement for its product acceptance criteria. Close both only after the implementation PR is merged and a production benchmark run proves the stronger cold-cache contract.