ci: re-enable setup-soldr cook/cargo-chef prebuild for faster builds

[zackees]

Context

In PR #253 (chore(deps): adopt upstream soldr/zccache/setup-soldr updates) we added prebuild-deps: none to every zackees/setup-soldr@v0 invocation across 11 workflow blocks. setup-soldr's default since their PR #140 is prebuild-deps: cargo-chef — it runs soldr cook to pre-warm target/deps/ with all transitive dependency artifacts on the cache-keyed Cargo.lock hash. When the cache hits, the user's first cargo build starts hot.

For fbuild we had to disable it because the cook step pre-populates target/ and the subsequent soldr cargo build -p fbuild-cli -p fbuild-daemon fails:

##[warning]setup-soldr detected a pre-populated shared target directory;
  a subsequent `soldr cargo build` using the same `--target-dir` may fail
  with a missing .rmeta error - see README 'Known limitations'.
...
error: could not compile `fbuild-cli` (bin "fbuild")

Documented upstream as a known shared-target-dir limitation when you cook → cargo build -p <subset>.

Why this matters

fbuild's CI is heavy. Every PR runs the full board matrix (80+ jobs), and each one compiles fbuild's entire dependency tree from cold (zccache helps but only for downstream compilations, not for the dep tree itself). The cook layer's cache-key (cook-${os}-${arch}-${libc}-rustc${release}-f${flagsHash}-l${lockHash}-soldr${ver}) is content-addressable on Cargo.lock — it would hit across branches automatically and turn "cold first build" into "warm-start build" for every CI run.

Per setup-soldr's design doc (PR #140), the win is measured in tens of seconds per job on cache hit. Across 80+ jobs that's potentially 30+ minutes of CI savings per PR.

Options to enable

1. Separate target dir for the cook step. Cook writes to target-chef/; the user's build still uses target/. Loses the warm-start benefit but plays nice with -p <subset> builds. Probably not what we want.

2. Build the full workspace instead of -p fbuild-cli -p fbuild-daemon. Cook prebuilds the whole transitive dep graph; if we then build the whole workspace, every .rmeta cook created is what cargo expects to see. Trade-off: workspace builds compile more code than the subset we need for distribution.

3. Tune prebuild-deps-flags to constrain chef's prebuild to the same -p subset (prebuild-deps-flags: \"-p fbuild-cli -p fbuild-daemon\"). Aligns cook's output with the actual build. Smallest change, but requires verifying chef respects -p for the prebuild phase.

4. Wait for upstream setup-soldr fix. Likely lowest-effort if it lands.

Option 3 is probably the right first try.

Acceptance criteria

• Pick a per-workflow strategy (likely Fix ESP32-S3 PSRAM board configuration in board JSON database #3, with cdc on boot warning #2 as fallback for template_native_build.yml which already builds -p fbuild-cli -p fbuild-daemon).
• Remove prebuild-deps: none from one workflow as a probe (suggest check-ubuntu.yml); verify clean run.
• Measure the time saved on a cache hit vs current cold-deps baseline. Threshold to roll out: ≥ 30s wall-clock per job on a hit.
• If clean, drop prebuild-deps: none workflow-by-workflow.
• If the upstream issue gets fixed first, ride that fix instead of working around it locally.

Reference

• Original disable: PR chore(deps): adopt upstream soldr/zccache/setup-soldr updates #253 (ci(workflows): disable setup-soldr prebuild-deps (workaround upstream regression)).
• setup-soldr PR that introduced default-on cook: feat(cook): default-on prebuild-deps via cargo-chef zackees/setup-soldr#140.
• Cook design doc: zackees/setup-soldr commit body of bd14e46.

🤖 Generated with Claude Code

[zackees]

Research update — what's still blocking re-enabling cook

Current state in fbuild

• 11 zackees/setup-soldr@v0 invocations across 10 workflow files in .github/workflows/ (bench-205.yml has two). All 11 carry prebuild-deps: none. None pin to a SHA — every push to setup-soldr@v0 reaches CI immediately.
• prebuild-deps-flags is unset everywhere (default --release).
• Only msrv.yml sets toolchain: 1.94.1 explicitly; the rest read it from rust-toolchain.toml.
• Workaround was added in PR chore(deps): adopt upstream soldr/zccache/setup-soldr updates #253 (merge commit 60a4ae3). template_native_build.yml already uses --target-dir "${PYTHON_TARGET_DIR}" for the python wheel build — that's the same isolation pattern soldr feat(bench): #205 AC#5 enforce <= 50 ms warm threshold in bench-fastled-examples #228 recommends as the safe path.

Upstream status — the underlying bug is still open in code, "closed" only in tracking

Two relevant upstream issues, both closed without a code fix:

• soldr feat(bench): #205 AC#5 enforce <= 50 ms warm threshold in bench-fastled-examples #228 "rust-plan restore leaves missing .rmeta on repeated soldr cargo build sharing target-dir" — closed 2026-04-24. The closing comment is essentially "use a separate --target-dir". The expected behavior the reporter asked for ("rust-plan restore merges with the existing on-disk target artifacts and produces a valid Cargo target tree") was never implemented.
• setup-soldr Replace Rust toolchain trampolines with soldr #53 "Repeated soldr cargo builds in one job can leave missing .rmeta artifacts when sharing target-dir" — closed 2026-05-11 by setup-soldr PR Fix Windows response file temp ownership #55, which only adds a CI warning annotation and a "Known limitations" README section. Diagnostic, not a fix.
• soldr fix(build-info): create parent dirs for nested example env names #410 "soldr cargo build poisons target// with orphan .rmeta files (local-first regression)" — closed 2026-05-21 (the day before PR fix(daemon): register test-emu endpoint OperationGuard and propagate exit code to CLI #140 made cook default-on).

No PR in either repo between PR #140 (2026-05-22) and soldr@d3ba5f72 / setup-soldr@99463f86 (2026-05-23/24) addresses the rust-plan restore code path that produced fbuild's failure. So option #4 (wait for upstream) is currently blocked.

Tool versions (as of 2026-05-24)

Tool	Latest release	fbuild floor
setup-soldr	v0.9.2 (2026-05-20)	@v0 (floating)
soldr	v0.7.31 (2026-05-24)	bundled by setup-soldr (default 0.7.30+)
zccache	1.9.1 (2026-05-24)	>=1.8.2 in pyproject.toml; zccache-artifact 1.8 in Cargo.toml
cargo-chef	v0.1.77 (2026-03-03)	n/a — invoked transitively via soldr cook

soldr cook requires soldr ≥ 0.7.x (soldr PR #409, merged 2026-05-21). The setup-soldr default already satisfies this. No minimum is tied to fbuild's 1.94.1 toolchain or the dylint nightly-2026-03-26 pin.

cargo-chef capability

cargo chef cook --package <name> is supported — Recipe records pub package: Option<Vec<String>> and forwards --package <name> to the inner cargo call. So option #3 (prebuild-deps-flags: "-p fbuild-cli -p fbuild-daemon") is plausibly correct at the chef layer. But the failure in PR #253 was not at the chef layer — it was at the soldr rust-plan restore step that runs after chef and before the user's cargo invocation. That code path is still the soldr #228 bug.

Concrete blockers

To flip even one workflow from prebuild-deps: none to default cook:

1. Either a code fix lands in soldr's rust-plan restore (no candidate PR exists as of 2026-05-24), or
2. A workflow uses option cdc on boot warning #2 (build the full workspace, no -p filter on either side) — sidesteps the bug because cook's output and the user's build target the same package set. This is the lowest-risk first probe and matches cargo-chef's "same cwd, same args" guidance, or
3. A workflow uses option Esp32-s3 PSRAM use #1 (separate --target-dir for the user build) — the workaround soldr feat(bench): #205 AC#5 enforce <= 50 ms warm threshold in bench-fastled-examples #228 itself recommends. Already proven to work in template_native_build.yml for the python wheel. Costs the warm-start benefit on target/ but keeps cook's value for other workflows.

Option #3 from the issue (prebuild-deps-flags: "-p ...") is unverified — chef supports it, but the failure mode is downstream of chef.

Suggested first probe

Pick check-ubuntu.yml (lowest-blast-radius — single Ubuntu job, no board matrix). Drop prebuild-deps: none, leave the existing soldr cargo check --workspace --all-targets (workspace-wide, no -p). If that reproduces the rust-plan restore error, fall back to option #1 with a dedicated --target-dir. Measure wall-clock against the current cold baseline; threshold from the issue is ≥ 30s/job.

Also worth doing: pin every zackees/setup-soldr@v0 to the v0.9.2 SHA before the probe so a future upstream regression can't silently turn off the experiment.

🤖 Generated with Claude Code

[zackees]

Upstream trackers filed

For end-to-end visibility into when this is unblocked:

• setup-soldr#159 — Default-on cook (PR #140) breaks downstream consumers with -p subset builds — track until rust-plan restore handles prepopulated target/ zackees/setup-soldr#159
  Action-side ask: either make cook's filter align with the consumer's -p subset, auto-detect collision and skip cook, or revert default to prebuild-deps: none.

• soldr#480 — Re-track: rust-plan restore fails on cook-prepopulated target/ (originally soldr#228, regression blast-radius widened by setup-soldr#140) zackees/soldr#480
  Re-tracker for the rust-plan restore code path that produces the missing-.rmeta failure (originally soldr#228; closed without fix; regression blast-radius widened by setup-soldr#140's default-on cook).

Either issue closing-with-fix unblocks the option #4 path in this issue (wait for upstream). Until then, options #1 (separate --target-dir, already proven in template_native_build.yml) and #2 (build full workspace, no -p filter) remain the only working enablement paths.

🤖 Generated with Claude Code

[zackees]

Closing — all five acceptance criteria are now met via the recent cascade work:

• ✅ Per-workflow strategy chosen = option Fix ESP32-S3 PSRAM board configuration in board JSON database #3 from this ticket. template_build.yml's setup-soldr config is now:

  cache-preset: foundation     # = build-cache + prebuild-deps: soldr-cook + ...
  prebuild-deps-flags: ""      # debug-profile cook to match `cargo build -p fbuild-cli -p fbuild-daemon` (debug)
  cache-key-suffix: fbuild-rust-debug

  The empty prebuild-deps-flags is the documented debug-profile signal; it makes cook's fnone key match what the workspace actually compiles, so the artifacts cook produces are exactly the artifacts cargo wants to reuse.

• ✅ prebuild-deps: none removed from every relevant workflow — not one at a time, but all at once via the shared template_build.yml (which all ~71 board build-*.yml workflows inherit) plus the core direct-Rust workflows (check-ubuntu/macos/windows, msrv, docs, dylint, fmt, acceptance-205, bench-205, template_native_build).

• ✅ Measured time savings: PR perf(ci): lighter board caches — shared soldr cook, drop heavy target-cache #280 captured the win on a 72-board CI matrix — warm per-board wall-clock dropped from ~5 min → ~2 m 23 s (~55% reduction), with 17/20 zccache hits on the Rust workspace compile. Cache footprint dropped from ~6.7 GB to ~437 MiB total. Well above the 30 s/job threshold this ticket asked for.

• ✅ Roll-out done: PR perf(ci): lighter board caches — shared soldr cook, drop heavy target-cache #280 (initial enable + coarse shared key) and PR ci: adopt setup-soldr cache-preset: foundation in template_build #289 (migrate to cache-preset: foundation after setup-soldr v0.9.17 shipped the preset).

• ✅ Upstream regression fixed: perf(cook): cargo-chef 0.0.1 version normalization poisons the warm build-cache for first-party crates zackees/soldr#566 / PR LPC845-BRK: deploy must always unwedge the shared VCOM (pyocd/SWD flash leaves CDC in Windows error 31) #565 — soldr cook no longer leaves the workspace in the stubbed 0.0.1 skeleton state (the root cause of the shared-target-dir issue referenced in this ticket's body). Shipped in soldr 0.7.44 → setup-soldr v0.9.16, and now v0.9.19 ships soldr 0.7.45 + zccache 1.11.7 with additional depgraph drift detection.

Closing as completed.