build: evaluate Nix to improve dx and builds

[drew]

We've had requests from the community to use tools such as nix. Evaluate nix and other tools for improving build infra and DX. Goal is to create more reproducible builds and reduce bash script sprawl that has accumulated.

[SDAChess]

Nix evaluation — proposed adoption plan

After evaluating a local Nix devshell on a branch, the proposal is to adopt Nix incrementally over five PRs. Each PR has a single visible exit criterion and leaves the existing mise-based workflows untouched until the last step.

The workpackages

PR 1 — Nix devshell (Linux + macOS) + Cargo bindeps refactor + unit tests

crates/openshell-driver-vm/Cargo.toml declares openshell-sandbox as a build dependency with artifact = "bin". build.rs reads CARGO_BIN_FILE_OPENSHELL_SANDBOX instead of locating a pre-compressed file in OPENSHELL_VM_RUNTIME_COMPRESSED_DIR. The supervisor stub is removed and mise run vm:supervisor is no longer part of the build path; cargo produces the supervisor as part of building the VM driver. This relies on Cargo's unstable artifact-dependencies feature, which is gated to nightly.

Native libs (libkrun.so, libkrunfw.so.5, gvproxy) come from Nix derivations on both Linux and macOS. Darwin support for the libkrun build is not yet available in the flake and is planned work as part of this PR.

Exit: nix develop -c cargo test --workspace --lib green on Linux + macOS.

PR 2 — E2E and integration tests under nix flake check

Add kubectl, helm, k3d, skaffold, python313, uv, and supporting utilities (socat, jq, zstd, openssh). Bring integration and e2e coverage under nix flake check so a single command covers unit, integration, and e2e on Linux. Where the current e2e/ shell scripts can be expressed as Nix checks they are migrated; remaining scripts run from inside nix develop.

Migration scope is open: the boundary between checks expressed in Nix and retained shell scripts is determined during the PR, and is expected to be the largest body of work in this sequence.

Exit: nix flake check green on Linux, covering unit tests, integration tests, and the e2e suite.

PR 3 — Formatting and lint

Expand treefmt-nix to cover rustfmt, ruff format, shfmt, prettier, on top of nixfmt. Wire clippy, ruff check, markdownlint-cli2, and the license-header script into nix flake check. Extend scripts/update_license_headers.py to handle .nix. nix fmt and nix flake check become the entry points.

Exit: nix flake check covers the same checks mise run pre-commit runs today.

PR 4 — Build all release-binary artifacts via Nix

Add crane. Per-binary derivations for openshell-cli, openshell-server, openshell-sandbox, openshell-driver-vm, openshell-router, openshell-tui. Python wheel via maturin under crane (or uv2nix for fully Nix-resolved Python deps). Cross-compile to x86_64-linux, aarch64-linux, aarch64-darwin.

Out of scope: .deb / .rpm / docker images / Helm chart packaging — downstream of the binaries; separate PR if needed.

Exit: every binary CI currently ships is reproducible via nix build .#<name>.

PR 5 — Replace CI, decommission mise

New GitHub Actions workflow using a Nix installer + a store cache. Lint/test/build jobs run via nix flake check and nix build. Shadow alongside the existing mise CI for ~a sprint. Once stable: delete mise.toml, mise.lock, Dockerfile.ci; sweep references in tasks/, .agents/skills/, CONTRIBUTING.md, AGENTS.md.

Exit: CI green on main with mise removed.

Open decisions

1. Toolchain scope (PR 1) — bindeps requires nightly, so openshell-driver-vm needs a nightly toolchain regardless. Through rust-overlay, a nightly toolchain is exposed the same way as the current stable pin (a one-line change in flake.nix / rust-toolchain.toml), so this is a preference question rather than a feasibility one:

   • Whole workspace on nightly — one cargo build covers everything with a single toolchain.
   • Nightly scoped to openshell-driver-vm via a per-crate rust-toolchain.toml — the rest of the workspace stays on the current stable pin.

   Not settled.

2. CI cache (PR 5) — hosted, GHA-scoped, or self-hosted?

Next step

PR 1 is in progress.