Skip to content

feat(coreutils): the toolbox is on by default - #445

Closed
raphaelvigee wants to merge 6 commits into
raphaelvigee/coreutils-template-driverfrom
raphaelvigee/coreutils-default-on
Closed

feat(coreutils): the toolbox is on by default#445
raphaelvigee wants to merge 6 commits into
raphaelvigee/coreutils-template-driverfrom
raphaelvigee/coreutils-default-on

Conversation

@raphaelvigee

@raphaelvigee raphaelvigee commented Aug 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

coreutils: true becomes the default on the exec/bash driver. Fifty-five lines, and the decision the whole stack exists for.

A toolbox nobody enables fixes nothing: the point of shipping these utilities is that a recipe behaves the same on Linux and macOS without anyone opting in. Leaving it off by default would mean the divergences in #438 keep biting everyone who hasn't heard of the flag.

What this changes

Every exec/bash target gets heph's cp, install, sed, sha256sum and the rest on its PATH — behind everything the environment provides, so a declared tool and a runner's environment both still win. Where the target's PATH previously resolved to a host binary that heph now supplies, it resolves to heph's.

coreutils: false is the escape hatch, and opting out is itself folded into the def hash — a cache-visible decision rather than a silent one.

Why it is its own PR

It has its own revert line. If the flip proves too disruptive it can go back without losing the toolbox, and #446 can be reverted without losing the flip. Bundling either pair would force an all-or-nothing rollback of a change this broad.

The stack

Merge bottom-up, and gh stack sync after each one lands — master is squash-only, so the rebase will conflict and the resolution rule in CLAUDE.md applies.

PR What
6 #446 drop the host directories from a target's PATH — breaking
5 #445 the toolbox on by default
4 #440 the template rule and the tmpl applet
3 #453 grep, find, xargs, sed, tar, gzip, zstd
2 #438 the crate, the entry points, the shim directory
1 #451 the runner PATH seam ← base, targets master

Only #451 builds automatically: since #449 a stacked PR is skipped unless it carries ci/force-ci. Every layer was checked locally on its own — cargo build --workspace --all-targets, cargo clippy --workspace --all-targets, and its unit tests — not just at the top of the stack.

🤖 Generated with Claude Code

https://claude.ai/code/session_0181d7hhbYWXT42Z1KQPM29Q

@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/coreutils-tmpl branch from 21f261c to f328a20 Compare August 29, 2026 22:44
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/coreutils-default-on branch from 26dfdb8 to 76b42ed Compare August 29, 2026 22:44
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/coreutils-tmpl branch from f328a20 to 49acb17 Compare September 3, 2026 16:26
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/coreutils-default-on branch from 76b42ed to 51b3ae2 Compare September 3, 2026 16:26
@raphaelvigee @claude
fix(execrunner): a runner carrying the environment out of band still …
f7b1a63 
…gets the PATH prefix

`PathPolicy.prefix` — the target's declared tools, and (with the builtin
toolbox) heph's own utilities — was composed onto `spec.env` only *after* the
runner's `prepare`. That is correct for every runner that leaves the
environment where it found it: `wrap` puts its own `PATH` in `spec.env`, and
`session` hands `spec.env` to the client, which forwards it to the agent.

It is wrong for a runner that carries the environment out of band. `oci` turns
the target's environment into `docker exec -e KEY=VALUE` arguments and returns
the *docker client's* environment in its place, so the composition decorated
the client process and the target inside the container got neither its declared
tools nor the builtins.

It failed silently, which is the part that matters: the container falls back to
the image's own tools, so a recipe that reaches for a declared tool keeps
working — with a different binary than its cache key names.

Compose the prefix onto the environment handed *to* the runner as well. For a
runner that leaves the environment in place nothing changes: the composition
after `prepare` still orders the runner's own `PATH` behind the prefix, and
`join_path` dedupes, so doing both is idempotent.

A runner that replaces `PATH` outright then owes its environment's own value
behind what it was handed. The oci runner reads the image's `PATH` from the
container config (`docker inspect`) and restores it behind the target's —
without that, entering a container would strip `/usr/bin` from every target
that declares a tool. Best-effort: an unreadable value costs the image's
directories, which is a degraded environment rather than a wrong one.

Found while regrounding the builtin-coreutils work against the runner model:
the toolbox puts heph's utilities in exactly this prefix, so the gap would have
shipped as "the builtins are on every target's PATH, except inside a
container", with nothing to see.

Tests: the seam's carried `PATH` and the `docker exec` argv are both asserted
without a daemon, and a docker-gated e2e pins the real thing — a declared tool
runs inside the container, and `cat` still resolves from the image.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0181d7hhbYWXT42Z1KQPM29Q
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/coreutils-tmpl branch from 49acb17 to 93b9688 Compare September 3, 2026 16:57
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/coreutils-default-on branch from 51b3ae2 to ac12896 Compare September 3, 2026 16:57
raphaelvigee and others added 5 commits September 3, 2026 19:05
@raphaelvigee @claude
feat(execrunner): a PATH tier for what heph supplies, behind the envi…
16a8597 
…ronment

`PathPolicy.prefix` is what the *target* declared, so it leads and wins over
everything the environment ships. There is no counterpart for what *heph*
supplies — the builtin utilities — which wants the opposite ordering: fill a
gap the environment leaves, never shadow a binary that environment deliberately
ships. A workspace that names a devenv or nix runner pinned those tools on
purpose, and heph's `sed` arriving in front of them is not a service.

Add `PathPolicy.suffix`, composed last:

    the target's tools ++ what it declared ++ the runner's PATH ++ heph's

It is composed into the environment *this process* spawns, so it deliberately
does not reach a runner that carries the environment out of band. That is the
wanted behaviour rather than a gap: those entries are host paths, and a
container's filesystem is not this one — a shim directory of symlinks into a
binary built for the host platform is worse inside an image than absent. The
prefix still is carried, because it is the target's own and lives on paths the
runner is responsible for making visible.

Inert here: nothing populates it yet. The builtin-coreutils work is the caller.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0181d7hhbYWXT42Z1KQPM29Q
@raphaelvigee @claude
feat(coreutils): ship POSIX utilities in the binary
11d00de 
A heph target runs its recipe with a sandbox PATH of the host's
directories, so `cp` means GNU coreutils on Linux and a BSD userland on
macOS. The sandbox isolates files; it does nothing about the two hosts
disagreeing. `install -D` does not exist on macOS at all, `wc` pads its
output so `[ "$(wc -l < f)" = 3 ]` is Linux-only, and `sort` collates by
locale — which silently changes build *outputs*, not just exit codes.
A build system whose contract is "same inputs, same outputs" cannot
leave the tools that produce those outputs undeclared and host-defined.

So heph ships its own. `crates/coreutils` compiles 40 MIT-licensed
uutils/coreutils applets into the binary and reaches them by re-exec —
`heph __coreutils <applet>`, or a symlink named after the applet
(argv[0] dispatch, busybox style). Dispatch is the first thing in
`main()`, before logging, clap, the self-update check or any runtime: a
build may invoke `cp` thousands of times and each one is a fresh process.
It runs ahead of the `__supervisor` and `__runner-exec` branches, so it
is tested against those argv shapes — eating one would kill the sidecar
with a broken pipe rather than an error anyone could read.

The shims are one directory under the heph home, materialized once per
(toolbox version, binary path) and contributed to `hexecrunner`'s
`PathPolicy` as a tier directly behind the target's own tools. A recipe
that provisions its own `sed` still gets that `sed` — the builtins only
displace the host's. Per-sandbox cost is one extra PATH entry: nothing
written per target, nothing staged, nothing to tear down.

`plugin-exec` takes a `CoreutilsShims` closure and a version rather than
depending on `crates/coreutils`. The driver needs a directory to put on
PATH and a number to hash, not knowledge of what an applet is — and most
of the workspace links `plugin-exec`, so the dependency would have
dragged forty utility crates into `engine`, `e2e` and `plugingo-e2e`
builds and test binaries (`cargo tree -p engine | grep -c uu_`: 45 before,
0 after). The closure also keeps materialization lazy, so a `heph query`
never touches the filesystem for it. `coreutils: true` with no supplied
shims is a hard error, not a shrug: running against the host's utilities
while the cache key claims heph's is the silently-wrong-build case.

Off by default (`coreutils: true` on the exec/bash driver). Turning it
on changes what every recipe's `cp` resolves to, and it moves every exec
target's cache key.

Cache correctness. The utilities are on a target's PATH without being
declared, and nothing can tell which of them a shell command will invoke
without parsing it, so `COREUTILS_VERSION` goes into the def hash whole
or not at all — bumping it invalidates every exec target in every
workspace, which is release-gated, not routine. Nothing is hashed while
the toolbox is off, so a workspace that never opts in keeps today's keys.

Cost, measured on this tree (aarch64-apple-darwin, rustc 1.96, the real
release profile): +7.55 MiB stripped, +19.3%. Trimming does not help —
dropping the eight lowest-value applets saves 1.37 MiB of that, because
the cost is a shared uucore+clap floor, not the applet count. Startup is
unchanged: `heph --version` already costs ~6 ms, essentially all of it
before `main`, and the applets add nothing to it.

Verified end to end on darwin/arm64: with the toolbox on, a bash target
resolves `cp` to `.heph3/coreutils/v1-<hash>/bin/cp` and reports
`cp (uutils coreutils) 0.10.0`; with it off the same target gets
`/bin/cp`, which rejects `--version`.

`heph tool coreutils list | which <name> | run <name> …` is the
diagnostic surface — shadowing `cp` silently is exactly the kind of
magic that produces an unanswerable bug report.

Not in this change: sed, grep, find, xargs, tar, gzip and the template
renderer; removing the host directories from the sandbox PATH; and
making the toolbox the default. Design and measurements in
docs/COREUTILS.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0181d7hhbYWXT42Z1KQPM29Q
@raphaelvigee @claude
feat(coreutils): the rest of the toolbox — grep, find, xargs, sed, ta…
2f656af 
…r, gzip, zstd

The seven utilities a build recipe reaches for that uutils does not carry, so
the toolbox covers a real recipe rather than only the easy half of one. Each is
the same shape as the applets already in the table — a name, a `run` fn, golden
tests — and each is here for a divergence that bites in practice:

- `grep`, `find`, `xargs` (`grep-*`, `ignore`): BSD `find` has no `-printf`, and
  the two `xargs` disagree about `-r`. Built on the crates ripgrep uses, so the
  matcher is a known quantity rather than a fresh regex engine.
- `sed`: the divergence that started this. GNU takes an optional `-i` suffix,
  BSD requires one, so `sed -i 's/a/b/' f` edits in place on Linux and eats the
  next argument as a filename on macOS.
- `tar`, `gzip`, `zstd`: **reproducible by construction** — no mtimes, no uid/
  gid, no name-ordering left to readdir, and compression levels pinned. An
  archive is usually a build *output*, so a tar that embeds a timestamp is a
  target that can never be a cache hit twice.

Landed as one change rather than three: they touch the same four files
(`Cargo.toml`, the applet table, a module apiece, `COREUTILS.md`) and ask the
reviewer the same question — is this the right crate, and is the behaviour
right — so the applet set reads better judged in one pass than in three.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0181d7hhbYWXT42Z1KQPM29Q
@raphaelvigee @claude
feat(template): a declared template rule, and the tmpl applet on the …
31f29a2 
…same renderer

Filling in a config file should not require a shell. `template(name, src, out,
vars)` renders a declared template with declared variables, in-process
(`crates/builtins/src/plugintemplate`) — inputs and outputs fully declared, no
`sed` pipeline, no quoting rules to get wrong.

The renderer itself lives in `crates/template` so the rule and the `tmpl`
applet are the same code rather than two implementations of one syntax. `tmpl`
covers the mid-recipe case (and `envsubst`); the rule is the one to reach for,
because a rule's inputs are in the cache key and a mid-recipe render's are not.

Two constraints the renderer carries deliberately:

- **No non-deterministic builtins.** No `now()`, no randomness — a template
  that can render differently from the same inputs is a target that can never
  be a cache hit.
- **`TEMPLATE_FORMAT_VERSION` in the def hash**, so a minijinja upgrade that
  changes rendering invalidates rather than silently diverging.

`vars` is hashed from a `BTreeMap` because it arrives as a `HashMap`, and
hashing it in iteration order would move the def hash on every run.

The rule and the applet land together because they are one feature: a shared
renderer used two ways. Splitting them would put a crate in trunk with one
caller and review the same rendering semantics twice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0181d7hhbYWXT42Z1KQPM29Q
@raphaelvigee @claude
feat(coreutils): the toolbox is on by default
d2f6297 
A toolbox nobody enables fixes nothing. The point of shipping `cp`,
`install` and `sha256sum` in the binary is that a recipe behaves the same
on Linux and macOS without anyone opting in, so `coreutils:` now defaults
to true.

`coreutils: false` is the escape hatch for a workspace that genuinely
wants the host's tools. Opting out is folded into the def hash as well, so
it is a cache-visible decision rather than a silent one: turning it off
invalidates the targets it affects, which is what you want, because their
outputs may change.

A driver built by `new_exec`/`new_bash` rather than from options still
starts off. Those are what test harnesses and the shell fallback use, and
they have no heph home to materialize shims under — switching the toolbox
on there would trip the "no shim directory was supplied" assertion rather
than doing anything useful. There is now a test pinning that, because the
distinction is not obvious from either constructor.

The flip caught one existing test: a spawn-diagnostic test built its
driver from options and never supplied shims, so it started failing the
moment the default changed. It now opts out explicitly — it is about the
error message, not about PATH composition. That is the assertion working
as intended: a host that builds from options must supply shims, and
degrading to "no builtins" instead would run the target against the
host's utilities while its cache key claimed heph's.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0181d7hhbYWXT42Z1KQPM29Q
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/coreutils-default-on branch from ac12896 to d2f6297 Compare September 3, 2026 17:30
@raphaelvigee
raphaelvigee changed the base branch from raphaelvigee/coreutils-tmpl to raphaelvigee/coreutils-template-driver September 3, 2026 17:31
This was referenced Sep 3, 2026
Merged
Merged
Merged
Merged
Merged
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/coreutils-template-driver branch from 31f29a2 to 8c87364 Compare October 16, 2026 17:05
@raphaelvigee
raphaelvigee removed this pull request from stack #454 October 16, 2026 17:05
@raphaelvigee

raphaelvigee commented Oct 16, 2026

Copy link
Copy Markdown
Member Author

Folded into #446 while restacking onto master: turning the toolbox on by default and taking the host directories off PATH are two halves of one change — neither is worth reviewing or merging without the other. Both now live in #446's single commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@raphaelvigee