feat(vm): list and prune the VM driver's prepared-image cache #5150

User Story

I run OpenShell MicroVM sandboxes on RHEL 9 hosts with KVM and on Macs, for several users per machine, with agent images that are rebuilt often. On one RHEL host, one user's ~/.local/state/openshell reached 41 GB and the root filesystem filled up. Every agent on the host then failed to start with "No space left on device". That was OpenShell 0.1.3; I did not re-measure that RHEL host on v0.1.6, so the 41 GB figure is from the original incident only.

The space was prepared image disks for image IDs that no sandbox used any more, and cache entries from an older OpenShell release. OpenShell gave me no way to see or remove them, so I wrote my own cleanup that guesses the cache file names.

Problem Statement

The VM driver turns each image it boots into a prepared root disk under its state directory, one per image ID (named by a hash of the image ID since 0.1.4, #4342). It never removes one:

• deleting the sandbox, or deleting the image from Docker, leaves the prepared disk;
• after an upgrade, "the driver rebuilds older bootstrap and prepared-image cache entries on first use" (docs/how-it-works/sandboxes/runtimes.mdx), and the old entries stay;
• there is no CLI or API to list the cache, see which sandboxes use an entry, or prune it.

A prepared disk of a typical agent image is about 5 GB, so a few image rebuilds use tens of GB.

Reproduced on v0.1.6 (2026-11-02)

Stock v0.1.6 (openshell, openshell-gateway, openshell-driver-vm darwin-aarch64 release tarballs), vm driver, default state dir under a fresh account (~/.local/state/openshell/vm-driver), macOS 15.8 on Apple silicon. I created one --keep sandbox per image with openshell sandbox create --name <n> --keep --from <image> -- true, and deleted it with openshell sandbox delete <n> straight after, measuring du -sk (allocated) and stat -f %z (apparent size) of vm-driver/images after each step:

openshell sandbox list showed no sandbox at the end. Every deleted sandbox left its prepared disk behind: sandbox-prepared-rootfs-ext4-umoci-v4-openshell-0.1.6-image-account-<hash>/rootfs.ext4 (54 to 373 MiB allocated and 1.0 to 1.3 GiB apparent each, for these small images), or the bootstrap rootfs for ubuntu:24.04, which was also the configured bootstrap image; nothing was removed by delete or by time. The six prepared entries add up to 6.7 GiB apparent size, which is what a file system without sparse files, or a backup tool, would count. I did not reach 41 GB, because these images are small; a real agent image is larger, and the entries grow by one per image ID.

No list or prune command exists in v0.1.6:

• openshell --help and openshell sandbox --help list no cache, image, prune or gc command (sandbox: create, get, list, delete, stop, start, suspend, resume, exec, connect, upload, download, ssh-config, provider, template);
• openshell image cache list fails with error: unrecognized subcommand 'image';
• openshell-gateway --help has only generate-certs and config, and openshell-driver-vm --help has no cache command either. The only "cache" in all help output is the JWKS key cache TTL.

Impact / Why This Matters

• Hosts fill up silently, which breaks every agent on the host, not only sandboxes.
• A downstream tool has to compute OpenShell's internal cache names to clean up. That breaks whenever the naming changes, as it did in 0.1.4.
• Admins of shared hosts cannot tell whose cache entries are safe to remove.

Proposed Design

1. A driver capability with two calls, for example ListImageCache (entry, image reference and ID, size, last used, cache version, sandboxes that use it) and PruneImageCache (by entry, by unused-since, or all unused). The gateway exposes them, and the CLI gets openshell image cache list and openshell image cache prune [--unused] [--older-than DUR] [--dry-run].
2. At driver start, remove entries whose cache version this release no longer boots, unless a sandbox still references them.
3. An optional size cap in [openshell.drivers.vm], for example image_cache_max_gib, that evicts least recently used unreferenced entries.
4. Docker, Podman and Kubernetes drivers can report "no cache" for the capability.

Acceptance Criteria

• openshell image cache list shows each prepared disk with its image, size, last use and the sandboxes that use it.
• openshell image cache prune --unused removes only entries no sandbox references, and reports the bytes freed.
• After an upgrade that changes the cache version, the old entries are removed at driver start or listed as prunable.
• An entry in use by a stopped kept sandbox is never removed.

Alternatives Considered

• Downstream cleanup by file name (what I do today). It depends on internal names and breaks on changes such as fix(vm): bind cached registry images to verified manifest digests #4342.
• Middleware, interceptors, providers. They do not touch driver storage, so none applies.
• Only a size cap. Simple, but admins still cannot see what the space is.

Notional example

$ openshell image cache list
ENTRY         IMAGE                       SIZE    LAST USED   SANDBOXES
img-7c1e...   agent-claude:1.4 (sha256:…)  5.1 GB  2 days ago  dev-a
img-02ab...   agent-claude:1.3 (sha256:…)  5.0 GB  9 days ago  -
boot-v3       bootstrap (old cache v3)    512 MB  -           -
$ openshell image cache prune --unused
removed 2 entries, 5.5 GB freed

Agent Investigation

• Cache naming: crates/openshell-driver-vm/src/driver.rs prepared_image_cache_identity and bootstrap_image_cache_identity; fix(vm): bind cached registry images to verified manifest digests #4342 (v0.1.4).
• Docs: docs/how-it-works/sandboxes/runtimes.mdx, the paragraph on rebuilding older cache entries after an upgrade.
• I found no existing issue for listing or pruning the VM image cache (searched issue titles on 2026-11-02). feat(cli): add sandbox prune command to delete ERROR-phase sandboxes #2594 (prune ERROR-phase sandboxes) is about sandboxes, not the cache.

Checklist

• I've reviewed existing issues and the published docs
• This is a design proposal, not a "please build this" request