Orphaned volumes with no surviving record

[blooop]

Question

Orphaned docker volumes whose workspace is already gone: reclaim them, or write down why not?

Since devlaunch#325 a delete removes the named volumes its devcontainer created, and the names come from devpod's own record of what it substituted, never guessed from a pattern. That closed the leak going forward. It leaves a population behind: 39 orphaned volumes holding 37.28 GB measured on one machine (devlaunch#324), plus anything a bare devpod delete outside dl strands from here on.

The bind: an orphan has no surviving record, because the record died with the workspace. So the one mechanism the codebase trusts is unavailable exactly where the bytes are, and the alternative is matching <basename>-pixi and dind-var-lib-docker-<id> patterns against docker volume ls, which is the guessing the existing design refuses on the grounds that a made-up name is somebody else's disk. Principle 1 is on the side of that refusal; principle 2 wants the 37 GB.

Three candidate answers, and the third is the one worth thinking hardest about:

1. Leave them, and say so in docs/cleanup.md with the measurement, so the bytes are documented rather than silent. Cheapest, and principle 1's default.
2. Reclaim from the pattern behind an explicit flag, reporting every name before removing it. Principle 1 permits an explicit human answer; the question is whether a pattern is ever a good enough basis to offer, given dind-var-lib-docker-<devcontainerId> could plausibly belong to a devcontainer devlaunch never opened.
3. Write our own record at create time, so the orphans of the future are reclaimable by reading rather than guessing, and the existing 39 stay case 1. This is the only option that gets principle 2 without weakening principle 1, at the cost of new state devlaunch has to keep correct. Weigh what happens when that record and reality disagree, since a record that can be wrong is a guess with better manners.

Note a volume is unlike an image: deleting one that still matters is data loss, not a rebuild.

How the answer gets tested: whichever way it goes, a test asserts no docker volume rm is issued for a name that came from anywhere other than the sanctioned source, and the docs guard covers the new paragraph.

[blooop]

agent-decided (principle 1, no data loss): candidate 1 for the 39 that exist, candidate 3 for every one after them, and candidate 2 is not built at all.

The existing population stays. There is no route to it that is not a pattern, and the pattern is refused for the reason the design already gives. Its row in #455's table says nothing, deliberately, carries the 39 volumes / 37.28 GB measurement and #324's cross-reference method, and points at docker volume ls the way the images paragraph points at docker system df.

Why candidate 2 is refused as a design and not merely deferred. The ticket asks whether a pattern is ever a good enough basis to offer. It is not, and the sharp form is stronger than "somebody else's disk": dind-var-lib-docker-<devcontainerId> is written by the docker-in-docker feature, and devcontainerId is devpod's hash over folder, architecture and config (#447 traced CalculatePrebuildHash). Every devcontainer tool that runs that feature produces a name of that shape, and the ones devlaunch never opened are precisely the ones that are in no list devlaunch can read. So subtracting the live devpod workspaces does not narrow the candidate set towards ours; it narrows it towards volumes whose owner devlaunch cannot see, which is the wrong direction. And the offer is itself the harm: a flag whose report is a list of names the user also cannot attribute puts a decision in front of somebody with no means to make it, and a volume is not an image, so the cost of the wrong answer is not a rebuild. Better to print no list than a list nobody can check.

Candidate 3, and the reason it is not new state. It is a second read of one document. Both names come from workspace_result.json -> SubstitutionContext.LocalWorkspaceFolder and .DevContainerID; today that read happens once, at delete time, immediately before devpod delete destroys the file. This adds a second read of the same two fields at the end of a completed up, kept. Every name devlaunch ever hands docker volume rm still originates in a substitution devpod performed and wrote down. Nothing is synthesized, so the sanctioned-source test the ticket asks for holds unchanged.

devlaunch has already made this exact trade twice, and both precedents are load-bearing here:

• metadata.json's devpod_workspace_id (Build: persist the devpod workspace id instead of re-deriving it, and reconcile the orphans #88): a derived value written down precisely so that when the derivation moves, the population does not become unaddressable at once. Same failure, same fix, one artifact over.
• flows/provision/verdict_cache.rs: a per-workspace marker under devlaunch's own cache, written after a completed up, about facts read out of that same workspace_result.json. The doctrine for a devlaunch-kept copy of a devpod fact is already written there, including "a devpod home copied between machines" as a named hazard.

Where it lives: a sibling marker under the cache, not a metadata.json field. Three reasons. metadata.json is the pinned shared format with a schema version, so a field there is a format change with a migration story, where a marker is versioned by "unparseable means absent". Its worktree records are what --prune drops, so the record would be destroyed by the command best placed to use it, and ordering inside one command is a weaker guarantee than two independent files. And it must not be a field on the verdict marker either: that marker's presence means AlreadyProvisioned, a volume copy has to be written on every completed up regardless of the verdict, and one file with two write conditions makes one condition govern the other.

What the record has to survive, one at a time.

• devpod delete outside dl. Survives, and this is the entire population candidate 3 buys. devpod removes its own workspace directory and leaves the volumes (Does devpod record the image it built for a workspace #447 measured it and read the delete path: plain docker rm, no -v). devlaunch's copy is in devlaunch's cache and is untouched.
• A crash between the volume existing and the record being written. Does not survive, and cannot, for anybody. SaveWorkspaceResult runs at the tail of devPodUp after every failing branch has returned (cmd/up.go:452), so a create that dies leaves the dind volume standing with no record anywhere on the machine. That is VolumeSweep::NothingNamed today; a copy neither widens nor closes it. Two things make the residual small rather than open-ended: the names are deterministic for a folder plus config, so any later completed up of the same workspace records the same names and the volume is reclaimable again, leaving only workspaces whose up never once completed and which are then deleted. Do not close it by predicting devcontainerId. A predicted name is a guess with a hash on it, and the hash's inputs are devpod internals with no published contract.
• --purge. Survives if the ordering is a property rather than a habit: delete the workspaces first, which sweeps from devpod's own live record, then remove the cache directory holding the copies. Interrupted between the two, the copies that are lost belong to workspaces already swept, so nothing is lost that was not already reclaimed. The reverse ordering is the one that loses bytes permanently, and it should be unrepresentable rather than merely not written.
• Two machines sharing nothing. Volumes are per-daemon, copies are per-cache, so the honest case is fine. The dishonest one is a cache copied or restored between machines, and verdict_cache's mtime-equality anchor is unavailable here because the anchor dies with the workspace. The guards are different and there are two, neither of them a pattern. A copy is consulted only where devpod does not list that workspace; and docker itself refuses to release a volume any container still refers to. Measured today on docker 29.7.2: docker volume rm --force on a volume held by a container in state exited exits 1 with volume is in use, and on an absent name exits 0. So a stale copy is a no-op and a copy that names a live volume is a reported refusal, never a deletion.
• A scratch XDG_CACHE_HOME run. This is what makes candidate 3 fit the map's hard constraint, and it cuts both ways. A reclaim that reads copies under the cache finds nothing in a scratch cache, so a scratch run is harmless by construction. That is the property Remove the workspace's pixi and dind volumes on delete #325 said a pattern sweep in --purge could never have, and it is now the property this route has. The cost, which is not a regression but is worth a sentence in docs/development.md beside the scratch-cache advice: a scratch run that creates a workspace writes its copy into the cache it then throws away, so remove such workspaces with dl <ws> rm before discarding the cache or their volumes outlive the only record of them.

So: a record that can be wrong, and how it is wrong. It can be wrong in exactly two ways, and both are handled by something other than trust. It can name a volume already gone, which --force makes a silent success. It can name a volume something else now holds, which docker refuses and which is reported as VolumeRefusal::Docker, the arm that already exists for it. What it cannot do is the thing a pattern does, which is name a volume nobody ever recorded. That is a difference in kind and not in manners, and it is the whole answer to the ticket's framing.

The unit is the record, not the clone directory. Worth stating because it is this map's own lesson (#445). A pass that walked orphaned clones and swept their volumes would reason over an enumeration that does not cover what it affects: a copy whose clone the user already deleted by hand names volumes no clone-shaped walk will ever reach. So the domain is the set of kept copies, and the precondition per copy is that no workspace devpod lists carries that id. --prune is the natural surface, because it is already the command that owns "the workspace is gone and its leftovers are ours", and because today every prune that removes an orphaned clone manufactures a permanent orphan: devpod's record died at the outside delete, so after this the volumes are unreachable forever. The names ride on each individual record rather than as a plan-wide list, per the map's second precedent, so the plan and the acting pass cannot disagree about which volumes belonged to which entry. docs/cleanup.md's "the --prune half of the pair still frees no volume at all" changes with it, and its reason changes rather than disappears: prune still never deletes a workspace, it reclaims the volumes of one devpod has already forgotten.

The type shape, since the answer turns on a distinction between a name that was read and a name that was inferred. Do not put provenance on the name. struct VolumeName { name: String, from_pattern: bool }, or a Provenance enum with a Pattern arm, makes an inferred name representable and then relies on nobody building one, which is a comment wearing a type's clothes. Instead there is no constructor for an inferred name at all: names exist only as the return of Substitutions::volume_names, Substitutions is only ever built by parse_substitutions over bytes from a devpod record or devlaunch's copy of one, so there is nothing to tag. Where the distinction genuinely has to be visible is the report, and it belongs to the occasion rather than the value: a two-arm enum for which read fed this sweep, devpod's live result read before the delete took it away, or devlaunch's kept copy for a workspace devpod no longer lists. Two arms and no third, so adding a pattern arm later is a compile error at every match instead of a branch nobody notices. That is the refusal enforced by exhaustiveness rather than by prose.

Two more shape calls that follow from the same argument. Store the derived names, not the substituted values, as Option<NonEmpty<String>> and never an empty list: freezing the names at read time is what keeps each one traceable to one real substitution, where re-deriving later under a changed template would produce a string devpod never substituted, which is the pattern route arriving by the back door; and the empty list would have to mean either "nothing was recorded" or "recorded nothing", the ambiguity devcontainer_volumes already returns None to avoid. And the copy is dropped once, on proof: after a removal that came back Removed for a workspace devpod does not list, kept on Refused so the retry stays possible. That is a deliberate divergence from verdict_cache's "nothing ever deletes a marker", and the reason it diverges is that there the delete would have been a second unproven mechanism, while here it is conditioned on the removal that made the copy pointless.

Read-only probes, and what this container could not establish. The docker facts above were measured on this repo's devcontainer against its own docker-in-docker daemon, docker 29.7.2-2, DockerRootDir /var/lib/docker, one pre-existing anonymous volume. That daemon is not the host's, and the in-container ~/.devpod holds a default context with the docker provider and no workspaces, so #324's 39 volumes are not visible from in here and were not re-counted. Nothing above depends on re-counting them: the population is case 1 either way, and the two docker behaviours the decision leans on are daemon behaviours, not host state.

Build work. Not #454 and not #455. #455 carries the row for both halves either way. A new build ticket is needed for the copy and the reclaim, and it also answers part of the map's open "does reclaiming beyond clones need sub-scopes" note: no new surface, --prune with the record as its domain.