Reclaim orphaned disk: stale clone directories and left-behind images

[blooop]

Question

Two kinds of orphaned disk have no owner in any existing plan, and both grow monotonically: (1) stale clone directories — metadata is append-mostly (PR #129 found 39 entries for 6 live workspaces), and a clone whose devpod workspace is gone just sits in <cache>/repos/... forever; (2) built images left behind when workspaces are deleted (dl --purge = devpod delete --force per workspace + remove_tree(cache_dir), dl.py:702-766 — it never touches docker image anything). Issue #88's reconciler owns the third kind (leaked named volumes, ~9.9GB dead measured on its thread) and devpod-record adoption — do not duplicate that. Decide what devlaunch should do about kinds (1) and (2).

Decide against the principles and the established purge contract (PRs #129/#134/#136: delete only what devlaunch created, refuse to delete the only copy of unsaved work, remove what is permitted and name the rest):

1. Kind (1) stale clones: where does detection live (dl --ls showing them? a dl --prune verb? part of --purge?) and what is the deletion contract — PR Report what a workspace holds, and refuse to delete the only copy of it #134's unsaved-work guard (workspace_state.py) is the ready-made safety predicate; Build: persist the devpod workspace id instead of re-deriving it, and reconcile the orphans #88's "report, never silently delete" is the precedent for orphans.
2. Kind (2) images: does devlaunch ever remove images at all, or only report what its workspaces left behind (image ids/tags + sizes via docker CLI), leaving removal to the user? Devlaunch didn't build the layer store; deleting shared layers can break other containers — reporting-only is the likely maintainable answer, but decide it explicitly.
3. Whether any of this runs automatically (e.g. detection during --ls) vs only on demand — nothing may slow the launch hot path.
4. State how the outcome is tested (fixture cache with a stale clone + live clone; the stale one is detected, the live one and any unsaved-work clone are never deletion candidates).

Code changes graduate as build ticket(s); this ticket only decides. Pointers: dl.py:702-766 (purge), devlaunch/worktree/storage.py (metadata), devlaunch/workspace_state.py (unsaved guard), issue #88 (volume/record scope — the boundary).

[blooop]

breadcrumb: grilling started — fresh subagent resolving against the principles.

[blooop]

agent-decided (Long-term maintainability):

Kind (1) stale clones: a new on-demand dl --prune that classifies every clone directory three ways and removes only the unambiguously orphaned, clean ones — reusing remove_tree, the unsaved-work guard and the report-then-confirm shape verbatim. Kind (2) images: devlaunch never deletes an image and never inventories one; it prints one line naming the boundary and pointing at docker system df.

Measured first, on the host, read-only (nothing deleted)

fact	number
clone directories under <cache>/repos/*/*/	52 (+ 7 .bare caches)
live devpod workspaces	17
clone dirs no live workspace sources	37
bytes in those 37 (hardlink-aware, st_blocks)	4.00 GB — against 7.86 GB live
bare caches, all 7 together	0.08 GB
metadata.json worktree records	49 (for 17 live workspaces — #129's "append-mostly" is still true and worse)
clone dirs with no metadata record	3
metadata records pointing at a missing dir	0
stale clones holding unsaved work	13 of 37 — incl. wayfinder-devlaunch-kilarabo (2 unpushed commits) and devlaunch-aid-lerozazo (2 unpushed commits + dirty tree)
devpod records pointing at a missing/stub folder (the #88 shape)	0 today (was 36/39 on #88's thread)
docker system df	images 55 / 131.3 GB, 86.5 GB reclaimable; local volumes 43 / 43.18 GB reclaimable; build cache 13.88 GB
cost of a full orphan scan	devpod list 418 ms + dir walk 3.5 ms + 37 unsaved probes 593 ms = 1017 ms

Two of these decide most of the ticket. 13 of 37 stale clones hold unsaved work, so a naive "no workspace points here → delete" would have destroyed unpushed commits on this very host — PR #134's guard is not a formality here, it is load-bearing. And 1017 ms is roughly two warm launches, which settles sub-question 3 on its own.

1. Kind (1): where detection lives, and the deletion contract

Not --purge, not --ls, a verb of its own

--purge is not the seam: it already removes the whole cache, stale clones included. The problem is that the only way to reclaim 4 GB of dead clones today is to also destroy the 7.86 GB of live ones and every bare cache. Folding orphan handling into --purge would change nothing about that.

--ls is not the seam either, and this is a constructive-modeling refusal (principle 3), not a taste call. --ls and --ls --json are a list of devpod workspaces; an orphaned clone has no workspace, so it could only appear there as a row whose id, state and lastUsed are null — a tag-plus-null-fields shape for "this row is not actually a workspace". Two different things sharing one table, distinguished by sentinels. Rejected. --ls stays the workspace list.

So: dl --prune, mirroring --purge's existing shape exactly — print the plan, name what is being left standing and why, confirm, -y/--yes to skip. Nothing new to learn, no new machinery: it reuses remove_tree (dl.py:553), Refusal (dl.py:510), list_workspaces (dl.py:1503), holds_unsaved_work (workspace_state.py:144) and storage.remove_worktree (storage.py:372). The name's confusability with --purge was weighed and accepted: git remote prune and docker … prune both already mean "remove what nothing references any more", which is precisely the semantics, so a reader in a year arrives already knowing it. --gc was rejected because in git it also connotes running automatically, which is exactly what sub-question 3 says no to.

The predicate, and the trap in it

#129 asked "is this workspace mine?" and answered devpod → cache (source path contained in the cache dir). --prune asks the reverse — "is this clone still referenced?", cache → devpod — and that direction is only sound if devpod's source records are trustworthy. #88 measured 36 of 39 devpod records pointing at folders that no longer exist. Under that state, a live workspace's healthy clone at the new path is sourced by nobody and would be a deletion candidate. Today's host reads 17/17 healthy, but the failure mode is recurring, not historical: any future id/path change, a hand-moved cache, or a partially-applied #88 reconciliation reproduces it.

So the classification is a three-constructor sum, not a boolean:

Referenced(workspace_id)          # a live devpod workspace sources this exact path
Orphaned(unsaved: str | None)     # nothing sources it, and no record ties it to a live workspace
Disputed(workspace_id)            # devpod knows this workspace, but sources it somewhere else

Disputed is reached when the metadata record for this directory names a workspace_id devpod still lists, while devpod's source.localFolder for that id is some other path — i.e. exactly #88's shape, detectable from data already in hand (list_workspaces() + storage.get_worktree_by_workspace_id), with no new state and no schema change. It is never a deletion candidate, not even under --force; it is reported and pointed at #88.

#136 set the bar for adding a constructor: three outcomes may only be two decisions. Here they really are three — delete it, keep it, or go fix your records — and collapsing Disputed into Orphaned deletes live work while collapsing it into Referenced silently hides disk. It earns its place. Note also that putting unsaved inside Orphaned rather than beside the classification makes "unsaved work on a referenced clone" unrepresentable: the guard only exists where it can be acted on.

Two details a builder will otherwise get wrong:

• Compare resolved paths, not strings. Path.resolve() both sides. A symlinked cache dir (or XDG_CACHE_HOME under a symlinked /tmp) makes every clone look unreferenced otherwise — a silent total-loss bug in the dangerous direction.
• Take the per-repo lock. workspace_clone.py:184 and :243 hold repo_manager.lock_path(owner, repo) while creating a clone. A concurrent cold launch has a fully-populated clone on disk before devpod up registers a workspace for it — so unlocked, --prune can delete a clone out from under a launch in progress. --prune must hold the same lock per repo dir while it classifies and removes that repo's clones.

What it does and does not remove

• Removes Orphaned(unsaved=None) clone directories, with remove_tree, not shutil.rmtree. Clones are precisely the directories a container wrote as uid 1000, so partial refusal is the normal case here rather than the exotic one Remove what a purge is permitted to, and name what refused #136 was written for. Refusals are named, exit 1 when any remain (Remove what a purge is permitted to, and name what refused #136's contract, unchanged).
• Refuses Orphaned(unsaved=...) by default, naming what deleting would destroy in Report what a workspace holds, and refuse to delete the only copy of it #134's own words, and says how to insist. --force promotes only this arm. Measured, that is 13 of 37 on this host — mostly pixi.lock churn written by containers, which is noise, but two of them are unpushed commits, and devlaunch cannot tell the difference. Erring conservative costs the user a --force; erring the other way costs work that cannot be recovered (dl --purge deletes only the workspaces devlaunch created #129's framing, unchanged).
• Drops the metadata record for each clone it removes, and drops records whose directory is already gone. This is where dl --purge deletes only the workspaces devlaunch created #129's "39 entries for 6 live workspaces" (now 49 for 17) finally gets an owner, and it is five lines inside the loop that is already running.
• Never touches .bare. Measured at 0.08 GB for all 7 repos, and it is the thing that makes the next clone fast. Say so in the docstring so nobody adds it later.
• Never touches anything outside <cache>/repos, never a devpod workspace, never a container, never an image, never a volume.

2. Kind (2): images — report the boundary, not an inventory

Three options were weighed.

Delete them. Rejected outright. Devlaunch did not build the layer store, does not know which layers are shared with a user's own images, and the whole purge trilogy exists to establish that devlaunch deletes only what it created. It did not create these.

Inventory them — image ids/tags + sizes via the docker CLI, attributed to devlaunch's workspaces. This is the option the ticket expected to win, and it loses on principle 1 after checking what the attribution would actually rest on. I inspected the labels on a real devpod-built image: there is no devpod or devlaunch label anywhere — only upstream dev.containers.* and devcontainer.metadata from the base image. The sole handle is the name shape: repository <workspace-id>-<5 hex>, tag devpod-<32 hex> (e.g. devlaunch-wayfinder-devlaunch-pobezene-210ad:devpod-587f7760…). That is an undocumented devpod internal. An inventory built on it is a guess that will go quietly empty — or quietly wrong — the day devpod changes its naming, and the specific failure mode is a wrong list of image ids that a user pastes into docker image rm. It also spends a new docker-CLI dependency, a new parser and a new report surface to restate, worse, a number Docker already computes exactly.

Name the boundary and point at the tool that is right by construction. Winner. After --purge and after --prune, one line:

devlaunch does not manage Docker images or volumes — the containers these workspaces used may still be holding disk. docker system df shows what Docker is holding.

Zero docker dependency (no subprocess, so nothing to fail when Docker is absent or the daemon is down), zero parsing, zero rot. It appears at the one moment the user is thinking about disk. It deliberately points at docker system df — information — and not at docker image prune -a, because that is unscoped: recommending it would destroy images devlaunch did not create, the exact footgun #129 removed from --purge. Being aggregate is also what keeps it from colliding with #88, which owns volumes: one line covers both kinds without devlaunch enumerating either.

For scale, this line is pointing at 86.5 GB of reclaimable images and 43.2 GB of volumes on this host, against the 4.00 GB --prune itself reclaims. Devlaunch's honest contribution here is knowing when to say a number is not its to compute.

Also considered and rejected: making future attribution possible by passing a devlaunch label through devpod up (dl.py:1732-1750). It adds a concept, couples devlaunch to another devpod flag, does nothing about the 86.5 GB already on disk, and its only purpose would be to enable the deletion this decision just ruled out. Recorded so nobody re-treads it; the seam is named in case a future ticket disagrees.

3. Automatic or on demand: on demand only, and nowhere near the launch path

Measured: 1017 ms for a full scan (418 ms devpod list + 3.5 ms walk + 593 ms of unsaved probes over 37 clones, one git subprocess each). That is about two warm launches. It scales with the number of stale clones, i.e. it gets slower exactly as the problem gets worse.

• Never on the launch path. Not negotiable.
• Not on --ls either, not even as a hint line. --ls already pays one devpod status per workspace and Report per-workspace disk size in dl --ls #153 is about to add a size walk to it; a third, unrelated question does not belong there, and a hint would be a second reporting surface for something --prune already reports.
• dl --prune answered n is the report. That is why there is no separate --orphans flag: one concept, one command, and the read-only view is the first half of the command that acts.
• --prune costs exactly one devpod list and zero devpod status (workspace state is irrelevant to whether a clone is referenced). Pin that in test/test_devpod_spawn_counts.py alongside test_purge_reads_the_workspace_list_exactly_once.

Composition with #153

--prune must import the hardlink-aware directory-size function that #153 lands and must not grow its own. #153 owns bytes-per-live-workspace in --ls; --prune owns bytes-not-attached-to-any-workspace; they share exactly one primitive and nothing else. That makes #153 a soft ordering prerequisite — build it first, and --prune gets its per-orphan and total byte figures for free, in the same units and with the same hardlink semantics the --ls column uses. If #153 slips, --prune ships without sizes rather than duplicating the primitive.

4. How the outcome is tested

Fixture, in a scratch XDG_CACHE_HOME only (never XDG_CONFIG_HOME), following #134's technique of real git repositories with a local bare as their remote — that technique earned its keep by catching a git log --not --remotes bug no mock would have. Cache contains repos/o/r/.bare plus four clones, with list_workspaces() stubbed:

1. referenced — a stub workspace sources it → Referenced, never a candidate, under --force too.
2. orphaned, clean — nothing sources it → Orphaned(None), the only thing removed.
3. orphaned, dirty — an unpushed commit and an uncommitted file → Orphaned("…"), refused by default with the description printed, removed only under --force.
4. disputed — metadata ties it to workspace id W; the stub lists W but sources it elsewhere (Build: persist the devpod workspace id instead of re-deriving it, and reconcile the orphans #88's shape) → Disputed, never removed, reported, not even under --force.

Assertions beyond the four arms:

• .bare is never classified and never removed.
• A metadata record whose directory is already gone is dropped; records for surviving clones are untouched.
• Idempotent: a second run removes nothing and prints "nothing to prune", exit 0.
• Resolved-path comparison: a cache reached through a symlink still classifies the referenced clone as Referenced (this is the total-loss regression, so it gets its own test).
• Refusal path: a clone directory the test process cannot empty is reported by path and reason and the exit code is 1, while its clean siblings still go — skipped under root, with the same comment Remove what a purge is permitted to, and name what refused #136 used (root is refused by nothing, so under root it would pass with the fix reverted).
• Concurrency: --prune blocks while repo_manager.lock_path(owner, repo) is held, so it cannot delete a clone a cold launch is still filling.
• Round-trip count pinned in test_devpod_spawn_counts.py: one devpod list, zero devpod status.
• The boundary line is asserted in the --purge and --prune output — it is documentation that can rot, so a test holds it.

Red-first, in the #129/#136 style: land the classifier with the command untouched and show the command tests red; then mutate the guard off (Orphaned always treated as clean) and show the dirty-clone test red; then mutate Disputed into Orphaned and show the disputed test red. Each mutation reverted, each recorded in the PR.

Recommendation on #88 — leave it off this map

Do not adopt #88 onto map #139. It is a correctness ticket (workspaces unaddressable, initializeCommand failing) that predates this map, with its own root-cause analysis, its own retracted-premises record and its own Done-when; its disk effect is a side benefit. Adopting it would drag the riskiest item in the area — a migration that rewrites devpod's own workspace.json — onto a latency-and-disk map, and #88's thread already admits the end-to-end verification needs a Docker host and a human.

The dependency people would adopt it for has been designed out: the Disputed arm makes --prune safe under exactly the state #88 exists to repair, so --prune does not wait on #88 and does not partially do its job. What is needed is a one-way link, not ownership: --prune's disputed report points at #88 by number. If #88 ever lands, Disputed becomes rare but must stay — it is also the shape of a hand-moved cache and of any future path-scheme change, which is the class of bug that produced #88 in the first place.

Out of scope for this decision

• Volumes (43.18 GB reclaimable here) — Build: persist the devpod workspace id instead of re-deriving it, and reconcile the orphans #88's, per the map's own out-of-scope ruling. The aggregate docker system df pointer is deliberately aggregate so it does not duplicate it.
• Image-layer sharing / per-workspace image duplication — ruled out by the map. Worth one recorded observation for whoever revisits it: devpod builds a separate image per workspace even when the devcontainer config hash is identical (five devlaunch-* images all tagged devpod-587f7760…, five distinct image ids, 1.74 GB each). Whether those share layers underneath is the Docker layer store's business, which is precisely why the map ruled it out. Recorded, not reopened.
• Build cache (13.88 GB) — Docker's, same reasoning as images, and docker system df names it too.
• unmigrated-clones.txt / orphaned-workspaces.txt, the sidecar lists migration.py writes into the cache root — --prune makes the first largely redundant, but tidying them is migration's business, not this ticket's.
• Automatic/TTL-based pruning, disk-pressure sensing, and any --prune on the launch path — ruled out above; disk-pressure sensing is already its own "not yet specified" item on the map.

[blooop]

breadcrumb: resolved and indexed on the map. Graduated two build tickets: #159 dl --prune (blocked by #153, whose hardlink-aware size function it imports rather than duplicating) and #160 the Docker boundary line (blocked by #159; small enough to fold into #159's last slice). The #88 recommendation — leave it off this map, one-way pointer only — is recorded in the map's Out of scope.