Build: persist the devpod workspace id instead of re-deriving it, and reconcile the orphans

[blooop]

Work

Stop re-deriving the devpod workspace id. Persist the id dl actually created and address devpod by that, then reconcile the workspaces already orphaned.

Re-scoped. This ticket previously proposed giving run_devpod an explicit cwd. That is retracted — see "Retracted premises" at the bottom. The cause below is observed on the host, not hypothesised.

Root cause

PR #81 (fix/64-workspace-id-parse-boundary — commits 5db5c42, 3875246) changed both the workspace id and the clone-directory leaf from <branch> to <repo-slug>-<branch-slug>-<8-char syllable suffix>. It migrated dl's own metadata and left devpod's records pointing at the old paths.

For devlaunch-main on the reporting host:

side	records the workspace at	actual state
~/.cache/devlaunch/metadata.json	…/repos/blooop/devlaunch/devlaunch-main-zovomobo	35 entries, real checkout, init-host.sh present
~/.devpod/contexts/default/workspaces/devlaunch-main/workspace.json	…/repos/blooop/devlaunch/main	1 file, no .git

The metadata record kept its original created_at (2026-03-01T18:39:40, matching devpod's creationTimestamp) while carrying the new id and path — the same record, rewritten in place. devpod's record was never touched.

Why the failure looks like a missing hook

The old devlaunch/main checkout is gone. On recreate, devpod finds its recorded source folder absent and reconstitutes it from cached workspace_result.json, which contains exactly:

{"name":"devlaunch","initializeCommand":{"":[".devcontainer/claude-code/init-host.sh"]}}

That is byte-for-byte the shape of the lone devlaunch/main/.devcontainer/devcontainer.json left on disk. devpod then chdirs into that folder and runs the hook — but init-host.sh is repo content, not devcontainer config, so it was never restored. Absent → the reported error. Fresh create works because it clones to the new path and registers a workspace whose source matches.

Scale

36 of 39 devpod workspaces on the reporting host have a recorded source folder that is missing (35) or a config-only stub (1). Affects bencher, wayfinder, python_template, rockerc, lifetime-foc-rig, scratch_env, devlaunch. Only three are healthy — all created after the scheme change.

Diagnostic, for anyone reproducing:

devpod list --output json | jq -r '.[] | "\(.id)|\(.source.localFolder)"' |
  while IFS='|' read -r id p; do
    [ -d "$p/.git" ] || echo "BROKEN $id -> $p"
  done

The change

1. Follow the record, don't re-derive (the durable fix)

WorktreeInfo.devpod_workspace_id already exists (worktree/models.py:71) and is never assigned anywhere — grep it: one declaration, zero writers. Had dl stored the devpod workspace id it created and used that to address devpod, changing the derivation would have been harmless.

This is the same lesson the codebase already learned for the path. From remove_workspace_by_id's own docstring:

Following the record matters because the derivation has changed: every workspace created before the current id scheme has a bare branch name as its clone-directory leaf. Re-deriving the leaf here looked for a directory that never existed […] The stored path makes old and new workspaces both removable with no migration.

Exactly that, for the id. Persist devpod_workspace_id when a workspace is created, and have every devpod-addressing path (up, ssh, stop, rm, recreate/restart/reset) use the stored id, falling back to derivation only when there is no record.

2. Reconcile what is already orphaned

A derivation fix does not repair existing state — the 36 broken records predate it. Needs a one-time reconciliation that, per devpod workspace whose recorded source is missing or has no .git:

• if a metadata record for the same (owner, repo, branch) points at a healthy clone, adopt it: rewrite devpod's source.localFolder and populate devpod_workspace_id;
• if no healthy clone exists, say so and let the user drop the workspace — do not silently delete.

Decide deliberately whether this is dl migrate, dl prune --repair, or automatic on startup, and whether writing ~/.devpod/.../workspace.json directly is acceptable versus driving it through devpod CLI. Containers bind-mount the dead source path, so any repaired workspace needs a rebuild regardless — the reconciliation should say that rather than implying the container is reusable.

Must be idempotent, and must cope with metadata that has already been migrated to the new scheme while devpod's has not (the current host state).

3. Version skew

Installed dl on the reporting host is 0.0.9, which still computes branch-name paths, while a main-based run already rewrote metadata.json to the new scheme. Note explicitly what happens when an older dl meets migrated metadata. PR #86 (Release 0.0.10) is what ships PR #81 — the migration should land before or with it, not after.

Done when

• A test asserts devpod_workspace_id is persisted on create and that devpod-addressing paths use the stored id in preference to a derived one.
• A test asserts a workspace created under the old id/path scheme is still addressable after the derivation changes — i.e. the regression that caused this cannot recur. Simulate the scheme change; don't hardcode today's suffix.
• A test covers reconciliation: orphaned record + healthy clone → adopted; orphaned record + no clone → reported, not deleted; running it twice changes nothing.
• pixi run ci green.

Verification honesty

The root cause above is observed from on-host state (devpod list, metadata.json, workspace.json, workspace_result.json) — not inferred. What cannot be verified in a session without a Docker daemon is that a repaired workspace then recreates successfully end to end. That last step is the human's, on the host: dl blooop/devlaunch recreate.

Retracted premises

Recorded so nobody re-treads them:

• cwd on the devpod spawn is not the fix. devpod chdirs into the workspace's recorded source folder before running initializeCommand, identically whether the workspace is addressed by path or by id. dl's working directory never reaches the hook. Verified by running devpod v0.26.1 for real against a throwaway project with the same bare relative hook — no Docker needed, since initializeCommand fires before any container exists.
• The original ticket's supporting facts were read off a stale branch. At dl.py:1367-1370 on main, when the workspace already exists, workspace_spec is a bare workspace id, not a clone path, and custom_id is None. The proposed cwd=workspace_spec would have handed subprocess a nonexistent directory named after a workspace id — on exactly the three subcommands it targeted.
• "The fast path never calls ensure_workspace" is also not the cause. It is true that if workspace_id in existing_ids skips clone_mgr.ensure_workspace(), but adding a clone check there would inspect the new path and find it healthy, while devpod keeps using the old one. No check about the clone catches a disagreement over where the workspace lives.
• recreate/restart fail on a relative initializeCommand #87's "not a stale clone" was right for the wrong reason. It ruled this out on repo-content grounds — correct, there is no commit where the initializeCommand reference exists without the script. But the folder devpod runs in is not a checkout at all; it is a config-only stub devpod itself wrote from cache. Content-level reasoning could not see it.

[blooop]

breadcrumb: claimed; single-ticket map #87. Ruled out missing file, stale clone, and -- <cmd> mis-parsing before filing. Cause is a labelled hypothesis (relative hook path vs devpod's inherited cwd) — the build stage confirms it before implementing. No repro possible in-session: devpod v0.26.1 present, no Docker daemon. Launching /tdd in fresh context.

[blooop]

breadcrumb: Stopped at the "Confirm first" gate. The hypothesis is disproven, so no code was written. Nothing was branched, committed or pushed.

What I did

Ran devpod v0.26.1 for real, in an isolated devpod home, against a throwaway project whose devcontainer.json carries the same bare relative initializeCommand shape (.devcontainer/claude-code/init-host.sh). The hook wrote its own working directory to a log. No Docker daemon is needed to reach the hook — devpod runs initializeCommand before it touches a container, so the run dies after the hook, which is exactly the window this question lives in.

Observed (not reasoned)

1. Create by path, launched from a directory with no .devcontainer in it: hook ran, and its cwd was the workspace's source folder, not the launcher's.
2. up <id> --recreate — the exact shape dl … recreate produces — launched from a different directory with no .devcontainer: hook ran, cwd again the workspace's source folder.

devpod chdirs into the workspace's recorded local source before running the hook, and it does so identically whether the workspace is addressed by path or by id. The directory dl is launched from does not reach initializeCommand at all. Giving the devpod spawn an explicit working directory would change nothing about this failure.

3. I then reproduced the reported error verbatim — same three lines, same wording — by a different route: leave the source folder in place and its devcontainer.json intact, but remove the hook file itself. The failure is "the folder devpod resolved does not contain that file", not "devpod was standing in the wrong folder".

Two things the ticket records as established that are not

• recreate/restart/reset do not pass a clone path. When the workspace already exists, the spec handed to workspace_up is the bare workspace id and the custom id is dropped — the clone-path branch is the else, taken only when the workspace does not yet exist. So the proposed change, implemented literally, would hand subprocess a working directory named after a workspace id, which never exists as a directory — on precisely the three subcommands the ticket targets. The "missing directory" case would not be an edge case, it would be the only case.
• That same branch skips the clone manager entirely. Nothing on the existing-workspace path ensures the clone is present or populated before devpod is asked to rebuild from it. Given finding 3, this is where I would look next: the failure is consistent with devpod's recorded source folder for the existing workspace no longer holding what the repo holds, and dl never checks.

Cheapest next move, on the host

Ask devpod which folder it is actually using for the existing workspace, then look in it:

• read the recorded local source folder for the blooop/devlaunch default-branch workspace out of devpod list --output json
• check whether that folder has .devcontainer/claude-code/init-host.sh, and whether it is the same folder a fresh workspace gets

If it is a stale or foreign folder, this is a "dl never re-verifies the clone on the existing-workspace path" bug and needs a different ticket. If the folder is correct and the file is there, then the cause is something neither this ticket nor #87 has named yet and the map should reopen.

handoff

• Where we are: build stage stopped deliberately at the confirm gate. Hypothesis falsified by direct observation; no implementation attempted; working tree clean.
• Open thread: why devpod's source folder for the existing workspace lacks the hook file, when a fresh workspace's does not. Needs one host-side look, since no session here can see that machine's devpod records.
• First move on resume: run the two host checks above, then re-scope. Do not implement the cwd change — it is inert for this failure and actively wrong on the existing-workspace path.

[blooop]

breadcrumb: build stage stopped at the confirm gate — hypothesis disproven, no code written. Correcting my own ticket: I wrote it from a checkout of the stale slow_install branch, not origin/main. Two 'established' facts in the body are wrong on main:

• dl.py:1367-1370: when the workspace already exists, workspace_spec = workspace_id (a bare id) and custom_id = None. recreate/restart/reset therefore pass no clone path. cwd=workspace_spec would have handed subprocess a directory named after a workspace id — nonexistent on exactly the three subcommands the ticket targeted.
• That branch never calls clone_mgr.ensure_workspace(), so nothing on the existing-workspace path verifies the clone is present or populated before devpod is asked to rebuild from it.

The proposed fix is also inert, shown by experiment rather than argument: devpod v0.26.1 was run for real against a throwaway project with a bare relative initializeCommand (no Docker needed — the hook runs before any container). devpod chdirs into the workspace's recorded local source before running the hook, identically whether addressed by path or by id. dl's cwd never reaches initializeCommand. The human's exact three-line error was reproduced by a different mechanism: source folder present, devcontainer.json intact, hook file absent from that folder.

Parked pending one host check that settles it — devpod list --output json for the recorded source.localFolder of the devlaunch-main workspace, then whether that folder actually holds .devcontainer/claude-code/init-host.sh. If stale or foreign, this is a 'dl never re-verifies the clone on the fast path' bug and needs a re-scoped ticket.

[blooop]

Datum from #101 — there is a second class of orphan, and this ticket's reconciler is the natural owner.

Alongside the orphaned devpod workspace records this ticket already covers, the same
lifecycle leaks Docker named volumes, and it leaks them by design rather than by a
migration accident.

Source. devpod delete → pkg/driver/docker/docker.go:137 DeleteDevContainer →
d.Docker.Remove(ctx, container.ID) → pkg/docker/helper.go:137:

func (r *DockerHelper) Remove(ctx context.Context, id string) error {
	out, err := r.buildCmd(ctx, "rm", id).CombinedOutput()

Plain docker rm, not docker rm -v. A DeleteVolume helper exists (helper.go:98) and
GitHub code search over loft-sh/devpod returns exactly one hit for it — its own definition,
zero callers. So devpod never removes a volume. Docker never garbage-collects a named
volume regardless. dl adds nothing: purge_all_data (dl.py:321) is devpod delete --force
plus shutil.rmtree of the cache.

Every devcontainer here carries at least one named volume —
source=${localWorkspaceFolderBasename}-pixi in .devcontainer/devcontainer.json, plus
dind-var-lib-docker-<devcontainerId> once the docker-in-docker feature is on. Both outlive
their workspace permanently.

On this host today, whose devpod list is a clean three workspaces:

Local Volumes   14 total   2 ACTIVE   13.11 GB   92% reclaimable

Only devlaunch-main-zovomobo-pixi and devlaunch-t1-vebilote-pixi show LINKS 1. Twelve show
LINKS 0: main-pixi 2.705 GB, w1-pixi 1.947 GB, w2-pixi 1.877 GB, vscode 821 MB,
bugfix1-pixi 524 MB, aid-pixi 519 MB, slow_install-pixi 519 MB, aid_auto_2-pixi 518 MB,
t1-pixi 429 MB, dind-var-lib-docker-default-wf-9ef1b 279 kB (+2 unrelated compose volumes).
That is ~9.9 GB of dead volumes against 1.0 GB of live ones — 10:1 — and some date to July.

Suggested scope, matching this ticket's existing contract. The reconciler already promises
"if no healthy clone exists, say so and let the user drop the workspace — do not silently
delete". Same rule for volumes: for each *-pixi / dind-var-lib-docker-* volume with no live
container and no matching workspace, report it with its size and let the user run
docker volume rm. Reporting is cheap and safe; deleting Docker volumes on the user's behalf is
not something dl should start doing — especially since -pixi names derive from a directory
basename (${localWorkspaceFolderBasename}), so two repos with a main worktree share one
volume, which is already flagged as an open question on #93.

Note this leak is independent of the nested-daemon work — every one of the twelve orphans is
a plain -pixi volume from the existing non-dind devcontainer. dind adds a second named volume
per workspace to a leak that was already there.

[blooop]

breadcrumb: re-entering the build stage on the re-scoped ticket. Restating what I will build, and the seams I will test at.

What I am building

1. Persist the id. The record dl writes when it prepares a workspace clone carries the devpod workspace id dl is about to hand devpod as --id, instead of leaving that field declared and never written.
2. Follow the record when the derivation misses. A launch still derives an id and asks devpod about it first. When devpod does not answer to the derived id, the record for the same (owner, repo, branch) is consulted before the launch is treated as cold; a stored id devpod does recognise wins, and the launch attaches to it instead of cloning a second workspace. Every devpod-addressing subcommand (up, stop, rm, restart, recreate, reset, attach) is downstream of that one resolution, so they are fixed together.
3. Reconcile the orphans, as a command somebody runs, not as startup behaviour.

Three decisions the ticket asked me to make deliberately

• Surface: a top-level --reconcile, not prune --repair and not automatic. --prune states as its contract that it never touches a devpod workspace; repair is exactly a devpod-workspace write, so folding it in would retract that promise for every existing --prune user. Automatic-on-startup mutates another tool's records without asking, which this codebase does not do anywhere else.
• It writes devpod's own workspace record directly. devpod v0.26.1 exposes no command that changes an existing workspace's source (build, delete, list, logs, ssh, status, stop, up, plus config subcommands — none of them re-points a source). The only CLI-shaped route is a create, which needs a container daemon and would destroy the very record being repaired. The write is one field of one JSON file devpod itself keeps per workspace.
• Adoption is by path, never by id. The id is the thing that changed, so it cannot join the two records; the recorded source path still names owner and repo exactly. This is the same reasoning --prune's placement rules already run on.

Order of preference when the derivation moves

Derived-first-then-record rather than record-first, and it is a trade rather than a shortcut. Reading the record on every launch means loading metadata under its lock on the warm attach path, which #145 deliberately removed. Asking devpod about the derived id is already paid for by the existing status round trip, so the record is read only when that misses — the exact case this ticket is about. The two orders differ in observable behaviour only when devpod knows the derived id and a different stored one for one triple, which after this change requires a workspace dl did not create.

Seams I will test at

1. The record, after a clone is prepared — WorkspaceCloneManager → MetadataStorage. Pins: the stored devpod workspace id equals the id dl hands devpod, for a new clone and for a re-prepared existing one.
2. The id a devpod subcommand is addressed by — dl.main([...]) with devpod's spawn recorded. Pins: derived when they agree; the stored one once the derivation has moved under the record. The scheme change is simulated at the derivation, not written down as a literal suffix, so this test cannot rot into asserting today's ids.
3. The reconciler's plan and its effect — over a scratch devpod home and a scratch cache. Pins: orphan + healthy clone → adopted (devpod's source rewritten, the record's id filled in); orphan + no clone → named, exit code says so, nothing deleted; a second run finds nothing to do.

Boundaries I am declaring rather than deciding

• Docker named volumes (the What does the nested daemon cost in disk and time? #101 datum above) are not in this build. They are a leak in devpod's delete path, not a record disagreement, and nothing in this ticket's "Done when" covers them. Worth its own ticket.
• A refused rename still advances the schema header, stranding those records forever #180 (a refused rename advancing the schema header) is adjacent and is left alone. This build adds no schema version bump and no rename; it writes one previously-unwritten field that every existing reader already tolerates.

[blooop]

breadcrumb: build stage complete — PR #210 open, all checks green (5-version CI matrix, e2e, prek, gate, codecov patch and project). Not merged, not self-reviewed; a fresh-context review stage follows.

The slice, in one line: dl now writes down the devpod workspace id it creates and follows it when the derivation misses, and dl --reconcile re-points the workspaces the last derivation change already orphaned.

The three decisions the ticket left open, settled

• Surface: a top-level dl --reconcile, not prune --repair and not automatic. --prune promises never to touch a devpod workspace; repair is nothing but a devpod-workspace write, so folding it in would retract that promise for everyone already relying on it.
• Mechanism: devpod's own workspace record is rewritten directly, because devpod v0.26.1 exposes no subcommand that changes an existing workspace's source — the only route through the CLI is a create, which needs a container daemon and would destroy the record being repaired. One field, rewritten from what devpod itself last wrote, through a temp file and a rename.
• Join: by path, never by id. The id is what moved; the recorded source folder still names owner and repo exactly, and its last component still names the branch in one of the three ways dl has named a clone directory. Where the match is not unique it is refused, not guessed.

Ordering, and the cost it avoids

The record is read only after devpod has denied the derived id. Reading it first would put the metadata lock, the parse and the migration's version check back on the warm attach path that #145 deliberately cleared — the path a person waits on. The two orders differ only when devpod knows the derived id and a different recorded one for one triple, which now takes a workspace dl did not create.

How this sits with #155

--prune's Disputed arm is untouched and stays untouched. --prune still refuses to guess which clone a misplaced workspace needs; --reconcile resolves the disagreement, after which those clones classify as referenced in the ordinary way. The prune docs now point at it.

Boundaries declared rather than decided

• Docker named volumes (the What does the nested daemon cost in disk and time? #101 datum on this ticket) are not in the PR. That is a leak in devpod's own delete path, not a record disagreement, and nothing in this ticket's "Done when" covers it. It wants a ticket of its own.
• A refused rename still advances the schema header, stranding those records forever #180 is untouched: no schema bump, no rename, one previously-unwritten field every existing reader already tolerates.

What a session here still cannot verify

That a repaired workspace recreates end to end needs Docker. Everything claimed is from records on disk and devpod's own CLI surface; the final dl blooop/devlaunch recreate remains the human's, on the host — and it is worth doing, because that host is the one the ticket was written from.

[blooop]

wf-review verdict on #210: Request changes — two blocking findings, both on reconcile safety invariants (candidate-index collisions resolve last-write-wins; claimed guard uses by_path keys instead of holder() and skips the unlocatable check). Spec axis fully clean — all four Done-when bullets met. Fix round dispatched on the same branch.

[blooop]

Fix round ce08cce re-reviewed by a fresh-context agent: both blocking findings resolved (collision → contested/Unadoptable both directions; holder() containment + unlocatable refusal matching --prune). #210 merged. Outstanding manual step: end-to-end recreate of a repaired workspace needs Docker on the host with the real orphans — run dl --reconcile there and recreate one repaired workspace to close the loop.

[blooop]

Manual e2e step closed (run on the reporting host, real Docker, dl-next at main 67e2acf).

The real orphans first. dl --reconcile against the real cache named all 4 (devlaunch-main, kinisi-ros-main, kinisi-ros-ags-fix-branch-clone-instance-id, kinisi-ros-ags-devcontainer-tooling-support) and adopted none — correctly: ~/.cache/devlaunch/repos/blooop/devlaunch/ and .../kinisi-robotics/kinisi_ros/ are now empty, so there is no clone to re-point at. The 36-of-39 state the ticket was written from has since decayed further — the new-name checkouts are gone too. Nothing was deleted; a second run finds the same 4; exit 0 per the "nothing to consent to" arm. Disposition of those 4 records is a human call (dl <ws> rm), exactly as designed.

The repair→recreate loop, end to end. Since no real orphan is repairable, the loop was exercised on a workspace created for the purpose (scratch XDG_CACHE_HOME, real devpod, real Docker): cold dl blooop/devlaunch up → record written with devpod_workspace_id: devlaunch-main-zovomobo (the automatic half, confirmed live) → clone moved so devpod's source went dead while the record knew the new location → plan printed Re-pointing 1, piped n aborted cleanly → -y rewrote devpod's source.localFolder (verified in devpod list --output json) → dl blooop/devlaunch recreate rebuilt the container against the repaired record → docker inspect shows the bind mount .../repos/blooop/devlaunch/main -> /workspaces/devlaunch-main-zovomobo and a one-shot git rev-parse inside it answers. rm then refused on the container's own pixi.lock churn (the unsaved-work guard, working) and --force deleted it. Real state untouched throughout.

One finding filed as #224: running --reconcile from inside <root>/<owner>/<repo> misreports every git-URL-sourced workspace as that repo's orphan — Path(url).resolve() resolves scp-like URLs against cwd. All failure modes point toward refusal, never adoption or deletion.

This comment was generated by AI during review follow-through.