Skip to content

[bug] Workspace delete can silently orphan a worktree on disk when git unregisters it mid-failure #6730

Closed

Description

@AviPeltz

Where did this happen?

Desktop app (macOS)

Version

1.24.0-canary.20260821030527 (git 2.50.1, Apple Git-155)

What happened?

Deleting a workspace ("Separate session and workspace loaders", branch separate-session-workspac) failed with:

Failed to remove worktree at /Users/avipeltz/.superset/worktrees/<project-id>/thin-zone

Root-caused this to three stacked bugs in the workspace-delete saga, confirmed by reading the source and reproducing the underlying git behavior directly:

  1. git worktree remove --force --force can partially fail while still fully unregistering the worktree. Reproduced directly (git 2.50.1): make a nested file inside a worktree undeletable (chflags uchg), then run git worktree remove --force --force <path>. Git prints error: failed to delete '<path>': Operation not permitted and exits 255 — but immediately afterward, git worktree list --porcelain no longer lists it and the repo's .git/worktrees/<name> admin entry is gone entirely, even though the working directory (and the worktree's own .git file) are still sitting on disk. Unregistration and file deletion are not atomic in git's own implementation.

  2. The app doesn't account for that. In packages/host-service/src/workers/tasks/git.ts (gitWorktreeRemoveTask), the git worktree remove --force --force call's error is discarded (.catch(() => {})), and the task's return value (stillRegistered) is derived solely from re-running git worktree list --porcelain afterward:

    await git
        .raw(["worktree", "remove", "--force", "--force", target])
        .catch(() => {});
    const raw = await git.raw(["worktree", "list", "--porcelain"]);
    return {
        stillRegistered: parseWorktreeList(raw).some(
            (w) => normalizeWorktreePath(w.path) === target,
        ),
    };

    In packages/host-service/src/trpc/router/workspace-cleanup/workspace-cleanup.ts, when stillRegistered comes back false, the code sets worktreeRemoved = true and moves on — it never re-checks existsSync(local.worktreePath) before trusting that. (Two other branches in the very same function — the "session" branch and the "project repo is missing" branch — already do exactly that existsSync + rm(path, { recursive: true, force: true }) fallback; it just isn't wired into the common "project + repo present" path that actually ran here.)

    When stillRegistered does come back true (as it apparently did on the attempt that produced this toast), the code throws:

    throw new TRPCError({
        code: "INTERNAL_SERVER_ERROR",
        message: `Failed to remove worktree at ${local.worktreePath}`,
    });

    — with no detail, because the real git error from step 1 was already thrown away.

  3. Nothing logs the underlying git failure anywhere. Checked host-service.log, the terminal-host daemon.log, and the Electron main.log — none contain any trace of the actual git worktree remove error. The toast text is the entire diagnostic record. The softer warnings this same saga produces (e.g. "N terminal(s) may still be running" from a failed PTY kill) are also only returned as strings in a warnings array to the renderer — never logged server-side — so even those vanish once the toast is dismissed.

Net effect: the specific workspace above is now stuck — git has zero record of it, but its directory (a partial sparse-checkout: apps/api, apps/discord-triage, apps/marketing, apps/mobile, apps/relay, apps/relay2, plus root files) is still on disk, and its workspaces DB row is still un-archived. Clicking Delete again as-is will very likely report success (git already shows nothing registered) and archive the row, permanently leaving the orphaned folder behind — nothing targets an already-unregistered path for cleanup.

Most likely trigger (circumstantial, not provable after the fact because of bug #3): runDestroyPhases step 3a (workspace-cleanup.ts) disposes PTY sessions tied to the workspace before step 3b removes the worktree, but a failed kill only pushes a warning — it doesn't block or retry step 3b. disposeSessionsByWorkspaceId (terminal.ts) only touches sessions tracked in the terminalSessions table, so a detached grandchild process (e.g. a dev server or install step whose parent PTY died but which wasn't fully reaped) would survive invisibly. host-service.log shows repeated [workspace-fs/watch] FSEvents overflow bursts for this exact path in the surrounding window — consistent with something still actively writing into the directory while the recursive delete ran.

Steps to reproduce

Reliable repro of the underlying git-level bug (not the full app flow, which requires timing a race):

git init -q main-repo && cd main-repo
git config user.email a@b.com && git config user.name test
echo hi > file.txt && git add . && git commit -q -m init
git worktree add -q --no-checkout ../wt1 -b wtbranch1
cd ../wt1 && git checkout wtbranch1
mkdir -p sub && echo x > sub/lockedfile
chflags uchg sub/lockedfile   # simulate a file the recursive delete can't remove
cd ../main-repo

git worktree remove --force --force ../wt1
# -> error: failed to delete '.../wt1': Operation not permitted (exit 255)

git worktree list --porcelain   # wt1 is already gone from the list
ls .git/worktrees/               # admin dir for wt1 is gone entirely
ls ../wt1                        # directory (and its .git file) still on disk

In the app, this manifests whenever git worktree remove --force --force hits any obstruction partway through its recursive delete (locked/open file, permission issue, a live process still writing into the tree) during workspace delete.

Logs, screenshots, or recordings

Full write-up with timeline, exact file/line references for every claim above, the missing-existsSync-fallback comparison, a concrete fix checklist, and immediate remediation steps for the specific stuck workspace: see docs/issues/workspace-delete-orphaned-worktree.md on the debug-workspace-delete branch.

Suggested fixes (detailed in that doc):

  • Log the real git stderr in git.ts's gitWorktreeRemoveTask instead of .catch(() => {}).
  • After removeWorktree reports stillRegistered: false, still check existsSync(local.worktreePath) before trusting worktreeRemoved = true; fall back to the same rm(..., { recursive: true, force: true }) pattern the other two branches in workspace-cleanup.ts already use.
  • Make the terminal-disposal step actually gate (or retry) the worktree-removal step instead of degrading a failed kill to a no-op warning.
  • Add a regression test for "git reports not-registered after remove, but the directory still exists on disk" — the existing suite covers "still registered → throws" and "git open failure blocks delete," but not this case.

Related issues

Activity

  1. github-actions commented on Aug 21, 2026

    @github-actions
    Contributor

    Summary

    The report is that a workspace delete surfaced Failed to remove worktree at …, and that the underlying cause is that git worktree remove --force --force is not atomic: git can unregister the worktree from the repo's admin area and then fail partway through the recursive directory delete (exit 255, error: failed to delete '<path>': Operation not permitted), leaving the directory on disk while git worktree list --porcelain no longer mentions it. The destroy saga treats that porcelain listing as the sole authority for "removed," never re-checks the filesystem afterwards, and throws away git's actual stderr — so the orphaned directory is invisible to the app and the failure leaves no diagnostic anywhere. I read the relevant code and the claims check out against what's on main; below are the confirmations plus two consequences the report doesn't cover.

    Affected code

    • packages/host-service/src/workers/tasks/git.ts:198-225 — gitWorktreeRemoveTask
    • packages/host-service/src/trpc/router/workspace-cleanup/workspace-cleanup.ts:459-541 — the "project + repo present" branch of step 3b
    • packages/host-service/src/trpc/router/workspace-cleanup/git-ops.ts:60-71 — cleanupGitOps.removeWorktree
    • packages/host-service/src/terminal/terminal.ts:2247-2287 — disposeSessionsByWorkspaceId
    • packages/host-service/src/runtime/archived-workspace-reconcile.ts:16-70 — the startup reconciler
    • packages/host-service/test/workspace-cleanup.test.ts:43-74, 380-417 — the mocked removeWorktree surface and its coverage

    Root cause

    1. The error is discarded and the registry read is treated as authoritative. git.ts:213-215 does .raw(["worktree","remove","--force","--force",target]).catch(() => {}), then derives stillRegistered purely from worktree list --porcelain at git.ts:218-223. The comment at git.ts:208-212 states the intent explicitly — "the registry read below is authoritative, not the command's locale- and version-dependent exit text" — and that is precisely the invariant the reported git behavior violates. Unregistration and recursive deletion are separate steps in git's implementation, so stillRegistered: false means "git no longer tracks this path," not "the directory is gone."

    2. The consumer never re-checks the disk. In workspace-cleanup.ts, worktreeRemoved is computed at line 460 before the removal (!existsSync(local.worktreePath)), and then at line 539 unconditionally overwritten with worktreeRemoved = true; whenever stillRegistered === false. There is no existsSync between the removal and that assignment. The asymmetry the report flags is real: the session branch (:432-450) and the missing-project-repo branch (:460-494) both do existsSync → rm(path, { recursive: true, force: true }) under a root guard, and the integration test comment at workspace-cleanup.integration.test.ts:352 even calls out "--force --force plus the existsSync fallback" — but that fallback is not wired into the common path that actually runs here.

    3. Nothing is logged server-side. The throw at :534-537 carries only the path; git's stderr was dropped at git.ts:215. The warnings array (:412, :416, :455, :481, :507, :559, :569) is returned to the renderer and only rendered as toasts (useDestroyDialogState.ts:152, useBulkWorkspaceDelete.ts:194) — never written to a log. So both the hard failure and the soft warnings are gone once the toast is dismissed.

    Two consequences worth adding:

    • The retry does silently orphan, as predicted. On a second Delete against the already-unregistered path: existsSync is true → repo is present → removeWorktree runs, git fails with is not a working tree (swallowed at git.ts:215) → the listing shows nothing → stillRegistered: false → line 539 sets worktreeRemoved = true → success, row stays archived, directory stays on disk.
    • The reconciler can't recover it and will spin on it. selectStranded (archived-workspace-reconcile.ts:78-86) picks up exactly this shape — archived row, no live owner, directory still exists — and calls destroyWorkspace with force: true, which walks the same faulty path and reports success without deleting anything. That repeats every host-service start, logging resumed 1 interrupted delete(s) forever while the orphan persists. The safety net designed for this state is defeated by the same bug.

    Note on the suspected trigger (disposeSessionsByWorkspaceId letting a detached grandchild survive): plausible from the code — it only iterates terminalSessions rows (terminal.ts:2251-2260) and a failed close only increments failed, which becomes a warning at workspace-cleanup.ts:411-413 and does not gate step 3b. But it isn't the only way to obstruct the delete (an open file handle, a Finder/editor lock, an immutable flag would all do it), and bug #3 means there's no evidence either way for this particular incident. The disk-recheck defect stands independent of the trigger.

    Reproduction

    The git-level behavior in the report is a faithful, self-contained repro of the precondition. For the app-level defect, the cleanest reproduction is a unit test — and note that today's mocked default (workspace-cleanup.test.ts:68-71, { stillRegistered: false }) already runs against a tmp directory that still exists on disk, so the existing happy-path tests are unknowingly asserting the buggy behavior and will need their mocks updated to actually remove the directory:

    test("git unregisters the worktree but the directory survives → does not report removed", async () => {
      const tmp = mkdtempSync(join(tmpdir(), "workspace-delete-"));
      const repo = mkdtempSync(join(tmpdir(), "workspace-delete-repo-"));
      try {
        const ctx = makeCtx({
          workspace: { id: "ws-1", projectId: "p-1", worktreePath: tmp, branch: "feature" },
          project: { id: "p-1", repoPath: repo },
          // git's own partial-failure shape: unregistered, directory left behind.
          removeWorktree: async () => ({ stillRegistered: false }),
        });
        const caller = workspaceCleanupRouter.createCaller(ctx);
        const result = await caller.destroy({ workspaceId: "ws-1", deleteBranch: false, force: true });
        // Today: worktreeRemoved === true and `tmp` is still on disk.
        expect(existsSync(tmp)).toBe(false);
      } finally {
        rmSync(tmp, { recursive: true, force: true });
        rmSync(repo, { recursive: true, force: true });
      }
    });

    An end-to-end version is possible in the integration suite on macOS by chflags uchg-ing a nested file before destroy (with an after hook to chflags nouchg, or the temp dir leaks), though that's platform-specific.

    Suggested fix direction

    1. Stop discarding the git error (git.ts:213-215). Capture stderr and return it alongside the verdict — e.g. { stillRegistered, directoryExists, removeError } — and check existsSync(target) inside the task, right after git exits, rather than back on the event loop; that's both closer to the event and off the loop. Update the git.ts:208-212 comment, since the invariant it asserts ("the registry read is authoritative") is what's wrong here.
    2. Re-check the disk before trusting removal (workspace-cleanup.ts:539). Replace the bare worktreeRemoved = true with the same existsSync → rm(..., { recursive: true, force: true }) fallback the other two branches use, guarded by isInsideProjectWorktreesRoot so a corrupt worktreePath can never aim a recursive delete at user data. Include the captured removeError in the :534-537 throw message so the toast carries the real reason.
    3. Consider gating step 3b on terminal disposal — at minimum retry once when killed.failed > 0 before attempting the removal, since a live writer is the most likely obstruction.
    4. Log the warnings server-side in addition to returning them, so the soft-failure path leaves a trace.
    5. Regression tests for "unregistered but still on disk" (above) and for the reconciler not looping on an orphan it can't clear.

    For the specific stuck workspace, once (2) lands, re-running Delete will take the new fallback and clear both the directory and the row. Before then it can be unstuck manually by removing the leftover worktree directory (and clearing any chflags uchg / permission obstruction on files inside it) and then deleting the workspace from the app.

    Cross-references in the report to #4941, #2863, #5499, #5631, #6070 and #6380 are plausible from the code shape but I did not verify those issues' contents.

  2. added a commit that references this issue on Sep 4, 2026
    52f7d9e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions