Skip to content

[bug] Workspace delete times out at 120s — git's single-threaded recursive delete can't finish a node_modules-heavy worktree #6887

Open

Description

@gemueselasagne

Where did this happen?

Desktop app (macOS)

Version

1.24.2 (git 2.50.1, Apple Git-155), Darwin arm64

What happened?

Deleting a workspace fails with:

Failed to verify worktree removal at ~/.superset/worktrees/<project-id>/<worktree>:
[host-worker] Task "git/removeWorktree" timed out after 120000ms

This is not a hang, and not #6730's swallowed-error case — the removal is genuinely
still running when the 120 s task timeout fires. gitWorktreeRemoveTask lets git do
the recursive delete, and git's remove_dir_recursively is single-threaded.

Numbers from one affected worktree (a pnpm/Turborepo JS monorepo):

filesystem entries in the worktree 161,521
…inside node_modules 140,157 (87%)
directories 17,972
workspaces on this machine carrying node_modules 22 of 26

Stripping just the node_modules trees with a 4-way parallel rm -rf took 56 s
on this machine. Git's single-threaded walk over the same set, plus the remaining ~21k
entries, does not fit in 120 s.

Environment factors that make it much worse, and which are ordinary for a heavy
multi-workspace user: 10-core / 16 GB machine, ~26 workspaces open, load average ~18,
swap fully consumed (5.7 GB of 6 GB), disk 96% full. Cold metadata I/O is the
bottleneck — du -sh ~/.superset/worktrees did not complete in 10 minutes.

Ruled out as causes: stale .git lock files, hooks, submodules, core.fsmonitor,
nested repos, immutable flags. git worktree list --porcelain returns in 50 ms. Git
itself is healthy, and a synthetic worktree of 160k files in only 3.2k directories
removes in 19 s — directory count and cold cache are what push the real tree over.

Why raising the timeout isn't the fix

The timer is armed at dispatch, not enqueue — WorkerTaskRunner.drainQueue sets
task.timeoutHandle when a slot picks the task up — so worker-pool queueing is already
correctly excluded. The 120 s is real, single-threaded delete work.

Steps to reproduce

  1. Open a workspace on a large JS monorepo and install dependencies, so the worktree
    carries ~150k files across many nested node_modules.
  2. Have enough other workspaces open that memory/IO are under pressure and the tree's
    metadata is cold.
  3. Delete the workspace.
  4. After 120 s, the toast above appears. The workspace correctly un-archives and stays
    visible — but it cannot be deleted, and retrying hits the same wall.

Suggested fix

gitWorktreeRemoveTask doesn't need git to do the deleting. The task's own comment
already notes that --force --force also unregisters a worktree whose directory is
already gone
. So:

  1. rm -rf the directory first with a parallel/native delete (or fs.rm with
    concurrency) — measurably 4–8× faster than git's serial walk on the same tree.
  2. Then call git worktree remove --force --force purely to unregister — near-instant.
  3. Then verify with git worktree list --porcelain, exactly as today.

Additionally:

  • Scale the timeout with tree size, or emit progress, rather than a flat 120 s cap.
  • Distinguish "still working" from "failed" in the toast. Today a timeout reads as a
    hard failure, with no hint that the operation was progressing or that it would
    succeed against a smaller tree.

Workaround

Delete node_modules before deleting the workspace. It is 87% of the inodes, and
removing it drops the tree to ~21k entries, after which the delete finishes in under a
second.

find <worktree> -maxdepth 4 -name node_modules -type d -prune -print0 \
  | xargs -0 -P4 rm -rf

Related issues

Activity

  1. github-actions commented on Aug 26, 2026

    @github-actions
    Contributor

    Summary

    The report is that deleting a workspace whose worktree carries a large node_modules tree fails with Failed to verify worktree removal at …: [host-worker] Task "git/removeWorktree" timed out after 120000ms, and that the timeout is not a hang — the recursive delete is genuinely still running when the flat 120 s budget expires, because gitWorktreeRemoveTask delegates the whole recursive delete to git worktree remove, whose directory walk is single-threaded. The analysis in the issue matches what the code does, including the claim about where the timer is armed. I could not measure the timings, but every structural claim checks out against the source.

    Affected code

    • packages/host-service/src/workers/tasks/git.ts:198-236 — gitWorktreeRemoveTask. The only deleting step is git worktree remove --force --force <target> (git.ts:223-225); nothing in the app touches the filesystem first. The handler already reports phases (resolve-path, worktree-remove, worktree-list).
    • packages/host-service/src/trpc/router/workspace-cleanup/git-ops.ts:60-71 — the flat timeoutMs: 120_000, with a comment that already anticipates exactly this case ("can take a while for large trees (node_modules etc.)").
    • packages/host-service/src/workers/WorkerTaskRunner.ts:316-320 — the timeout handle is set in drainQueue at the moment a slot dispatches the task, so pool queueing is excluded from the budget. The reporter's claim here is correct; raising or scaling the number is not masking a queueing problem.
    • packages/host-service/src/trpc/router/workspace-cleanup/workspace-cleanup.ts:512-540 — stillRegistered defaults to true and any task failure becomes an INTERNAL_SERVER_ERROR (Failed to verify worktree removal at …), which un-archives the row. That is why the workspace correctly reappears and stays retryable.

    Suspected root cause

    A flat, non-extendable 120 s wall-clock budget over work whose cost is proportional to inode and directory count, done by a process the app chose not to parallelize. git worktree remove deletes the tree serially; the task hands git the entire job and then measures it against a constant.

    Two consequences that aren't in the report and that matter for triage:

    1. The timeout leaves a half-deleted worktree. handleTaskTimeout retires the slot (WorkerTaskRunner.ts:500-509 → shutdownSlot), the worker runs killAndReapTrackedChildren (host-worker.ts:44-51), and that SIGKILLs the in-flight git child (worker-child-tracker.ts:45-73). So each attempt destroys part of the tree and then throws. Retries should be making progress against a shrinking tree even though the toast says "failed" — worth confirming with the reporter, since "retrying hits the same wall" is consistent either way (partial progress per attempt smaller than what remains) and distinguishes "budget too small" from "no progress at all".
    2. The reported message has no phase suffix. Current main always appends in phase "…" (WorkerTaskRunner.ts:492, pinned by packages/host-service/test/workspace-cleanup.test.ts:571-603), and the handler labels itself from its first statement. A 1.24.2 build predating phase reporting explains the bare message; on a current build the same failure should read … in phase "worktree-remove", which would positively pin the delete rather than the worktree list read. Ask for that if the reporter can reproduce on a newer build.

    Related, same class, not the failure reported here: the non-force path runs a git status preflight over the same cold tree under a 15 s budget (git-ops.ts:46-58), which on a 160k-file tree would fail closed first.

    Reproduction

    Real repro needs the size, not the specific repo:

    1. Create a worktree and populate it to ~150k entries across many nested directories (a node_modules install, or a synthetic generator — the reporter notes directory count matters more than file count, and a flat 160k/3.2k-dir tree removed in 19 s for them).
    2. Drop the metadata cache (purge on macOS, or reboot) so the walk is cold.
    3. Delete the workspace with force: true.

    For a deterministic unit-level repro of the failure surface, the harness already exists — packages/host-service/test/workspace-cleanup.test.ts:571 injects a WorkerTaskError via makeCtx({ removeWorktree: … }). For the budget itself, a bun:test against WorkerTaskRunner with a task that sleeps past timeoutMs asserts the timer-at-dispatch behaviour and the resulting shutdownSlot; that's the honest scope of what's testable without a 150k-file fixture.

    Fix direction

    The suggested inversion is sound and has precedent in this file: the missing-repo branch already does rm(path, { recursive: true, force: true }) directly (workspace-cleanup.ts:483-494) under the isInsideProjectWorktreesRoot guard (packages/host-service/src/trpc/router/workspace-creation/shared/worktree-paths.ts:49). So:

    1. In gitWorktreeRemoveTask, fs.rm the directory first, then call git worktree remove --force --force purely to unregister (the comment at git.ts:217-221 already establishes that it handles an absent directory), then keep the worktree list --porcelain verify unchanged. Node's recursive rm fans out with Promise.all per directory, which is where the 4–8× over git's serial walk comes from.
    2. The root guard must be plumbed. The task payload today is only { repoPath, worktreePath, gitEnv } (git.ts:199). An in-app rm -rf needs projectId + worktreeBaseDir passed in so it can apply the same guard as workspace-cleanup.ts:471-482 — otherwise a corrupt worktreePath points an unguarded recursive delete at user data, which is precisely the risk those two existing branches were written to avoid. Don't land step 1 without this.
    3. A faster delete still needs a budget that isn't a constant. On a swap-thrashing, 96%-full disk even a parallel delete can exceed any fixed cap. Since reportPhase already exists end-to-end, the sturdier change is a heartbeat: have the delete report progress (entries removed) and have the runner reset the timer on progress, failing only on genuine stall. Scaling the cap by pre-counted tree size is simpler but re-walks the cold tree to get the count, which is itself the expensive operation.
    4. UX: after a timeout the tree is partially removed and the operation was progressing, so the toast should say so and offer retry rather than reading as a hard failure. Cheap and independent of the rest.

    One caveat on the mechanics: fs.rm's fan-out runs on the libuv threadpool, which is shared process-wide (default 4 threads) and not per worker thread. A 150k-entry delete will monopolize it and can stall other host-service filesystem I/O for the duration — worth bounding the concurrency rather than letting it fan out unbounded, especially given this reporter's "26 workspaces on one machine" scenario is the one that triggers the bug in the first place.

  2. added 2 commits that reference this issue on Oct 7, 2026
    0cbbb5e
    ec61910
  3. tsavo-at-pieces commented on Nov 6, 2026

    @tsavo-at-pieces
    Contributor

    Confirmed the same120s failure with a Rust target-heavy worktree during an independently preserved cleanup on macOS. CLI is2.0.5; inspected host-service source is localfork commitf8d0a3131def87cfd525ece3ace6867540011610 (not a claim that every loaded module has that version).

    Source: packages/host-service/src/trpc/router/workspace-cleanup/git-ops.ts:60 dispatches gitWorktreeRemoveTask with timeoutMs120_000. packages/host-service/src/workers/tasks/git.ts:354 performs git worktree remove --force --force.

    A stock workspaceCleanup.destroy with deleteBranch=false, force=true, skipTeardown=true returnedHTTP500 after about126s. While that same dispatch was pending, its git child was observed actively traversing target/debug/deps; it was doing deletion work. After failure the worker/child was absent, the workspace row was active again, its branch unchanged at754898429f90c45f26d4dd1f5e933ee2e61720e4, and the folder partially removed. Native sessions and children had already been verified gone; source/index/working edits and private run state were preserved first.

    This extends the existing report beyond node_modules. Suggested acceptance remains: large-tree progress survives the timeout strategy; cancellation/failure settles child ownership; reconciliation distinguishes registration from remaining disk; selected branches and unrelated live owners remain safe. No new product test or repair was run here. Manual recovery is being verified separately, without replaying terminal closes.

    Private receipt is retained in the rem cleanup audit under parallel-audit-b/3be93abc-26a2-406b-8e6f-07806ca11cbe-final-engine-cleanup.json; public cleanup index: https://github.com/open-runtime/runtime_embeddable_memory_cli/pull/2167 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions