Repository navigation
Description
Where did this happen?
Desktop app (macOS)
Version
1.24.2 (git 2.50.1, Apple Git-155), Darwin arm64
What happened?
Deleting a workspace fails with:
Failed to verify worktree removal at ~/.superset/worktrees/<project-id>/<worktree>:
[host-worker] Task "git/removeWorktree" timed out after 120000ms
This is not a hang, and not #6730's swallowed-error case — the removal is genuinely
still running when the 120 s task timeout fires. gitWorktreeRemoveTask lets git do
the recursive delete, and git's remove_dir_recursively is single-threaded.
Numbers from one affected worktree (a pnpm/Turborepo JS monorepo):
| filesystem entries in the worktree | 161,521 |
…inside node_modules |
140,157 (87%) |
| directories | 17,972 |
workspaces on this machine carrying node_modules |
22 of 26 |
Stripping just the node_modules trees with a 4-way parallel rm -rf took 56 s
on this machine. Git's single-threaded walk over the same set, plus the remaining ~21k
entries, does not fit in 120 s.
Environment factors that make it much worse, and which are ordinary for a heavy
multi-workspace user: 10-core / 16 GB machine, ~26 workspaces open, load average ~18,
swap fully consumed (5.7 GB of 6 GB), disk 96% full. Cold metadata I/O is the
bottleneck — du -sh ~/.superset/worktrees did not complete in 10 minutes.
Ruled out as causes: stale .git lock files, hooks, submodules, core.fsmonitor,
nested repos, immutable flags. git worktree list --porcelain returns in 50 ms. Git
itself is healthy, and a synthetic worktree of 160k files in only 3.2k directories
removes in 19 s — directory count and cold cache are what push the real tree over.
Why raising the timeout isn't the fix
The timer is armed at dispatch, not enqueue — WorkerTaskRunner.drainQueue sets
task.timeoutHandle when a slot picks the task up — so worker-pool queueing is already
correctly excluded. The 120 s is real, single-threaded delete work.
Steps to reproduce
- Open a workspace on a large JS monorepo and install dependencies, so the worktree
carries ~150k files across many nestednode_modules. - Have enough other workspaces open that memory/IO are under pressure and the tree's
metadata is cold. - Delete the workspace.
- After 120 s, the toast above appears. The workspace correctly un-archives and stays
visible — but it cannot be deleted, and retrying hits the same wall.
Suggested fix
gitWorktreeRemoveTask doesn't need git to do the deleting. The task's own comment
already notes that --force --force also unregisters a worktree whose directory is
already gone. So:
rm -rfthe directory first with a parallel/native delete (orfs.rmwith
concurrency) — measurably 4–8× faster than git's serial walk on the same tree.- Then call
git worktree remove --force --forcepurely to unregister — near-instant. - Then verify with
git worktree list --porcelain, exactly as today.
Additionally:
- Scale the timeout with tree size, or emit progress, rather than a flat 120 s cap.
- Distinguish "still working" from "failed" in the toast. Today a timeout reads as a
hard failure, with no hint that the operation was progressing or that it would
succeed against a smaller tree.
Workaround
Delete node_modules before deleting the workspace. It is 87% of the inodes, and
removing it drops the tree to ~21k entries, after which the delete finishes in under a
second.
find <worktree> -maxdepth 4 -name node_modules -type d -prune -print0 \
| xargs -0 -P4 rm -rfRelated issues
- [bug] Workspace delete can silently orphan a worktree on disk when git unregisters it mid-failure #6730 — same command, different failure mode (git unregisters mid-failure and the
app trusts it). Both trace back togitWorktreeRemoveTaskletting git own the
recursive delete; doing therm -rfin-app first shrinks the surface for that bug too. - Feature request: auto-prune orphaned worktrees / storage cleanup command #5631 — auto-prune / storage cleanup. A "reclaim
node_modulesfrom idle
workspaces" action would make workspaces deletable again. Worth noting for that issue
that with pnpm the win is inodes, not bytes: everynode_modulesfile here is a
hardlink with a link count of 85, shared across all workspaces and a single 8.9 GB
content-addressable store, so stripping nine workspaces freed ~0 disk while cutting
each tree from 161k to 21k entries.duover the worktree root double-counts those
hardlinks badly. It's the metadata volume that breaks delete, not the disk usage. - Superset creates ~19 short-lived processes every 2.5 seconds, scaling with open worktree count #6717 — process/scaling cost with open worktree count; same "many workspaces on
one machine" pressure.
Activity
github-actions commentedon Aug 26, 2026
Summary
The report is that deleting a workspace whose worktree carries a large
node_modulestree fails withFailed to verify worktree removal at …: [host-worker] Task "git/removeWorktree" timed out after 120000ms, and that the timeout is not a hang — the recursive delete is genuinely still running when the flat 120 s budget expires, becausegitWorktreeRemoveTaskdelegates the whole recursive delete togit worktree remove, whose directory walk is single-threaded. The analysis in the issue matches what the code does, including the claim about where the timer is armed. I could not measure the timings, but every structural claim checks out against the source.Affected code
packages/host-service/src/workers/tasks/git.ts:198-236—gitWorktreeRemoveTask. The only deleting step isgit worktree remove --force --force <target>(git.ts:223-225); nothing in the app touches the filesystem first. The handler already reports phases (resolve-path,worktree-remove,worktree-list).packages/host-service/src/trpc/router/workspace-cleanup/git-ops.ts:60-71— the flattimeoutMs: 120_000, with a comment that already anticipates exactly this case ("can take a while for large trees (node_modules etc.)").packages/host-service/src/workers/WorkerTaskRunner.ts:316-320— the timeout handle is set indrainQueueat the moment a slot dispatches the task, so pool queueing is excluded from the budget. The reporter's claim here is correct; raising or scaling the number is not masking a queueing problem.packages/host-service/src/trpc/router/workspace-cleanup/workspace-cleanup.ts:512-540—stillRegistereddefaults totrueand any task failure becomes anINTERNAL_SERVER_ERROR(Failed to verify worktree removal at …), which un-archives the row. That is why the workspace correctly reappears and stays retryable.Suspected root cause
A flat, non-extendable 120 s wall-clock budget over work whose cost is proportional to inode and directory count, done by a process the app chose not to parallelize.
git worktree removedeletes the tree serially; the task hands git the entire job and then measures it against a constant.Two consequences that aren't in the report and that matter for triage:
handleTaskTimeoutretires the slot (WorkerTaskRunner.ts:500-509→shutdownSlot), the worker runskillAndReapTrackedChildren(host-worker.ts:44-51), and that SIGKILLs the in-flightgitchild (worker-child-tracker.ts:45-73). So each attempt destroys part of the tree and then throws. Retries should be making progress against a shrinking tree even though the toast says "failed" — worth confirming with the reporter, since "retrying hits the same wall" is consistent either way (partial progress per attempt smaller than what remains) and distinguishes "budget too small" from "no progress at all".mainalways appendsin phase "…"(WorkerTaskRunner.ts:492, pinned bypackages/host-service/test/workspace-cleanup.test.ts:571-603), and the handler labels itself from its first statement. A 1.24.2 build predating phase reporting explains the bare message; on a current build the same failure should read… in phase "worktree-remove", which would positively pin the delete rather than theworktree listread. Ask for that if the reporter can reproduce on a newer build.Related, same class, not the failure reported here: the non-
forcepath runs agit statuspreflight over the same cold tree under a 15 s budget (git-ops.ts:46-58), which on a 160k-file tree would fail closed first.Reproduction
Real repro needs the size, not the specific repo:
node_modulesinstall, or a synthetic generator — the reporter notes directory count matters more than file count, and a flat 160k/3.2k-dir tree removed in 19 s for them).purgeon macOS, or reboot) so the walk is cold.force: true.For a deterministic unit-level repro of the failure surface, the harness already exists —
packages/host-service/test/workspace-cleanup.test.ts:571injects aWorkerTaskErrorviamakeCtx({ removeWorktree: … }). For the budget itself, abun:testagainstWorkerTaskRunnerwith a task that sleeps pasttimeoutMsasserts the timer-at-dispatch behaviour and the resultingshutdownSlot; that's the honest scope of what's testable without a 150k-file fixture.Fix direction
The suggested inversion is sound and has precedent in this file: the missing-repo branch already does
rm(path, { recursive: true, force: true })directly (workspace-cleanup.ts:483-494) under theisInsideProjectWorktreesRootguard (packages/host-service/src/trpc/router/workspace-creation/shared/worktree-paths.ts:49). So:gitWorktreeRemoveTask,fs.rmthe directory first, then callgit worktree remove --force --forcepurely to unregister (the comment atgit.ts:217-221already establishes that it handles an absent directory), then keep theworktree list --porcelainverify unchanged. Node's recursivermfans out withPromise.allper directory, which is where the 4–8× over git's serial walk comes from.{ repoPath, worktreePath, gitEnv }(git.ts:199). An in-apprm -rfneedsprojectId+worktreeBaseDirpassed in so it can apply the same guard asworkspace-cleanup.ts:471-482— otherwise a corruptworktreePathpoints an unguarded recursive delete at user data, which is precisely the risk those two existing branches were written to avoid. Don't land step 1 without this.reportPhasealready exists end-to-end, the sturdier change is a heartbeat: have the delete report progress (entries removed) and have the runner reset the timer on progress, failing only on genuine stall. Scaling the cap by pre-counted tree size is simpler but re-walks the cold tree to get the count, which is itself the expensive operation.One caveat on the mechanics:
fs.rm's fan-out runs on the libuv threadpool, which is shared process-wide (default 4 threads) and not per worker thread. A 150k-entry delete will monopolize it and can stall other host-service filesystem I/O for the duration — worth bounding the concurrency rather than letting it fan out unbounded, especially given this reporter's "26 workspaces on one machine" scenario is the one that triggers the bug in the first place.fix(host-service): delete worktree trees in the app, let git only unr…
fix(host-service): delete worktree trees in the app, let git only unr…
tsavo-at-pieces commentedon Nov 6, 2026
Confirmed the same120s failure with a Rust target-heavy worktree during an independently preserved cleanup on macOS. CLI is2.0.5; inspected host-service source is localfork commitf8d0a3131def87cfd525ece3ace6867540011610 (not a claim that every loaded module has that version).
Source:
packages/host-service/src/trpc/router/workspace-cleanup/git-ops.ts:60dispatchesgitWorktreeRemoveTaskwith timeoutMs120_000.packages/host-service/src/workers/tasks/git.ts:354performsgit worktree remove --force --force.A stock
workspaceCleanup.destroywith deleteBranch=false, force=true, skipTeardown=true returnedHTTP500 after about126s. While that same dispatch was pending, its git child was observed actively traversing target/debug/deps; it was doing deletion work. After failure the worker/child was absent, the workspace row was active again, its branch unchanged at754898429f90c45f26d4dd1f5e933ee2e61720e4, and the folder partially removed. Native sessions and children had already been verified gone; source/index/working edits and private run state were preserved first.This extends the existing report beyond node_modules. Suggested acceptance remains: large-tree progress survives the timeout strategy; cancellation/failure settles child ownership; reconciliation distinguishes registration from remaining disk; selected branches and unrelated live owners remain safe. No new product test or repair was run here. Manual recovery is being verified separately, without replaying terminal closes.
Private receipt is retained in the rem cleanup audit under
parallel-audit-b/3be93abc-26a2-406b-8e6f-07806ca11cbe-final-engine-cleanup.json; public cleanup index: https://github.com/open-runtime/runtime_embeddable_memory_cli/pull/2167 .