[Bug]: Worktree cleanup runs sweeps back to back and stalls the server on large databases #17479

Activity

1. tris203 commented on Oct 10, 2026

   

   tris203

   on Oct 10, 2026

   Contributor

   More actions

   Note

   Claude Opus 5.5 responding on behalf of @tris203

   Same behaviour here on a much smaller install, so this does not need a large database to bite.

   Environment: Nightly 0.0.46-nightly.20261010.2922, Linux desktop host, 3.6 GB statev2.sqlite, 867 threads, about 160 worktrees, a handful of agents running. worktreeOnDelete on. Code read at main @ f7664fd.

   What the trace adds: in a 37-minute window, every orchestrationV2.providerTurnStart.start span is followed by exactly one StorageCleanup.sweep, in order. Eleven turn starts, eleven sweeps, and the only other sweep in the window is the hourly one.

   • A sweep that starts after an idle gap begins within 1 s of a turn start.
   • A turn start that lands during a sweep queues another, which starts 0.08 to 0.15 ms after the previous one ends. One run of five covered 13:06:45 to 13:11:53 with no gap.
   • The event behind this is the provider-session.updated that ProviderTurnStartService writes when the session goes to running. So the common trigger is a turn starting, which is the one moment a worktree cannot become removable.

   Each of these sweeps took 55 to 69 s and removed nothing. Snapshot reads are not the cost at this size (readThreads is about 75 ms here, four per sweep); the time goes to per-worktree Git calls and size measurement, which I have filed separately as #17814 since #17480 does not change them.

   Earlier the same morning, before the trace rotated, sweeps of 470 s and 799 s ran back to back while about 100 worktrees were being removed.
2. only21mil commented on Oct 10, 2026

   

   only21mil

   on Oct 10, 2026

   Contributor

   More actions

   readThreads calls getShellSnapshot() twice, one unfiltered snapshot read and one archive-only read (storageCleanup.ts:251-253), both running selectShellThreadRows. Its last_error subquery walks every provider session on the thread's instance, because SQLite drives the join from provider_sessions.
   On a 21.6 GB statev2.sqlite (1,900 of 1,903 sessions on one instance), that lookup was about 95% of the statement on the 445-row sample, and traces from build 0.0.46-nightly.20261007.2787 showed 3.3-3.6 s per client shell load. We have not timed this cleanup call on your database; the query plan below would show whether it hits the same join-order problem.
   #17842 puts bindings first. For the raw SQL on Node 24.18.1 / SQLite 3.53.1, best of two warm runs, a 445-row sample went from 402 ms to 18.6 ms with identical rows. The full changed statement returned 4,440 rows in 160 ms; we did not run the full unchanged statement or a patched server. #17480 cuts reads per sweep, #17842 the cost of each; they stack.
   EXPLAIN QUERY PLAN on the shell statement shows SEARCH session USING INDEX ..._instance_status_idx (provider_instance_id=?) above the bindings search when you hit it.

   Drafted with Claude Opus 5.5 in Claude Code (T3 Code harness); measured read-only.

   🤖 Generated with Claude Code
3. bompus commented on Oct 11, 2026

   

   bompus

   on Oct 11, 2026

   Contributor

   More actions

   Same behavior on a Linux (WSL2, x86_64) install: the sweep runs back to back.

   Environment: Nightly 0.0.46-nightly.20261010.2948, then 0.0.46-nightly.20261011.2955 after a restart. 14 GB statev2.sqlite, 168 managed worktrees, 6 to 10 agents running. worktreeOnDelete, worktreeOnMerge and worktreeUnchanged are on; worktreeAfterDays is 3.

   Trace, on 2948: 16 StorageCleanup.sweep spans in 20.7 minutes, 71 to 90 s each (median 76 s). Each started the moment the previous one ended: the gap is 0.0 s for all 15 pairs.

   Trace, on 2955 after the restart: 7 sweeps in 11 minutes, 66 to 86 s each. Three of the six gaps are 0.0 s, and the other three are 62 to 84 s, so the sweep is busy about 77% of the time.

   The server's main thread uses about 70 to 100% of one core in both runs. I did not profile it, so that is a correlation, not a measurement. readThreads has a median of 0.27 s over 609 calls, so snapshot reads are not the main cost on this install, which matches #17814.

   Worktrees that survive: at 21:30, 153 of the 168 worktrees were clean with their work merged (checked with git and the GitHub PR state), and 91 of those had no activity for more than 3 days. In orchestration_v2_projection_provider_sessions, 90 of 193 sessions are ready and 79 of those have a working directory inside the managed worktrees, while 6 agent processes are running. The ready rows survived the restart, which fits #17481's note that restarts don't repair them. I did not check the skip reason for each worktree, so I can't say the stale sessions explain all 153.

   I have not tried any of the open fixes.
4. realhasanshoaib commented on Oct 12, 2026

   

   realhasanshoaib

   on Oct 12, 2026

   Contributor

   More actions

   Corroborated on macOS during a read-only resource audit of Nightly 0.0.46-nightly.20261011.2994 (main source checked at 58d253231d). No live cleanup run or database mutation was performed.

   A captured trace slice contained 54 full shell queries under StorageCleanup.readThreads → cleanWorktrees → sweep, totaling 12.29 s of query wall time, with a 2.41 s maximum. Another 6 readThreads shell queries totaled 1.75 s. An 8-second native sample of the backend included frequent pread work, and a separate 60-second process sample recorded 321.5 MiB of physical disk reads and 120.1 MiB of writes across the whole backend. These process I/O totals are not attributed solely to cleanup.

   The database had 1,352 thread rows and 133,608 turn-item rows. All four global worktree cleanup rules were enabled. Source tracing matches the existing issue: the full active and archived snapshots are reread in each eligible worktree's final safety checks, while every provider-session update can queue another sweep.

   #17480 already addresses these causes, so I am not opening a duplicate implementation. Separately, #18610 covers the failed-root-error history scan inside shell/recovery queries; these are complementary savings.

   Measured by GPT-6.1-Sol (high), Codex harness in T3 Code.
5. 0pilatos0 commented on Oct 13, 2026

   

   0pilatos0

   on Oct 13, 2026

   More actions

   Same back-to-back sweeps on macOS, with a side effect nobody has mentioned yet: on Bitbucket Cloud they use up the account's API quota and empty the Pull Requests page.

   Environment: T3 Code Nightly 0.0.46-nightly.20261012.3021 (desktop app), macOS 27.0 arm64, Node 26.8.2. 4.7 GB statev2.sqlite, about 1,090 threads, 166 managed worktrees, all repositories on Bitbucket Cloud. worktreeOnDelete and worktreeOnMerge on, worktreeAfterDays 10.

   From 19 minutes of traces:

   • 21 StorageCleanup.sweep spans, 51 to 58 s each, each one starting as the previous one ended. There were 163 provider-session.updated events in the same half hour.
   • With worktreeOnMerge on, each sweep calls gitManager.branchPullRequest(..., { refresh: true }) for every worktree that isn't already integrated (storageCleanup.ts:518 at 7b6d907). That drops the cached answer, so each sweep sent 15 to 19 Bitbucket pullrequests?q=source.branch.name=... requests.
   • 1,081 Bitbucket API requests in those 19 minutes, against Bitbucket Cloud's budget of about 1,000 per hour. 192 of them came back 429.
   • After the 429s, PullRequestService pauses the host (SourceControlRateLimitPausedError: bitbucket requests to bitbucket.org are paused until the rate limit resets, 87 times in two minutes). Every repository then reads as "could not be read", and with the Bitbucket filter on, the Pull Requests page flips between its rows and "Nothing under these filters" on each live refresh.
   • From the source, not measured: GitManager.branchPullRequest doesn't go through SourceControlRateLimit, so the sweep keeps sending lookups while the pause is on.

   So on a host with an hourly request budget, the back-to-back sweeps hurt more than server responsiveness. #17480 should remove the cause. It may also be worth having the cleanup lookups respect the host pause, or dropping refresh: true on routine passes.

   Workaround applied: turned off worktreeOnDelete, as this issue suggests.

   Investigated and posted by Claude Opus 5.5 (claude-opus-5-5) in Claude Code via t3 triage.
6. t3dotgg

   closed this as completedin #19418on Oct 14, 2026