Skip to content

perf: parallelize copy-ignored with rayon - #1721

Merged
max-sixty merged 5 commits into
mainfrom
feat/parallel-copy-ignored
Mar 26, 2026
Merged

max-sixty merged 5 commits into
mainfrom
feat/parallel-copy-ignored

Conversation

@worktrunk-bot

@worktrunk-bot worktrunk-bot commented Mar 25, 2026 •

Copy link
Copy Markdown
Collaborator

Parallelizes the copy-ignored directory walks with rayon for wt step copy-ignored.

The copy logic (copy_dir_recursive, remove_if_exists, create_symlink) is extracted from step_commands.rs into a new src/copy.rs library module, making the production code accessible to benchmarks.

Benchmarking thread scaling (1, 2, 4, 8 threads × deep/wide directory layouts) showed performance peaks at 2–4 threads and regresses at 8+ due to SSD I/O contention. copy_dir_recursive now creates a dedicated 4-thread pool rather than using the global pool (which is sized at 2x CPU cores for network I/O).

The benches/cow_copy.rs benchmark compares serial vs production-parallel copying across two directory shapes: deep (Rust target/ — narrow tree) and wide (many sibling directories).

Part of #1720

This was written by Claude Code on behalf of maximilian

Convert both the top-level entry copy loop and the recursive directory
walk (copy_dir_recursive_fallback) to use rayon's parallel iterators.
Each entry has a distinct destination path, so parallel copies are safe.

The global rayon thread pool (max 8 threads, configured in main.rs)
bounds concurrency naturally.

Part of #1720

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
max-sixty and others added 4 commits March 25, 2026 16:35
Extract copy_dir_recursive, remove_if_exists, and create_symlink from
step_commands.rs into a new src/copy.rs library module. This makes the
production copy code accessible to benchmarks.

Update cow_copy benchmark to measure the real production code (parallel)
against a serial baseline, with two directory layouts: deep (narrow tree
like Rust target/) and wide (many sibling directories).

Add test_copy_ignored_error_nested_file_parent_creation to cover the
create_dir_all error path in the top-level file copy loop.

Co-Authored-By: Claude <noreply@anthropic.com>
# Conflicts:
#	src/commands/step_commands.rs
Benchmarks show filesystem copy performance peaks at 2-4 threads and
regresses at 8+ due to SSD I/O contention. The global rayon pool (2x
CPU cores, tuned for network I/O) is too large for local filesystem
work.

copy_dir_recursive now creates a dedicated 4-thread pool instead of
using the global pool.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
@max-sixty
max-sixty merged commit a0f4617 into main Mar 26, 2026
23 checks passed
@max-sixty
max-sixty deleted the feat/parallel-copy-ignored branch March 26, 2026 01:18
max-sixty added a commit that referenced this pull request Apr 3, 2026
The value was empirically chosen via thread scaling benchmarks in #1721;
reference the issue instead of vague "benchmarked on APFS and ext4".

Co-Authored-By: Claude <noreply@anthropic.com>
max-sixty added a commit that referenced this pull request Apr 3, 2026
The #1721 benchmarks measured single-directory copies, not the shared
pool workload. Note the distinction while explaining why the value
still applies.

Co-Authored-By: Claude <noreply@anthropic.com>
max-sixty added a commit that referenced this pull request Apr 3, 2026
The hardcoded 4-thread cap was benchmarked for single-directory copies
(#1721) but the shared pool workload is different. Use rayon's default
(num cores) until we benchmark the shared pool specifically.

Co-Authored-By: Claude <noreply@anthropic.com>
max-sixty added a commit that referenced this pull request Apr 3, 2026
Replace per-call thread pool creation in `copy_dir_recursive` with a
shared static `COPY_POOL` (`LazyLock<ThreadPool>`), following the
established pattern used by `HEAVY_OPS_SEMAPHORE` and `LLM_SEMAPHORE`.

The EMFILE fix in #1864 used two coordinating pieces: a
`current_thread_index()` check in `copy_dir_recursive` to detect ambient
pool context, plus caller-side pool creation in `step_copy_ignored`. A
single shared pool is simpler — nested `install()` calls are no-ops in
rayon, so `copy_dir_recursive` works correctly both standalone and when
called from `in_copy_pool()`.

Also removes the hardcoded 4-thread cap (`MAX_COPY_THREADS`) in favor of
rayon's default (num cores). The original benchmarks (#1721) tuned
thread count for single-directory per-call pools, which is a different
workload than the shared pool. Left a TODO to benchmark the shared pool
specifically.

Follow-up to #1864.

> _This was written by Claude Code on behalf of @max-sixty_

---------

Co-authored-by: Claude <noreply@anthropic.com>
max-sixty added a commit that referenced this pull request Apr 4, 2026
#1901 removed the dedicated copy pool, so `copy_dir_recursive` started
using the global rayon pool (2× CPU cores, tuned for network I/O). This
caused ~1000% CPU on copy-heavy operations like `wt step copy-ignored`
with large `target/` directories.

Restores a 4-thread `LazyLock` pool internal to `copy_dir_recursive` —
no public API change needed. The pool is the same approach benchmarked
in #1721 (performance peaks at 2–4 threads for disk I/O).

> _This was written by Claude Code on behalf of @max-sixty_

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants