Repository navigation
perf: parallelize copy-ignored with rayon - #1721
Merged
Merged
Conversation
Convert both the top-level entry copy loop and the recursive directory walk (copy_dir_recursive_fallback) to use rayon's parallel iterators. Each entry has a distinct destination path, so parallel copies are safe. The global rayon thread pool (max 8 threads, configured in main.rs) bounds concurrency naturally. Part of #1720 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Closed
Extract copy_dir_recursive, remove_if_exists, and create_symlink from step_commands.rs into a new src/copy.rs library module. This makes the production copy code accessible to benchmarks. Update cow_copy benchmark to measure the real production code (parallel) against a serial baseline, with two directory layouts: deep (narrow tree like Rust target/) and wide (many sibling directories). Add test_copy_ignored_error_nested_file_parent_creation to cover the create_dir_all error path in the top-level file copy loop. Co-Authored-By: Claude <noreply@anthropic.com>
# Conflicts: # src/commands/step_commands.rs
Benchmarks show filesystem copy performance peaks at 2-4 threads and regresses at 8+ due to SSD I/O contention. The global rayon pool (2x CPU cores, tuned for network I/O) is too large for local filesystem work. copy_dir_recursive now creates a dedicated 4-thread pool instead of using the global pool. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
max-sixty
added a commit
that referenced
this pull request
Apr 3, 2026
The value was empirically chosen via thread scaling benchmarks in #1721; reference the issue instead of vague "benchmarked on APFS and ext4". Co-Authored-By: Claude <noreply@anthropic.com>
max-sixty
added a commit
that referenced
this pull request
Apr 3, 2026
The #1721 benchmarks measured single-directory copies, not the shared pool workload. Note the distinction while explaining why the value still applies. Co-Authored-By: Claude <noreply@anthropic.com>
max-sixty
added a commit
that referenced
this pull request
Apr 3, 2026
The hardcoded 4-thread cap was benchmarked for single-directory copies (#1721) but the shared pool workload is different. Use rayon's default (num cores) until we benchmark the shared pool specifically. Co-Authored-By: Claude <noreply@anthropic.com>
max-sixty
added a commit
that referenced
this pull request
Apr 3, 2026
Replace per-call thread pool creation in `copy_dir_recursive` with a shared static `COPY_POOL` (`LazyLock<ThreadPool>`), following the established pattern used by `HEAVY_OPS_SEMAPHORE` and `LLM_SEMAPHORE`. The EMFILE fix in #1864 used two coordinating pieces: a `current_thread_index()` check in `copy_dir_recursive` to detect ambient pool context, plus caller-side pool creation in `step_copy_ignored`. A single shared pool is simpler — nested `install()` calls are no-ops in rayon, so `copy_dir_recursive` works correctly both standalone and when called from `in_copy_pool()`. Also removes the hardcoded 4-thread cap (`MAX_COPY_THREADS`) in favor of rayon's default (num cores). The original benchmarks (#1721) tuned thread count for single-directory per-call pools, which is a different workload than the shared pool. Left a TODO to benchmark the shared pool specifically. Follow-up to #1864. > _This was written by Claude Code on behalf of @max-sixty_ --------- Co-authored-by: Claude <noreply@anthropic.com>
max-sixty
added a commit
that referenced
this pull request
Apr 4, 2026
#1901 removed the dedicated copy pool, so `copy_dir_recursive` started using the global rayon pool (2× CPU cores, tuned for network I/O). This caused ~1000% CPU on copy-heavy operations like `wt step copy-ignored` with large `target/` directories. Restores a 4-thread `LazyLock` pool internal to `copy_dir_recursive` — no public API change needed. The pool is the same approach benchmarked in #1721 (performance peaks at 2–4 threads for disk I/O). > _This was written by Claude Code on behalf of @max-sixty_ Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parallelizes the copy-ignored directory walks with rayon for
wt step copy-ignored.The copy logic (
copy_dir_recursive,remove_if_exists,create_symlink) is extracted fromstep_commands.rsinto a newsrc/copy.rslibrary module, making the production code accessible to benchmarks.Benchmarking thread scaling (1, 2, 4, 8 threads × deep/wide directory layouts) showed performance peaks at 2–4 threads and regresses at 8+ due to SSD I/O contention.
copy_dir_recursivenow creates a dedicated 4-thread pool rather than using the global pool (which is sized at 2x CPU cores for network I/O).The
benches/cow_copy.rsbenchmark compares serial vs production-parallel copying across two directory shapes: deep (Rust target/ — narrow tree) and wide (many sibling directories).Part of #1720