Repository navigation
Fix/copy ignored too many open files - #1864
Conversation
Add a unit test that reproduces the conditions leading to EMFILE
("too many open files") when step_copy_ignored processes a large tree.
The test mirrors what step_copy_ignored does at runtime: it creates a
single 4-thread rayon pool, installs it, then uses par_iter to call
copy_dir_recursive concurrently for 20 independent source directories
(simulating a repo with node_modules/, target/, .cache/, etc. all
listed as gitignored entries to copy).
Before the fix in copy_dir_recursive, each concurrent call would create
its own 4-thread pool via ThreadPoolBuilder::build(). With N threads in
the outer pool and M directories, up to min(N,M) pools could exist
simultaneously, each spawning worker threads and consuming OS resources.
On systems with a low file-descriptor limit (macOS default: 256) this
reliably produces EMFILE.
The test asserts correctness (all 100 files reach the destination with
matching content), which also serves as the liveness check: the test
would hang or panic with EMFILE under the old behaviour.
wt step copy-ignored could fail with EMFILE ("too many open files") on
large gitignored trees, silently when run as a post-start hook via
wt switch --create, visibly when run directly as wt step copy-ignored.
Root cause
----------
step_copy_ignored iterated over the top-level ignored entries using
par_iter() on the global rayon thread pool (sized at ~2x CPU cores for
network I/O). For each directory entry, it called copy_dir_recursive(),
which unconditionally created a fresh 4-thread rayon pool via
ThreadPoolBuilder::build(). With several large top-level directories
(node_modules/, target/, .cache/, etc.), many pools were constructed
simultaneously — each spawning OS threads and consuming file-descriptor
resources on top of the concurrent file opens (two fds per
reflink/copy). On macOS the default soft fd limit is 256; this is easy
to exhaust.
Fix — two coordinated changes
------------------------------
1. copy_dir_recursive (src/copy.rs)
Check rayon::current_thread_index() before building a pool. When
the function is called from inside an existing rayon worker thread,
that index is Some(_), meaning a pool is already installed. In that
case, delegate directly to copy_dir_recursive_inner, which uses
par_iter() and naturally inherits the caller's pool through rayon's
work-stealing mechanism. When called from outside any pool (the
standalone / direct-invocation path), behaviour is unchanged: a
dedicated 4-thread pool is built and installed.
2. step_copy_ignored (src/commands/step_commands.rs)
Create one 4-thread pool before the outer loop and wrap the entire
par_iter in pool.install(). Because copy_dir_recursive now detects
it is already running on a rayon worker, all copy operations — the
outer loop over top-level entries and all recursive descents into
subdirectories — share this single pool. Total concurrency is capped
at MAX_COPY_THREADS (4) regardless of the number of top-level
entries, eliminating the concurrent pool explosion.
Behaviour preserved
-------------------
- Standalone invocations of copy_dir_recursive (e.g. the copy_and_remove
fallback path for cross-device renames) are unaffected: they still
build their own pool because current_thread_index() returns None.
- Parallelism within a single directory copy is unchanged: rayon
work-stealing distributes file copies across the 4 pool threads.
- All existing copy-ignored integration tests continue to pass.
cd5b2ef to
21e42d5
Compare
| // Mirror what step_copy_ignored does: one shared pool wrapping a par_iter | ||
| // that calls copy_dir_recursive for each directory entry. | ||
| let pool = rayon::ThreadPoolBuilder::new() | ||
| .num_threads(4) | ||
| .build() | ||
| .unwrap(); | ||
| pool.install(|| { | ||
| (0..DIR_COUNT) | ||
| .into_par_iter() | ||
| .try_for_each(|i| -> anyhow::Result<()> { | ||
| let src = src_root.path().join(format!("dir-{i}")); | ||
| let dst = dst_root.path().join(format!("dir-{i}")); | ||
| copy_dir_recursive(&src, &dst, false) | ||
| }) | ||
| }) | ||
| .unwrap(); |
There was a problem hiding this comment.
we need to test our actual code; not a duplicate of our code
@worktrunk-bot can you suggest some options?
There was a problem hiding this comment.
Agreed — the unit test currently duplicates the pool+par_iter orchestration from step_copy_ignored rather than exercising the real code. A few options:
Option 1: Integration test with many ignored directories (recommended)
Add an integration test alongside the existing step_copy_ignored tests that creates enough ignored directories to trigger the concurrent-pool scenario. This exercises the real step_copy_ignored → copy_dir_recursive path end-to-end:
#[rstest]
fn test_copy_ignored_many_directories(mut repo: TestRepo) {
let feature_path = repo.add_worktree("feature");
// Create enough ignored directories to trigger concurrent pool pressure
let mut gitignore_entries = String::new();
let mut worktreeinclude_entries = String::new();
for i in 0..200 {
let dir_name = format!("ignored-dir-{i}/");
let dir = repo.root_path().join(format!("ignored-dir-{i}"));
fs::create_dir_all(&dir).unwrap();
for j in 0..10 {
fs::write(dir.join(format!("file-{j}.txt")), format!("content {i}-{j}")).unwrap();
}
gitignore_entries.push_str(&dir_name);
gitignore_entries.push('\n');
worktreeinclude_entries.push_str(&dir_name);
worktreeinclude_entries.push('\n');
}
fs::write(repo.root_path().join(".gitignore"), &gitignore_entries).unwrap();
fs::write(repo.root_path().join(".worktreeinclude"), &worktreeinclude_entries).unwrap();
run_copy_ignored(&repo, &feature_path);
// Spot-check a sample of copied directories
for i in [0, 99, 199] {
let dst_file = feature_path.join(format!("ignored-dir-{i}/file-0.txt"));
assert!(dst_file.exists(), "ignored-dir-{i}/file-0.txt should be copied");
}
}This tests the actual code path and would have caught the original EMFILE bug.
Option 2: Minimal unit test for just the pool-reuse detection
If we also want a fast unit test for the copy_dir_recursive pool-reuse logic specifically (without the scale), keep it simple — just verify it works when called from within a rayon context:
#[test]
fn test_copy_dir_recursive_works_inside_rayon_pool() {
let src = TempDir::new().unwrap();
let dst = TempDir::new().unwrap();
fs::write(src.path().join("a.txt"), "hello").unwrap();
let pool = rayon::ThreadPoolBuilder::new().num_threads(2).build().unwrap();
pool.install(|| copy_dir_recursive(src.path(), &dst.path().join("out"), false)).unwrap();
assert_eq!(fs::read_to_string(dst.path().join("out/a.txt")).unwrap(), "hello");
}This is lightweight and tests copy_dir_recursive directly without duplicating the caller's orchestration.
My recommendation: Option 1 (integration test) is the most valuable since it exercises the real code path. Option 2 could supplement it as a small fast-feedback unit test, but Option 1 alone is sufficient.
|
@worktrunk-bot note the request above |
|
@worktrunk-bot do request #1 from above (unless @fspeirs objects / has other views) |
Move the pool-reuse regression test from a unit test that duplicated the pool+par_iter orchestration into an integration test that exercises the real step_copy_ignored code path with 200 ignored directories. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Done — replaced the unit test with an integration test that exercises the real |
for more information, see https://pre-commit.ci
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@max-sixty That's exactly what I thought about this morning! Happy with that. |
Replace per-call thread pool creation in `copy_dir_recursive` with a shared static `COPY_POOL` (`LazyLock<ThreadPool>`), following the established pattern used by `HEAVY_OPS_SEMAPHORE` and `LLM_SEMAPHORE`. The EMFILE fix in #1864 used two coordinating pieces: a `current_thread_index()` check in `copy_dir_recursive` to detect ambient pool context, plus caller-side pool creation in `step_copy_ignored`. A single shared pool is simpler — nested `install()` calls are no-ops in rayon, so `copy_dir_recursive` works correctly both standalone and when called from `in_copy_pool()`. Also removes the hardcoded 4-thread cap (`MAX_COPY_THREADS`) in favor of rayon's default (num cores). The original benchmarks (#1721) tuned thread count for single-directory per-call pools, which is a different workload than the shared pool. Left a TODO to benchmark the shared pool specifically. Follow-up to #1864. > _This was written by Claude Code on behalf of @max-sixty_ --------- Co-authored-by: Claude <noreply@anthropic.com>
The outer `.par_iter()` in `step_copy_ignored` ran on the global rayon pool (2× CPU cores), but each iteration called `copy_dir_recursive` which enters the 4-thread `COPY_POOL` via `install()` and blocks. This meant many global-pool threads sat idle waiting for just 4 copy workers — effectively serializing top-level entries rather than copying them in parallel. Adds `copy_pool_install()` to wrap the outer loop so all parallelism (top-level entries and recursive directory walks) shares the same 4-thread pool directly. This restores the pre-0.33 behavior of 4 concurrent copy threads across all paths, without the per-folder pool creation that caused EMFILE in #1864. > _This was written by Claude Code on behalf of @max-sixty_ Co-authored-by: Claude <noreply@anthropic.com>
This PR fixes a problem when copying large
.gitignoredtrees between worktrees.The underlying problem is that each iteration of
entries_to_copycreated its own Rayon thread pool each withMAX_COPY_THREADSwhich eventually exhausted the number of possible open files on my macOS 26.4 installation.I uncovered this when trying to create worktrees with very large gitignored subtrees.
The fix changes
step_copy_ignoredto create a 4-thread pool and havecopy_dir_recursivecheck if it's being called from a Rayon thread context and skip creating its own thread pool if that is the case.Closes #1865