Skip to content

Fix/copy ignored too many open files - #1864

Merged
max-sixty merged 5 commits into
max-sixty:mainfrom
fspeirs:fix/copy-ignored-too-many-open-files
Apr 2, 2026
Merged

max-sixty merged 5 commits into
max-sixty:mainfrom
fspeirs:fix/copy-ignored-too-many-open-files

Conversation

@fspeirs

@fspeirs fspeirs commented Apr 1, 2026 •

Copy link
Copy Markdown
Contributor

This PR fixes a problem when copying large .gitignored trees between worktrees.

The underlying problem is that each iteration of entries_to_copy created its own Rayon thread pool each with MAX_COPY_THREADS which eventually exhausted the number of possible open files on my macOS 26.4 installation.

I uncovered this when trying to create worktrees with very large gitignored subtrees.

The fix changes step_copy_ignored to create a 4-thread pool and have copy_dir_recursive check if it's being called from a Rayon thread context and skip creating its own thread pool if that is the case.

Closes #1865

Add a unit test that reproduces the conditions leading to EMFILE
("too many open files") when step_copy_ignored processes a large tree.

The test mirrors what step_copy_ignored does at runtime: it creates a
single 4-thread rayon pool, installs it, then uses par_iter to call
copy_dir_recursive concurrently for 20 independent source directories
(simulating a repo with node_modules/, target/, .cache/, etc. all
listed as gitignored entries to copy).

Before the fix in copy_dir_recursive, each concurrent call would create
its own 4-thread pool via ThreadPoolBuilder::build(). With N threads in
the outer pool and M directories, up to min(N,M) pools could exist
simultaneously, each spawning worker threads and consuming OS resources.
On systems with a low file-descriptor limit (macOS default: 256) this
reliably produces EMFILE.

The test asserts correctness (all 100 files reach the destination with
matching content), which also serves as the liveness check: the test
would hang or panic with EMFILE under the old behaviour.
wt step copy-ignored could fail with EMFILE ("too many open files") on
large gitignored trees, silently when run as a post-start hook via
wt switch --create, visibly when run directly as wt step copy-ignored.

Root cause
----------
step_copy_ignored iterated over the top-level ignored entries using
par_iter() on the global rayon thread pool (sized at ~2x CPU cores for
network I/O). For each directory entry, it called copy_dir_recursive(),
which unconditionally created a fresh 4-thread rayon pool via
ThreadPoolBuilder::build(). With several large top-level directories
(node_modules/, target/, .cache/, etc.), many pools were constructed
simultaneously — each spawning OS threads and consuming file-descriptor
resources on top of the concurrent file opens (two fds per
reflink/copy). On macOS the default soft fd limit is 256; this is easy
to exhaust.

Fix — two coordinated changes
------------------------------
1. copy_dir_recursive (src/copy.rs)
   Check rayon::current_thread_index() before building a pool. When
   the function is called from inside an existing rayon worker thread,
   that index is Some(_), meaning a pool is already installed. In that
   case, delegate directly to copy_dir_recursive_inner, which uses
   par_iter() and naturally inherits the caller's pool through rayon's
   work-stealing mechanism. When called from outside any pool (the
   standalone / direct-invocation path), behaviour is unchanged: a
   dedicated 4-thread pool is built and installed.

2. step_copy_ignored (src/commands/step_commands.rs)
   Create one 4-thread pool before the outer loop and wrap the entire
   par_iter in pool.install(). Because copy_dir_recursive now detects
   it is already running on a rayon worker, all copy operations — the
   outer loop over top-level entries and all recursive descents into
   subdirectories — share this single pool. Total concurrency is capped
   at MAX_COPY_THREADS (4) regardless of the number of top-level
   entries, eliminating the concurrent pool explosion.

Behaviour preserved
-------------------
- Standalone invocations of copy_dir_recursive (e.g. the copy_and_remove
  fallback path for cross-device renames) are unaffected: they still
  build their own pool because current_thread_index() returns None.
- Parallelism within a single directory copy is unchanged: rayon
  work-stealing distributes file copies across the 4 pool threads.
- All existing copy-ignored integration tests continue to pass.
@fspeirs
fspeirs force-pushed the fix/copy-ignored-too-many-open-files branch from cd5b2ef to 21e42d5 Compare April 1, 2026 13:28
Comment thread src/copy.rs Outdated
Comment on lines +184 to +199
// Mirror what step_copy_ignored does: one shared pool wrapping a par_iter
// that calls copy_dir_recursive for each directory entry.
let pool = rayon::ThreadPoolBuilder::new()
.num_threads(4)
.build()
.unwrap();
pool.install(|| {
(0..DIR_COUNT)
.into_par_iter()
.try_for_each(|i| -> anyhow::Result<()> {
let src = src_root.path().join(format!("dir-{i}"));
let dst = dst_root.path().join(format!("dir-{i}"));
copy_dir_recursive(&src, &dst, false)
})
})
.unwrap();

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to test our actual code; not a duplicate of our code

@worktrunk-bot can you suggest some options?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — the unit test currently duplicates the pool+par_iter orchestration from step_copy_ignored rather than exercising the real code. A few options:

Option 1: Integration test with many ignored directories (recommended)

Add an integration test alongside the existing step_copy_ignored tests that creates enough ignored directories to trigger the concurrent-pool scenario. This exercises the real step_copy_ignored → copy_dir_recursive path end-to-end:

#[rstest]
fn test_copy_ignored_many_directories(mut repo: TestRepo) {
    let feature_path = repo.add_worktree("feature");

    // Create enough ignored directories to trigger concurrent pool pressure
    let mut gitignore_entries = String::new();
    let mut worktreeinclude_entries = String::new();
    for i in 0..200 {
        let dir_name = format!("ignored-dir-{i}/");
        let dir = repo.root_path().join(format!("ignored-dir-{i}"));
        fs::create_dir_all(&dir).unwrap();
        for j in 0..10 {
            fs::write(dir.join(format!("file-{j}.txt")), format!("content {i}-{j}")).unwrap();
        }
        gitignore_entries.push_str(&dir_name);
        gitignore_entries.push('\n');
        worktreeinclude_entries.push_str(&dir_name);
        worktreeinclude_entries.push('\n');
    }
    fs::write(repo.root_path().join(".gitignore"), &gitignore_entries).unwrap();
    fs::write(repo.root_path().join(".worktreeinclude"), &worktreeinclude_entries).unwrap();

    run_copy_ignored(&repo, &feature_path);

    // Spot-check a sample of copied directories
    for i in [0, 99, 199] {
        let dst_file = feature_path.join(format!("ignored-dir-{i}/file-0.txt"));
        assert!(dst_file.exists(), "ignored-dir-{i}/file-0.txt should be copied");
    }
}

This tests the actual code path and would have caught the original EMFILE bug.

Option 2: Minimal unit test for just the pool-reuse detection

If we also want a fast unit test for the copy_dir_recursive pool-reuse logic specifically (without the scale), keep it simple — just verify it works when called from within a rayon context:

#[test]
fn test_copy_dir_recursive_works_inside_rayon_pool() {
    let src = TempDir::new().unwrap();
    let dst = TempDir::new().unwrap();
    fs::write(src.path().join("a.txt"), "hello").unwrap();

    let pool = rayon::ThreadPoolBuilder::new().num_threads(2).build().unwrap();
    pool.install(|| copy_dir_recursive(src.path(), &dst.path().join("out"), false)).unwrap();

    assert_eq!(fs::read_to_string(dst.path().join("out/a.txt")).unwrap(), "hello");
}

This is lightweight and tests copy_dir_recursive directly without duplicating the caller's orchestration.

My recommendation: Option 1 (integration test) is the most valuable since it exercises the real code path. Option 2 could supplement it as a small fast-feedback unit test, but Option 1 alone is sufficient.

@max-sixty

Copy link
Copy Markdown
Owner

@worktrunk-bot note the request above

@max-sixty

Copy link
Copy Markdown
Owner

@worktrunk-bot do request #1 from above

(unless @fspeirs objects / has other views)

Move the pool-reuse regression test from a unit test that duplicated
the pool+par_iter orchestration into an integration test that exercises
the real step_copy_ignored code path with 200 ignored directories.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@worktrunk-bot

Copy link
Copy Markdown
Collaborator

Done — replaced the unit test with an integration test that exercises the real step_copy_ignored → copy_dir_recursive code path. The test creates 200 ignored directories (each with 10 files) and runs wt step copy-ignored, which would have caught the original EMFILE bug.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@fspeirs

fspeirs commented Apr 2, 2026

Copy link
Copy Markdown
Contributor Author

@max-sixty That's exactly what I thought about this morning! Happy with that.

@max-sixty
max-sixty merged commit 840e97e into max-sixty:main Apr 2, 2026
23 checks passed
max-sixty added a commit that referenced this pull request Apr 3, 2026
Replace per-call thread pool creation in `copy_dir_recursive` with a
shared static `COPY_POOL` (`LazyLock<ThreadPool>`), following the
established pattern used by `HEAVY_OPS_SEMAPHORE` and `LLM_SEMAPHORE`.

The EMFILE fix in #1864 used two coordinating pieces: a
`current_thread_index()` check in `copy_dir_recursive` to detect ambient
pool context, plus caller-side pool creation in `step_copy_ignored`. A
single shared pool is simpler — nested `install()` calls are no-ops in
rayon, so `copy_dir_recursive` works correctly both standalone and when
called from `in_copy_pool()`.

Also removes the hardcoded 4-thread cap (`MAX_COPY_THREADS`) in favor of
rayon's default (num cores). The original benchmarks (#1721) tuned
thread count for single-directory per-call pools, which is a different
workload than the shared pool. Left a TODO to benchmark the shared pool
specifically.

Follow-up to #1864.

> _This was written by Claude Code on behalf of @max-sixty_

---------

Co-authored-by: Claude <noreply@anthropic.com>
max-sixty added a commit that referenced this pull request Apr 5, 2026
The outer `.par_iter()` in `step_copy_ignored` ran on the global rayon
pool (2× CPU cores), but each iteration called `copy_dir_recursive`
which enters the 4-thread `COPY_POOL` via `install()` and blocks. This
meant many global-pool threads sat idle waiting for just 4 copy workers
— effectively serializing top-level entries rather than copying them in
parallel.

Adds `copy_pool_install()` to wrap the outer loop so all parallelism
(top-level entries and recursive directory walks) shares the same
4-thread pool directly. This restores the pre-0.33 behavior of 4
concurrent copy threads across all paths, without the per-folder pool
creation that caused EMFILE in #1864.

> _This was written by Claude Code on behalf of @max-sixty_

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

copy-ignored fails silently with very large trees

3 participants