Skip to content

agent: seed a worktree's untracked files in one batch, not four execs per file - #143

Merged
jaredLunde merged 1 commit into
mainfrom
worktree-seed-batch-copy
Oct 7, 2026
Merged

jaredLunde merged 1 commit into
mainfrom
worktree-seed-batch-copy

Conversation

@jaredLunde

Copy link
Copy Markdown
Contributor

Problem

Worktree::seed_from_parent copied each untracked-but-not-ignored file with four calls (is_file, create_dir_all, read_bytes, write_bytes), one file after another:

  • Remote (sandbox): each call is an exec round trip, so seeding cost 4 sequential execs per file: 4,011 execs to spawn a subagent over a working tree with 1,000 new files.
  • Local: the same loop ran blocking std::fs syscalls inline on an async worker, against LocalFs's own rule.

This is the "batch I/O instead of one round trip per item" pattern, here with sandbox execs in place of Tokio's blocking pool.

Change

Copy the ls-files -z list as one batch (Git::copy_files):

  • Local: one spawn_blocking over the whole list.
  • Remote: one fixed sh script (COPY_FILES_SCRIPT) per argv chunk of paths (COPY_ARGV_BUDGET = 128 KiB). Paths are passed as positional parameters and never as shell text, per ShellFs's rule. The script runs mkdir only when the directory changes (ls-files output is sorted), because forks are now the remaining cost.

The skip behavior is the same as before: both arms follow symlinks ([ -f ] / Path::is_file), so a symlink to a file is copied as its content, and a dangling symlink or a file that vanished since listing is skipped. The removed Git::is_file / Git::copy_file had no other callers.

Perf

New bench benches/worktree_seed.rs: Worktree::create against a real repo, with the remote arm on ShellFs (what a replica uses) over RealRunner plus an injected per-exec delay. Median of 3 runs, in ms.

mode files before (execs) after (execs)
remote, 2 ms/exec 10 306 (51) 76 (12)
remote, 2 ms/exec 100 2,426 (411) 187 (12)
remote, 2 ms/exec 1,000 25,355 (4,011) 1,296 (12)
remote, 0 ms/exec 1,000 14,552 (4,011) 1,386 (12)
local 1,000 68 73 (unchanged, now off the async workers)

The exec count is now flat. What remains at 1,000 files is about one cp fork per file inside the sandbox, and that cost doesn't grow with latency.

Tests

  • seeding_copies_awkward_untracked_paths_on_both_arms: spaces, a quote with $HOME and backticks, a leading -, a newline in a file name and in a directory name, a symlink to a file, and a dangling symlink, through both Git::Local and Git::Remote.
  • remote_seeding_spans_several_argv_chunks: checks that no file is dropped or duplicated at a chunk boundary.
  • argv_chunks_stay_within_budget_and_keep_every_item_in_order.

Locally: cargo test -p beyond-ai-agent passes, apart from one flake. exec_endpoint_response_cap::run_applies_the_exec_max_response_bytes_flag hit its 120s deadline once under full-suite load, then passed 3/3 in isolation on this branch; it doesn't touch worktrees. Clippy (-D warnings) and dprint check are clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr

… per file

`Worktree::create` copied each untracked file with a stat, mkdir, read and
write. In a sandbox every one of those is an exec round trip, so seeding
cost 4 sequential execs per file (4,011 for 1,000 files); locally the same
loop ran blocking `std::fs` calls on an async worker.

Copy the `ls-files` list in one batch instead: one `spawn_blocking` locally,
and remotely one fixed `sh` script per argv-sized chunk of paths (paths ride
as positional parameters, never as shell text). The script runs `mkdir` only
when the directory changes, since forks are its remaining cost.

New bench `worktree_seed` (median ms, 2 ms injected per exec, ShellFs):

  files   before (execs)   after (execs)
     10     306 (51)          76 (12)
    100   2,426 (411)        187 (12)
  1,000  25,355 (4,011)    1,296 (12)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr
@jaredLunde
jaredLunde force-pushed the worktree-seed-batch-copy branch from 2f1cf07 to db65aaa Compare October 7, 2026 05:59
@jaredLunde
jaredLunde merged commit eb59a65 into main Oct 7, 2026
21 checks passed
@jaredLunde
jaredLunde deleted the worktree-seed-batch-copy branch October 7, 2026 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant