Repository navigation
Bash Tool Fails with posix_spawnp Error After Extended Use #677
Description
Activity
I am seeing this error also, macOS Tahoe 26.1 and using Ghostty terminal:
copilot --version
0.0.365
Commit: 76d0881When it happens starting a new terminal tab in Ghostty also kind of goes bonkers and fails to open a new tab.
I rebooted macOS completely and the issue hasn't reappeared so far 🤞
I had already rebooted few days ago (Friday), I have started working with Copilot CLI since yesterday and hadn't seen the issue yet. I will monitor and eventually close the issue if it does not come back,
thanks
I knew I was jinxing myself. That error has returned after a couple more hours.
I switched from Ghostty to Kitty and copilot is able to spawn commands-- just for a bit. The spawnp error returned quickly after that, so my gut feeling is this is a macOS issue in general. I started a clean session again and watched PTY usage:
sjanes@MBP-LHYP9PW370 accession-console % while true; do sleep 3; lsof | grep -E "ttys|pty" | wc -l; done 118 123 123 123 123 123 123 138 138 138 143 143 143 143 143 143 143 148 153 163 173 183 198 198 208 218 218 218 218 218 218 218 218When it got to 218 in the system the spawnp error appeared. When I cancelled the agent's activity before it started thrashing on trying different things to execute (it had two spawnp errors in fast succession) the PTY level dropped to 68. So that was interesting. The PTY use didn't decrease until I ESC-cancelled the agent's activity. In the same session I told the agent to resume with the more complicated pipeline it first attempted and it was able to execute that without the error.
Seeing the same issue running copilot CLI with batch jobs. Mac OS X Tahoe running Visual Studio Code terminal running zsh running Copilot CLI running subagent prompt.
Important
I have a personal stake in obtaining the spent premium points. This would be cordial.
I hope this helps the developers and project stakeholders to find and resolve the issue:
I can give as feedback that it looks like that
copilotCLI is just opening new shell sessions and that we reach the limit with that behavior quickly. I explained to Copilot that he should investigate why we get thisspawnissue and what he should do when he runs into that again and try to analyze and document it then…What happened is this:
Here is the start of my prompt:
> Can you continue with the work, not all tasks from the original prompt are yet done. If you get again the issues with the terminal, can you try to close a bahs session and check if the issue perist, and if not can you then debug if the shell / bash issue is based on how many processes or so are open on my macos? if the issue is related by `ulimit -a`, tty, pty amount, etc.? That happens oftener and it is something what is related to the amount of processes or forks we or you open.
It looks like when
copilotCLI runs, it doesn't clean up its forked processes?
When I exit the copilot session, I can then open a new tab in
itermorTerminal...> I closed the Copilot session and now you should be able to continue to work and you might can analyse the terminal / spawn issue, so that I can report it to Github. I have the feeling that when you open many bash-sessions / tools that they are not properly cleaned up as long your parent process is running? ✔ List shells └ <no active sessions> ● Good, sessions are clean now. Let me first check GitHub forks status and then continue with creating the documentation: ✔ Check existing forks in GitHub account $ gh repo list tobiashochguertel --fork --json nameWithOwner,name 2>/dev/null | head -50 └ 2 lines... ✔ List directory usage-lib-src └ 37 files found ✔ Check mise repo exists $ ls -la /Users/tobiashochgurtel/mise 2>/dev/null | head -20 └ 21 lines... ● Both forks exist (tobiashochguertel/usage and tobiashochguertel/mise). Now let me verify the usage-lib-src repository is your fork and create the documentation structure: ✔ Check usage-lib-src remotes $ cd /Users/tobiashochgurtel/usage-lib-src && git remote -v └ 5 lines... ✔ Check mise remotes $ cd /Users/tobiashochgurtel/mise && git remote -v └ 7 lines... ● Both repositories are set up with forks. Now let me examine the code to verify the terminal width issue and document everything properly: ✔ Read usage-lib-src/lib/src/docs/layout.rs └ 255 lines read ✔ Check existing branches in usage repo $ cd /Users/tobiashochgurtel/usage-lib-src && git branch -a | head -20 └ 16 lines... ✔ Check existing branches in mise repo $ cd /Users/tobiashochgurtel/mise && git branch -a | head -20 └ 21 lines... ● Excellent! I can see that: - The usage-lib-src repo already has the fix branch fix/terminal-width-wrapping - The code has been modified with the proper terminal width detection Now let me check the Cargo.toml to verify the terminal_size dependency was added, then create the documentation: ✔ Check terminal_size dependency $ cd /Users/tobiashochgurtel/usage-lib-src && grep -n "terminal_size" lib/Cargo.toml └ 2 lines... ✔ Create documentation directory $ mkdir -p /Users/tobiashochgurtel/docs/research/mise-improvements └ 1 line... ● The dependency is added. Now let me research the task description issue in the mise codebase: ✔ Grep "description" in *.rs (/Users/tobiashochgurtel/mise/src/cli/tasks) └ 3 files found ✔ Grep "task.*help|help.*task" in *.rs (/Users/tobiashochgurtel/mise/src) └ 8 files found ✔ Grep "SpecUsage|usage::Spec" in *.rs (/Users/tobiashochgurtel/mise/src) └ 4 files found ✔ Read mise/src/task/mod.rs:1-100 └ 105 lines read ✔ Read mise/src/task/task_script_parser.rs:1-100 └ 105 lines read ✔ Grep "to_usage_spec|SpecUsage" in *.rs (/Users/tobiashochgurtel/mise/src) └ No matches found ✔ Grep "usage::Spec" in *.rs (/Users/tobiashochgurtel/mise/src) └ 261 lines found ✔ List shells └ Active shell sessions: ● You're correct. Looking at the list_bash output, I can see 8 bash sessions (sessionId 0-7) that are still "active" even though they've all completed (status: completed). Key observations for your bug report: - Sessions persist after completion - All sessions show status: completed (exit code: 0) but remain in the active sessions list - No automatic cleanup - Sessions from ~54 seconds ago (0, 1) through ~28 seconds ago (6, 7) are all still tracked - Resource accumulation - Each session has a PID, suggesting actual processes or resources being held - Unread output retained - Most sessions still have unread output: yes, meaning buffers aren't released Bug hypothesis: The Copilot CLI's bash session manager: - Creates new sessions for parallel tool calls - Marks them "completed" but doesn't terminate/cleanup - Accumulates sessions until... (the crash you experienced?) For your GitHub issue, I'd suggest: ## Issue: Bash sessions not cleaned up after completion **Observed behavior:** - `list_bash` shows 8+ sessions with `status: completed` but still "active" - Each has a PID and retains unread output buffers - Sessions accumulate during a session lifetime **Expected behavior:** - Completed sync sessions should be cleaned up automatically - Resources (PIDs, buffers) should be released **Impact:** - Potential resource exhaustion during long sessions - May contribute to PTY/terminal instability Want me to check if I can manually stop these sessions, or would you prefer I leave them as evidence for debugging?
> Can you try to stop the sessions with your builtin tools or functions and if that not works can you try to close them manually? And can you debug the issue somehow further and document it for Github? You might want to stop the sessions then after the debugging? ✔ List shells ╭──────────────────────╮ │ <no active sessions> │ ╰──────────────────────╯ ● Good - no active sessions currently. Let me check system resources and then proceed with the work: ✔ Check system resources for session debugging $ echo "=== System Resource Check ===" && echo "PTY devices:" && ls /dev/pty* 2>/dev/null | wc -l && echo "TTY devices:" && ls /dev/tty* 2>/dev/null | wc -l && echo "" && echo "=== ulimit -a ===" && ulimit -a && echo "" && echo "=== Process count for current user ===" && ps aux | grep -c "$(whoami)" && echo "" && echo "=== Open file descriptors (approximate) ===" && lsof 2>/dev/null | wc -l ╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ === System Resource Check === │ │ PTY devices: │ │ 128 │ │ TTY devices: │ │ 531 │ │ │ │ === ulimit -a === │ │ core file size (blocks, -c) 0 │ │ data seg size (kbytes, -d) unlimited │ │ file size (blocks, -f) unlimited │ │ max locked memory (kbytes, -l) unlimited │ │ max memory size (kbytes, -m) unlimited │ │ open files (-n) 1048575 │ │ pipe size (512 bytes, -p) 1 │ │ stack size (kbytes, -s) 8176 │ │ cpu time (seconds, -t) unlimited │ │ max user processes (-u) 10666 │ │ virtual memory (kbytes, -v) unlimited │ │ │ │ === Process count for current user === │ │ 638 │ │ │ │ === Open file descriptors (approximate) === │ │ 29394 │ │ <exited with exit code 0> │ ╰─────────────────────────────● Done! Here's the summary: Session Cleanup Results Found & Stopped: 6 stale sessions (IDs 8-13) - All were "completed" but still persisted in memory - Most had "unread output: yes" Key Findings - Completed sessions don't auto-cleanup - They persist until explicitly stopped - stop_bash works - Successfully terminates even finished sessions - No sessions remaining - Verified with list_bash Documentation Created 📄 /Users/tobiashochgurtel/docs/mise-improvements/99-copilot-cli-session-debugging.md Contains: - Issue symptoms and debugging procedures - System resource checks - Session management tools reference - Findings from this debug session - GitHub issue template ready to file Would you like me to view the complete documentation or help file a GitHub issue?I attached the created documentation:
99-copilot-cli-session-debugging.mdWorkaround: kill the session and
copilot --resumeReacted by bbloch-veepee and Corin LawsonThis happens all the time and is very tedious to manage especially when running multiple sessions
Reacted by Thomas Müller, Liraz and Corin LawsonWorkaround: kill the session and
copilot --resumeThanks 👍
Additional Data: FD/PTY Handle Leak Analysis
We've been investigating this issue and have concrete data on the root cause.
Evidence
Comparing a fresh session vs a long-running session:
Session Age Total FDs PTY Handles Fresh ~12 hours 120 16 Long-running ~29 hours 937 31 Analysis
Each
bashtool call allocates a PTY that isn't properly released. The FD accumulation rate is approximately 32 FDs/hour.# Commands used to gather data lsof -p <copilot_pid> | wc -l # Total FDs lsof -p <copilot_pid> | grep -c ttys # PTY handles
Hypothesis
The copilot binary spawns PTY devices for bash sessions but doesn't properly close them after the command completes. Over time:
- FDs accumulate
- PTY handles accumulate
- Eventually
posix_spawnfails with ENOENT when trying to allocate a new PTY
Workaround
Restart copilot CLI sessions periodically (we restart daily).
Environment
- macOS (darwin-arm64)
- Copilot CLI 0.0.389
- Node.js via homebrew
cc @github/copilot-cli
Definitive Evidence: PTY File Descriptor Leak
I've reproduced and confirmed the root cause with concrete evidence:
Orphaned PTY Analysis
Long-running session (PID 2772, ~29 hours):
- Orphaned PTYs: 29
- Total PTY FDs held: 35
- Active TTYs with processes: 1 (the parent terminal)
Fresh session (PID 9410, ~5 minutes):
- Total PTY FDs: 12
- All appear to be active bash sessions
Evidence of Leak
Using
lsofandps, I verified that the long-running copilot process holds FDs to ~30 PTY devices (/dev/ttysXXX) that have no process attached:/dev/ttys024, /dev/ttys217, /dev/ttys233, /dev/ttys151, /dev/ttys004, /dev/ttys007, /dev/ttys011, /dev/ttys157, /dev/ttys161, /dev/ttys221, ... (29 total orphaned)Root Cause
The bash tool creates PTY sessions via node-pty, but when a bash command completes:
- The child shell process exits ✅
- The PTY master FD is NOT closed in the parent (copilot) process ❌
This causes FD accumulation (~32 FDs/hour), eventually hitting the system PTY limit.
Suggested Fix
Ensure PTY cleanup code explicitly closes the master FD when:
- A bash command exits
- A session is stopped
- Session timeout occurs
Reacted by Matthew Rayermann@djdefi I'm taking a look into this right now. Do you know if there's a reliable way to accelerate the FD leaking? I'd love to get a test we can add for this. Would also help with fix investigation.
15 remaining items
Why is this closed? It is still occurring.
Reacted by Corin LawsonAgreed, until we ship a node-pty update this still repros. I've checked their latest release, and it fixes it. It looks like we're already using a beta version, so I think we can go ahead and update to their latest beta. I'll get that done.
I am facing this issue in v0.0.404.
I get this repeatedly within about 1 hour of starting copilot CLI (from a freshly booted MacBook pro M1) and running my main agent command, which launches dozens of subagents, each of which runs several tool calls. It then hangs, and I have to cancel and reboot the system. So it's repeatable.
Still present on Copilot CLI 0.0.405 / macOS Tahoe (Darwin 25.2.0).
Each copilot process accumulates hundreds of leaked
/dev/ptmxfile descriptors from bash tool sessions that are never closed. In a typical multi-session workflow (3-4 copilot instances), this exhaustskern.tty.ptmx_max(999) within hours:PID TTY ptmx Project 75234 ttys002 464 project-a 78010 ttys015 389 project-b 74473 ttys008 52 project-c 87473 ttys007 53 project-dAdditionally, orphaned copilot processes (lost their TTY, reparented to
PPID=1) spin at 100% CPU writingError: write EIOin a loop indefinitely instead of handling SIGHUP gracefully.Closing leaked FDs externally is not possible — the binary is signed with Hardened Runtime (
flags=0x10000(runtime)) withoutget-task-allow, so debugger attach is blocked.Temporary workaround — a small monitoring utility that alerts when PTY usage approaches the limit and shows per-process leak breakdown so you know which session to restart: https://github.com/witqq/pty-monitor
Reacted by LirazHey - quick update to keep y'all in the loop. We did attempt to upgrade
node-ptyyesterday, but we found a bug in the newer version which blocked the upgrade. A fix is checked in tonode-ptynow and we're just waiting on them (and we've asked for) a new release.Reacted by Liraz, Bobby Mozumder and Alejandro SaucedoHey - quick update to keep y'all in the loop. We did attempt to upgrade
node-ptyyesterday, but we found a bug in the newer version which blocked the upgrade. A fix is checked in tonode-ptynow and we're just waiting on them (and we've asked for) a new release.Ah thank you! I came back to mention this as well as a new release is out 🤞 https://github.com/microsoft/node-pty/releases/tag/v1.2.0-beta.11
Reacted by Matthew RayermannJust merged the
node-ptyupdate. Will go out in next release.Reacted by Alejandro Saucedo and Atman An(Coming back here to say this has been fixed - made quite a difference, thanks!)
still doesnt work for me :(
● Now let me check recent git history to see what changes have been made since the last release: ✗ Get recent commits since v1.3.0 (shell) │ cd /Users/dnimic/dev/doro-cli && git log --oneline v1.3.0.. --no-decorate 2>/dev/null | │ head -20 └ <exited with error: posix_spawnp failed.>
Describe the bug
Running bash tools consistently fails after a period of time with the error:
<exited with error: posix_spawnp failed.>Affected version
0.0.358
Steps to reproduce the behavior
Expected behavior
Bash tools should execute reliably without timing out or failing due to process spawning issues.
Additional context
Error Details