fix(pty): stop leaking one /dev/ptmx per macOS spawn (node-pty 1.1.0 low_fds[0])

[Juliusolsson05]

Fixes #1437

What was wrong

node-pty 1.1.0 (our pinned stable, and still npm latest) leaks one /dev/ptmx per spawn on macOS. In pty_posix_spawn, the low_fds guard always opens one ptmx at low_fds[0] (stdio is open, so the loop breaks at count == 0). Its cleanup, for (; count > 0; count--) close(low_fds[count]), never closes index 0. macOS caps PTYs system-wide (kern.tty.ptmx_max = 511), so after about 500 agent or terminal spawns every spawn on the machine fails with posix_spawnp failed.

Evidence, from the packaged app today (main pid 94543, 13.5 h uptime):

• 506 /dev/ptmx fds in main.
• Only 16 slave fds, exactly the 16 ttys that still have a process.
• So about 490 orphaned masters, with no slave and no process. That is the low_fds[0] signature.

Upstream fixed it in microsoft/node-pty#882 (af053f22, for microsoft/vscode#182212), but only on the 1.2.0 beta line.

Fix

scripts/patch-node-pty.mjs runs in postinstall before electron-rebuild. It ports #882's pty_posix_spawn onto 1.1.0's source:

• closes every opened low_fds entry, including index 0;
• closes the parent's slave fd (also leaked in 1.1.0);
• routes every early error through done: (1.1.0 leaked the master there);
• has PtyFork close the master on failure and throw posix_spawnp failed: <call>: <strerror>.

Why this ships: node-pty is compiled from source in postinstall, electron-builder rebuilds it per arch with prebuilds excluded, and lib/utils.js loads build/Release first.

Guarding: pty.cc is identified by sha256. A pristine file gets patched; an already-patched file is a no-op; any other file fails the install. node-pty is pinned to exactly 1.1.0. The WHEN THIS FAILS note in the script says what to do when node-pty moves.

Deliberate differences from #882 (see Execution notes in the plan):

• the cleanup loop is bounded by i < 3, because feat(opencode): OpenCode Terminal reports status, turns, conditions and history through opencode-terminal-headless #882's i <= count can read low_fds[3];
• the thrown message keeps the posix_spawnp failed prefix, which src/main/ipc/session.ts classifySpawnFailure keys the posix-spawnp signature on (fix(ipc): session:spawn launders provider exceptions before they cross IPC (#1267) #1324; pinned in session.test.ts).

Why not bump to 1.2.0-beta.15: the beta line brings unrelated ConPTY and API churn to get one function. This is the same trade as scripts/patch-xterm.mjs (#871).

Tests

• testing/unit/patchNodePty.test.ts (7) runs against the byte-exact published 1.1.0 pty.cc, committed under testing/fixtures/node-pty-1.1.0/ because the installed copy is the patched one after postinstall. It covers:
  • the fixture hash;
  • the leak shape in the pristine source;
  • the patched cleanup, slave close and goto done coverage;
  • master close and error text;
  • Linux and the rest of the file unchanged;
  • idempotence;
  • an unknown file refused and left untouched.
• Mutation: restoring the countdown loop turns 2 tests red.
• Review round 1 (a, b and c all MERGE-READY), with minors fixed fail-first (see the disposition comment):
  • the unit test pins the direction of the low_fds[i] != -1 and spawn_err != 0 guards; each inversion fails even with the hash updated;
  • a run of the real script through a symlinked path patches, where before it silently exited 0;
  • session.test.ts pins that the patched posix_spawnp failed: <call>: <strerror> messages keep the posix-spawnp signature.
• testing/system/nodePtyPtmxLeak.test.ts (darwin only) spawns 20 real PTYs, one at a time, and asserts lsof -p ptmx fds return to the baseline. It is a port of feat(opencode): OpenCode Terminal reports status, turns, conditions and history through opencode-terminal-headless #882's test. It fails on the unpatched build (expected 20 to be less than or equal to 0). Run on an isolated patched rebuild, the same measurement gives 0 → 0 (see the evidence comment).

Verification

• The patched pty.cc compiles with node-gyp (Node 24), in an isolated copy. The only warnings are the 2 pre-existing kqueue initialiser warnings.
• npx tsc -b clean.
• Unit test 8/8.

Boundary

• Not verified in the packaged app (never launch the app).
• After merge and a rebuild, lsof -p <main pid> | grep -c /dev/ptmx should track the live session count.
• The current machine only recovers by restarting the app (the owner's call).

🤖 Generated with Claude Code

[Juliusolsson05]

Real-PTY evidence, now that the q117 pause is lifted (one spawn at a time, 20 spawns, lsof -p ptmx count, the same measurement as testing/system/nodePtyPtmxLeak.test.ts):

node-pty build	open ptmx fds after 20 spawns
shared node_modules (pristine 1.1.0, what ships today)	before 0 → after 20
isolated copy patched by scripts/patch-node-pty.mjs and rebuilt with node-gyp	before 0 → after 0

• testing/system/nodePtyPtmxLeak.test.ts on the unpatched build: fails with expected 20 to be less than or equal to 0. That is one leaked master per spawn, exactly the low_fds[0] diagnosis.
• The patched copy was built in a scratch directory, and the shared node_modules was not rebuilt. After merge, npm install applies the patch before electron-rebuild.

🤖 Generated with Claude Code

[Juliusolsson05]

Round-1 disposition. a, b and c are all MERGE-READY, each with one minor finding. All three are fixed, each pinned by a test that fails without its fix:

• a (minor): the unit test didn't pin the guard's direction. Fixed in 2100279. The test now asserts the generated if (low_fds[i] != -1) and the final if (spawn_err != 0). Inverting either one fails, even with PATCHED_SHA256 updated to match (both mutations checked). This is the CI-side pin, because Linux CI never runs the macOS fd test.
• b (minor): run through a symlinked path, the CLI exited 0 without patching. Fixed in 2100279. The entry now compares real paths on both sides. It is pinned by running the real script through a symlinked directory, which was red before the fix.
• c (minor): the comment and plan wrongly said nothing in the app matches on posix_spawnp failed. Fixed in 7e346a6. src/main/ipc/session.ts classifySpawnFailure keys the posix-spawnp signature on that prefix (fix(ipc): session:spawn launders provider exceptions before they cross IPC (#1267) #1324), so the prefix is load-bearing. The comment and plan now say so, and session.test.ts pins that the patched messages (posix_spawnp failed: posix_spawn failed: No such file or directory, as reviewer a recorded from the patched build, and the full-table posix_openpt shape) keep that signature.
• Noted but not changed:
  • c's suspicion about patch-xterm.mjs's identical CLI guard is outside this PR and unreachable through npm's invocation; I'll file an issue if you want one.
  • The "marker comment" wording in the plan is only a nit. Idempotence is the sha256 check.
• Evidence from all three: each independently measured unpatched 0→20 and patched isolated rebuild 0→0. c also verified that the fixture equals npm pack node-pty@1.1.0's pty.cc (sha256) and that the port fidelity against feat(opencode): OpenCode Terminal reports status, turns, conditions and history through opencode-terminal-headless #882 holds, with exactly the two documented deviations.

Head: 7e346a6.

🤖 Generated with Claude Code