One reusable OpenSSH connection per workspace

[blooop]

Two commits, in this order.

1. feat: one reusable OpenSSH connection per workspace (#422)

dl <ws> -- <cmd> typed at a terminal has gone into the container over OpenSSH since M3, through the host alias devpod up publishes. It opened a fresh connection every time, and connection setup is nearly the whole cost of that trip. Three -o options on an argv clients/ssh.rs already built are the whole mechanism: ControlMaster=auto, a derived ControlPath, ControlPersist=60. auto means no pre-warm, so the spawn counts do not move; the argv does, and its pins say so.

The socket is derived rather than configured, and its digest is the load-bearing part. A master filters SendEnv against its own permit list, in silence, at exit 0 (#389 reproduced it: GOT=[], rc=0), so a master opened by a run with no token would hand the next run an empty GH_TOKEN and an unauthenticated gh with nothing anywhere to say so. The digest covers the host alias, the SendEnv permit list and $SSH_AUTH_SOCK, which makes that state unrepresentable rather than documented: a client whose list differs from the master's cannot find that master. Fields go in length-prefixed, so no two inputs encode alike. 16 hex characters of SHA-256, because a ControlPath has about 104 bytes to live in and an alias alone spends 30 of them.

Reuse::Multiplexed(ControlSocket) | Reuse::Direct is a sum and not an Option, because Direct has real causes: a path too long for sun_path, a directory dl cannot make, or a % in the path. The length one is subtler than it looks and CI caught it: muxserver_listen binds <ControlPath>.<16 characters> and renames it into place, so the budget is 17 bytes smaller than sun_path, and a check against the full 104 took the e2e suite down with unix_listener: path ... too long, exit 255, thirteen tests. SUN_PATH and LISTEN_SUFFIX are both named now. That last one is not hypothetical tidiness: OpenSSH runs ControlPath through percent_expand before it binds anything, and an unknown key there is fatal(), so a user whose XDG_CACHE_HOME held a % would have had every terminal session die. The derived name is hex; everything above it is the user's, so a % anywhere in the path means do not multiplex. Both arms run the same command and differ in latency only. Everything that can go wrong ends at Direct, so a session that cannot be multiplexed is one that runs unmultiplexed, never one that fails. The sockets live under an ssh-control leaf of dl's cache directory (the LAUNCH_LOCK_DIR precedent), 0700, since anyone who can reach a master's socket is inside the container.

2. fix: DRAIN_GRACE bounds the drain, not each pipe (#501)

capture computed the 500ms grace inside each collect and drained the two pipes serially, so one descendant holding both write ends was charged twice: a measured 1.008s where the bound says 500ms. One deadline is now taken once and handed to both calls. Nothing is lost by sharing it, since both drain threads start before the wait for the child does. lib.rs:489-491 already named ssh's ControlMaster as the production case for a held pipe, and commit 1 puts one on dl's hottest path, which is why this lands here rather than being rediscovered later as a latency mystery.

Tests

Test-first, and both red checks were run against the old behaviour rather than assumed.

• two_send_env_permit_lists_never_derive_one_socket (the mandatory property, all pairs over six lists including ["AB","C"] / ["A","BC"], which collide without the length prefixes). Red without the permit list in the key.
• the_same_request_derives_the_same_socket_every_time, the_agent_socket_a_master_pins_is_part_of_its_key, two_workspaces_never_share_a_master.
• the_whole_argv_is_what_dl_hands_to_openssh and a_session_that_cannot_multiplex_carries_no_control_options_at_all, the two argv pins.
• a_socket_path_too_long_for_sun_path_leaves_the_session_direct, a_socket_directory_dl_cannot_make_leaves_the_session_direct and a_cache_directory_with_a_percent_in_it_leaves_the_session_direct and the_room_openssh_takes_for_its_own_temporary_socket_is_counted_too, the fail-closed causes. the_socket_directory_is_this_users_alone for the mode.
• a_run_with_a_token_and_a_run_without_never_share_a_master and the_control_socket_lives_under_devlaunchs_own_cache_directory, the same facts at the flow.
• two_pipes_one_descendant_holds_cost_one_grace_between_them for capture pays DRAIN_GRACE once per pipe, so the drain bound is 1.0s not 0.5s #501: a real setsid'd descendant holding both pipes, asserted at under one and a half graces and at over four fifths of one, so an EOF that arrived cannot make it pass having pinned nothing. Measured 1.008s before, ~0.5s after.

cargo test --workspace, cargo clippy --locked --all-targets -- -D warnings, cargo fmt --check and pytest test all green. No public API moved, so no snapshot regeneration.

Closes #422. Closes #501.

🤖 Generated with Claude Code

Summary by Sourcery

Reuse OpenSSH connections per workspace and ensure captured command output drains within a single bounded grace period.

New Features:

• Reuse one OpenSSH connection per workspace for terminal commands through derived, workspace-scoped control sockets.
• Automatically isolate multiplexed connections by environment forwarding and SSH agent state while falling back to direct connections when multiplexing is unsafe.

Bug Fixes:

• Bound command output draining with one shared grace period across stdout and stderr instead of applying the grace separately to each pipe.

Enhancements:

• Store control sockets in a private cache directory and cleanly handle stale, inaccessible, unsafe, or overlong socket paths without failing sessions.

Documentation:

• Document the connection reuse behavior, performance impact, isolation guarantees, and fallback behavior.

Tests:

• Add coverage for control socket derivation, isolation, path safety, permissions, command arguments, and shared drain timing.

[sourcery-ai]

Sorry @blooop, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 6 days and 2 hours by commenting @sourcery-ai review. Upgrade to get a review now.

[sourcery-ai]

Reviewer's Guide

The PR makes terminal SSH launches reuse one OpenSSH master per workspace while preventing cross-run sharing when environment forwarding or agent state differs, gracefully falling back to direct connections when multiplexing is unsafe; it also fixes capture draining so stdout and stderr share a single post-exit grace period.

Sequence diagram for reusable workspace SSH connections

sequenceDiagram
    actor User
    participant DL as dl
    participant SSH as OpenSSH
    participant Master as Workspace master
    participant Container

    User->>DL: dl workspace -- command
    DL->>DL: Reuse::derive(workspace, SendEnv, SSH_AUTH_SOCK)
    alt Multiplexed
        DL->>SSH: command_args(ControlMaster=auto, ControlPath, ControlPersist=60)
        SSH->>Master: Open or reuse control socket
        Master->>Container: Execute command
        Container-->>Master: Output
        Master-->>DL: stdout and stderr
    else Direct
        DL->>SSH: command_args without control options
        SSH->>Container: Execute command
        Container-->>DL: stdout and stderr
    end
    DL-->>User: Command result

Loading

Sequence diagram for bounded stdout and stderr draining

sequenceDiagram
    participant Runner
    participant Child
    participant StdoutDrain
    participant StderrDrain

    Runner->>StdoutDrain: Start drain thread
    Runner->>StderrDrain: Start drain thread
    Runner->>Child: Wait for child exit
    Child-->>Runner: Exited
    Runner->>Runner: Compute one DRAIN_GRACE deadline
    Runner->>StdoutDrain: collect(deadline)
    Runner->>StderrDrain: collect(deadline)
    StdoutDrain-->>Runner: Captured stdout or timeout
    StderrDrain-->>Runner: Captured stderr or timeout
    Runner-->>Runner: Return captured result

Loading

Flow diagram for SSH control socket identity and fallback

flowchart TD
    A[Build workspace SSH request] --> B[Derive key from host alias, SendEnv permit list, and SSH_AUTH_SOCK]
    B --> C{Path fits and ssh-control directory is usable?}
    C -->|Yes| D[Use hashed ControlPath]
    D --> E[OpenSSH ControlMaster=auto]
    E --> F[Reuse workspace master for 60 seconds]
    C -->|No| G[Use Direct connection]
    G --> H[Run command without control options]

Loading

File-Level Changes

Change	Details	Files
Adds per-workspace OpenSSH connection multiplexing with safely derived control sockets and a direct-execution fallback.	Introduces a length-prefixed SHA-256 socket key covering workspace alias, permitted environment variables, and SSH agent socket. Creates a user-only control-socket directory under the application cache and falls back to direct SSH when setup or Unix socket path limits prevent multiplexing. Adds ControlMaster, ControlPath, and ControlPersist options only for multiplexed sessions. Integrates reuse derivation with terminal SSH flow and expands argv, isolation, fallback, permissions, and cache-location tests.	rust/devlaunch-core/src/clients/ssh.rs rust/devlaunch-core/src/flows/launch.rs CHANGELOG.md docs/performance.md
Bounds post-exit pipe draining with one shared deadline across stdout and stderr.	Computes DRAIN_GRACE once after child exit and passes the same deadline to both drain collectors. Adds a timing test using a descendant holding both pipe write ends to verify the grace is not charged twice. Updates drain documentation and existing collector test calls for the deadline parameter.	rust/devlaunch-runner/src/lib.rs rust/devlaunch-runner/src/tests.rs CHANGELOG.md

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#422	Give each workspace one reusable OpenSSH connection by adding ControlMaster=auto, a derived ControlPath, and ControlPersist=60 to the existing SSH command argv without pre-warming or introducing a new transport.	✅
#422	Derive the control socket identity from the workspace/host alias, the SendEnv permit list, and SSH_AUTH_SOCK, using length-prefixed hashing and falling back to direct SSH when the socket path or directory cannot be used.	✅
#422	Ensure the reuse implementation is safe and maintainable: represent multiplexed versus direct execution as a Reuse sum, keep sockets in dl's private cache directory, and add tests covering permit-list isolation, socket derivation, argv changes, and fallback behavior.	✅
#501	Make the stdout and stderr drain operations share a single DRAIN_GRACE deadline so that capturing a command with one descendant holding both pipes is bounded by approximately 0.5 seconds rather than paying the grace once per pipe.	✅

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[codecov]

Codecov Report

❌ Patch coverage is 99.36102% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.78%. Comparing base (2686879) to head (ae71b57).

Files with missing lines	Patch %	Lines
rust/devlaunch-core/src/clients/ssh.rs	99.58%	1 Missing ⚠️
rust/devlaunch-core/src/flows/launch.rs	98.41%	1 Missing ⚠️

Additional details and impacted files

Flag	Coverage Δ
python	42.98% <ø> (ø)
rust	96.08% <99.36%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
shipped code (rust)	96.08% <99.36%> (+0.02%)	⬆️
harness and tooling (python)	42.98% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.