test(e2e): port CLI exec, attach, and SSH host identity tests to testkit by matthewgrossman · Pull Request #5442 · NVIDIA/OpenShell

matthewgrossman · 2026-11-10T05:59:02Z

Summary

Port the e2e/rust tests that drive a long-running openshell CLI process (exec streaming, attached create, --no-keep, attach/reconnect, and SSH host identity) to testkit, and delete the legacy tests. Stacked on #5431 (PR1); review the last two commits.

Related Issue

Closes #5426
Part of #5281. Stacked on #5431.

Changes

testkit: CliChild (Cli::spawn/spawn_with_stdin, cx.cli_spawn/cx.cli_spawn_with_stdin): a long-running CLI process with stdin, a line reader, its pid, signal/kill and wait. Also Cli::gateway() and Cli::apply_env(), which export the CLI environment for a plain ssh client.
Conformance (runs on every conformance row):
- exec_cli.rs: cli_exec_large_output_is_complete, cli_exec_piped_stdin_crosses_grpc_message_limit
- cli.rs: cli_attached_create_streams_main_and_reports_exit, cli_no_keep_removes_sandbox_after_main_exit
- attach.rs: attach_reconnect_replays_history_for_the_same_main_process (it also stops and starts the sandbox while a connect session is open), attach_main_exit_255_is_not_retried_as_transport_failure
- ssh.rs: ssh_host_identity_stable_across_stop_start_and_new_on_recreate
New tests/suites/features/cli-attach with one Linux row (ubuntu-docker-rootful, deb) in .github/workflows/e2e-suites.yml: connect_recovers_its_ssh_transport, connect_forwards_pid_targeted_termination_and_reaps_ssh. It reads host procfs directly and fails with a clear message off Linux. The row isn't on k3s, because of Investigate delayed SSH input-owner release through kubectl port-forward #4445.
Deleted e2e/rust/tests/ssh_host_identity.rs, and the tests this PR ports from e2e/rust/tests/sandbox_lifecycle.rs. That file stays, holding only on_failure_policy_replaces_runtime_and_preserves_workspace until bug(vm): on-failure restart never completes because the preserved overlay fails e2fsck #5507 (a VM driver bug) is fixed, so its follow-up entry in tests/tmachine/artifacts.nix stays too, with an updated comment. Also removed:
- the stale [profile.e2e-kubernetes] waiver for canonical_main_connect_recovers_its_ssh_transport in .config/nextest.toml;
- the now-unused openshell_tty_cmd, shell_escape and nix dependency in e2e/rust.
Docs: tests/testkit/README.md, tests/suites/conformance/README.md, and TESTING.md.

Legacy test → new test

Legacy test	New test
`sandbox_lifecycle::sandbox_exec_large_output_is_complete`	`exec_cli::cli_exec_large_output_is_complete`
`sandbox_lifecycle::piped_exec_stdin_crosses_grpc_message_limit`	`exec_cli::cli_exec_piped_stdin_crosses_grpc_message_limit`
`canonical_main_exit_zero_completes_persistent_sandbox` + `canonical_main_nonzero_exit_preserves_status`	merged into `cli::cli_attached_create_streams_main_and_reports_exit`
`sandbox_create_with_no_keep_cleans_up_after_tty_command` + `…_preserves_failure_then_cleans_up`	merged into `cli::cli_no_keep_removes_sandbox_after_main_exit`
`canonical_main_disconnect_reconnect_replays_history_for_same_process`	`attach::attach_reconnect_replays_history_for_the_same_main_process`
`canonical_main_exit_255_is_not_retried_as_transport_failure`	`attach::attach_main_exit_255_is_not_retried_as_transport_failure`
`canonical_main_connect_recovers_its_ssh_transport`	`features/cli-attach::connect_recovers_its_ssh_transport`
`canonical_main_connect_forwards_pid_targeted_termination_and_reaps_ssh`	`features/cli-attach::connect_forwards_pid_targeted_termination_and_reaps_ssh`
`ssh_host_identity::ssh_host_identity_survives_restarts_and_changes_after_recreation`	split: `ssh::ssh_host_identity_stable_across_stop_start_and_new_on_recreate`; the gateway-restart part moves to PR6

Defaults applied

No cli_pty. Plain Cli::run drives attached sandbox create -- <cmd>, so the legacy TTY --no-keep case now runs without a terminal. Only connect_forwards_pid_targeted_termination_and_reaps_ssh needs one; it passes an openpty follower as the CliChild's stdin.
Split of the attach tests: reconnect/replay and exit 255 are in conformance; SSH transport recovery and pid-targeted termination are in features/cli-attach on one Linux row.
Named gateway: the ssh host identity test needs one, because sandbox ssh-config writes it into its ProxyCommand. Without one, the test fails with a clear message. It reads the fingerprint through the SDK and keeps the "auth.json unreadable" check.

Testing

cargo fmt and cargo clippy --all-targets are clean for testkit, conformance and features, on macOS and in Linux (Docker, rust:1.95-bookworm) for the Linux-only cli-attach code
cargo test -p openshell-testkit --lib passes, including the new CliChild unit tests
cargo check --features e2e,e2e-host-gateway --tests passes in e2e/rust
Lockfile alignment check passes
E2E tests added/updated (to be validated by Branch E2E on this PR)

Checklist

Follows Conventional Commits
Commits are signed off (DCO)

copy-pr-bot · 2026-11-10T05:59:06Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

github-actions · 2026-11-10T05:59:11Z

Label test:e2e applied, but pull-request/5442 does not exist yet. A maintainer needs to comment /ok to test bbbe0433ccf56073b6d28ed92514f2630ce16ba4 to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

matthewgrossman · 2026-11-10T05:59:33Z

/ok to test bbbe043

Move the driver-agnostic sandbox lifecycle, labels, templates, settings, image, process, Landlock and upload tests from e2e/rust to the conformance suite, the Podman OCI identity test to the podman driver suite, and the doctor, proxy-preflight and WebSocket tunnel checks to crate tests. Move the live-Internet benchmark to mise run perf:live-internet. Delete the legacy files and their references. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…d test image helpers Add cx.cli_sandbox (with policy files), ExecOutput::success, cx.sandbox_details, cx.wait_absent and Engine::build_test_image, use them in the ported lifecycle tests, and replace the duplicate policy file, sandbox get and test image code in the host-gateway, oci-image and podman suites. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Replace hand-written exit-code checks in the smoke, lifecycle, persistent_workspace, file_transfer and policy_advisor suites. Polls and checks that expect a failure keep reading the exit code. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…e helpers Add SandboxBuilder::create_ready, ExecOutput::expect_stdout, Sandbox::poll_exec, Context::delete_and_expect_absent and testkit::with_deadline, and convert std::io::Error into testkit::Error so I/O results take .context directly. Use them across the conformance suite, which also gets shared template, global-setting and label-list helpers and a split on-failure restart poll. Make Engine::build_image private so callers go through build_test_image, which removes the image. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…proxy Replace the hand-written WebSocket bridge with openshell_sdk::edge_tunnel::start_tunnel_proxy, so the test covers the client half of the tunnel and its edge-token headers as well as the router. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Move expect_listed, expect_absent, wait_absent and the CLI's sandbox get JSON (now get_json) from Context onto Sandbox, and drop delete_and_expect_absent. Give ExecOutput failure() and CliOutput expect_stdout(), so both outputs offer the same checks. Put with_deadline's description first, like poll_until. Make CliSandboxBuilder::run consume the builder like create, and rename its policy setter to policy_yaml. Update the testkit README's list of what a test can do. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Replace CliSandboxBuilder's private PolicyFile with testkit::YamlFile, and drop YamlFile's map_err calls now that I/O errors convert into testkit::Error. CliOutput::success now includes stdout in its failure message, like ExecOutput::success, since many commands report errors there. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…ting changes Add SandboxBuilder::image and CliSandboxBuilder::create_ready, so both builders set an image and wait for Ready the same way. Convert tonic::Status into testkit::Error, keeping its code, so gRPC results take .context() directly, and drop Error::from_status. Create the host-gateway sandbox with cx.sandbox(..).image(..).policy(..) instead of a hand-written CLI create. In credential gating, use create_ready and poll_exec. Drop the remaining map_err calls on I/O results in host-server-support. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman · 2026-11-10T18:19:24Z

/ok to test 1b45c0b

…ble driver The gateway feature matrix builds without Docker, Podman or the VM driver, where the test file has no drivers to check. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…ure restart Deleting a Docker sandbox that is still starting fails with a 409 because its supervisor volume is in use, so the labels test now waits for Ready before cleanup. The VM driver needs more than 45s to report an on-failure restart; the original e2e test allowed 60s. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…lders Context, SandboxBuilder, Sandbox and CliSandboxBuilder now share one Handles value (clients, CLI, cleanup), so CliSandboxBuilder owns its state like SandboxBuilder instead of borrowing the context. Both builders are #[must_use] at the type level, and CliSandboxBuilder::image and policy_yaml take impl Into<String> like SandboxBuilder::image. ExecOutput and CliOutput run success, failure and expect_stdout through one shared check, so their messages match and failure() also reports stderr. ExecOutput.command is renamed description, since it names the exec and its sandbox. CliOutput::message is renamed report_text. serde_json and serde_yml errors convert into testkit::Error, which removes the last map_err(Error::from_error) calls. expect_listed uses let-else instead of ensure plus expect. kubernetes-updates uses create_ready. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman · 2026-11-10T19:07:47Z

/ok to test d20313c

Read errors in gateway.rs and the image test use .context(..) now that the error types convert. Error::from_error and from_display are crate-private. Document why the global-setting helpers take &Cli, why the template test names and adopts its sandbox by hand, SandboxBuilder::restart_policy, and that the --no-keep test checks only gateway-visible removal. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Restore edge_tunnel_e2e with the --enable-websocket-tunnel flag in the Docker and Podman gateway wrappers, and ephemeral_cleanup with its Podman and VM lane entries. Both check what conformance cannot: the tunnel flag's wiring through a real gateway, and that drivers remove their resources. They move in later batches. Remove the in-process WebSocket tunnel server test and its openshell-sdk dev-dependency, which checked the listener but not the flag wiring. The --no-keep conformance test now says the driver-resource check stays in ephemeral_cleanup. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

The two live-Internet benchmarks were manual, needed the live Internet, and nothing referenced them. Remove the Python port and its perf:live-internet task instead of keeping it. #5472 tracks whether to bring one back. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

proxy_settings_are_inserted_before_nested_driver_tables covers insert_docker_driver_settings, which the remaining corporate proxy tests still use. It was removed with the incoherent-configuration test by mistake. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

On the VM driver the replacement sandbox never starts: the preserved overlay disk fails e2fsck, and the sandbox stays in Starting. The test fails on every VM lane. Ignore it with a link to #5507, which also tracks removing the ignore. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…2e/rust Restore `on_failure_policy_replaces_runtime_and_preserves_workspace` in `sandbox_lifecycle.rs` and `settings_management.rs` as they were, and drop their conformance counterparts. The on-failure restart test ran on the Docker and Kubernetes lanes but never on VM or Podman. Ported to conformance it fails on every VM row (#5507), and an `#[ignore]` there removes the coverage it gave Docker and Kubernetes. The global settings test changes a setting for the whole gateway, so it can't share a conformance run with other tests until the runner has a test group for it (#5295). `sandbox_setting_round_trip` stays in conformance. Signed-off-by: Evan Lezar <elezar@nvidia.com>

matthewgrossman · 2026-11-12T15:37:49Z

/ok to test 77794c7

… trip sandbox_setting_round_trip deleted the gateway-wide ocsf_json_enabled value and restored it on cleanup, which changes every sandbox on a shared gateway while it runs. Fail with a hint when a global value is set instead, as policy_advisor does. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Add Cli::spawn and Cli::spawn_with_stdin (with Context wrappers) that start the CLI and keep it running, for commands that stream output or read input. Stdout and stderr are drained in the background so the pipe never blocks. Also add Cli::apply_env so another program, such as ssh with a ProxyCommand from sandbox ssh-config, reaches the same gateway. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Move the CLI exec, attached create, no-keep, attach, and SSH host identity tests from e2e/rust to the conformance suite, and the procfs-based connect transport recovery and signal forwarding tests to a new features/cli-attach suite. Delete the ported legacy tests, ssh_host_identity.rs, the now-unused PTY harness helper, and the stale Kubernetes nextest waiver. sandbox_lifecycle.rs keeps only the on-failure restart test until #5507 is fixed. The gateway-restart check of SSH host identity moves to a later port. Part of #5281 Closes #5426 Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman · 2026-11-12T22:05:35Z

/ok to test 933f6ea

matthewgrossman added the test:e2e Requires end-to-end coverage label Nov 10, 2026

elezar force-pushed the test/5425-port-sandbox-lifecycle/matthewgrossman branch from 1ce77a5 to f8b1ced Compare November 10, 2026 10:44

matthewgrossman added 9 commits November 10, 2026 09:39

test(gateway): build the preflight report once

aa4b67d

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman force-pushed the test/5425-port-sandbox-lifecycle/matthewgrossman branch from f8b1ced to a96df3a Compare November 10, 2026 17:40

matthewgrossman force-pushed the test/5426-port-cli-attach-exec/matthewgrossman branch from bbbe043 to 1b45c0b Compare November 10, 2026 18:19

matthewgrossman added 3 commits November 10, 2026 10:31

test(gateway): compile the proxy preflight tests only with a proxy-ta…

84b2f06

…ble driver The gateway feature matrix builds without Docker, Podman or the VM driver, where the test file has no drivers to check. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman force-pushed the test/5426-port-cli-attach-exec/matthewgrossman branch from 1b45c0b to d20313c Compare November 10, 2026 19:07

matthewgrossman and others added 6 commits November 10, 2026 13:26

matthewgrossman force-pushed the test/5426-port-cli-attach-exec/matthewgrossman branch from d20313c to 77794c7 Compare November 12, 2026 15:37

matthewgrossman added 4 commits November 12, 2026 13:06

docs(conformance): point the global settings test at #5527

b5f0737

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman force-pushed the test/5426-port-cli-attach-exec/matthewgrossman branch from 77794c7 to 933f6ea Compare November 12, 2026 22:05

elezar force-pushed the test/5425-port-sandbox-lifecycle/matthewgrossman branch 2 times, most recently from 10e797e to 36d882d Compare November 13, 2026 10:01

Base automatically changed from test/5425-port-sandbox-lifecycle/matthewgrossman to main November 13, 2026 10:51

This was referenced Nov 13, 2026

test(podman): install util-linux-script in the e2e guest #4423

Open

test(conformance): migrate lifecycle main-result tests to lifecycle stories #4628

Closed

Navigation Menu

Repository navigation

test(e2e): port CLI exec, attach, and SSH host identity tests to testkit - #5442

test(e2e): port CLI exec, attach, and SSH host identity tests to testkit#5442
matthewgrossman wants to merge 22 commits into
mainfrom
test/5426-port-cli-attach-exec/matthewgrossman

matthewgrossman commented Nov 10, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Nov 10, 2026

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

matthewgrossman commented Nov 10, 2026

Uh oh!

matthewgrossman commented Nov 10, 2026

Uh oh!

matthewgrossman commented Nov 10, 2026

Uh oh!

matthewgrossman commented Nov 12, 2026

Uh oh!

matthewgrossman commented Nov 12, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

matthewgrossman commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Legacy test → new test

Defaults applied

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Nov 10, 2026

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

matthewgrossman commented Nov 10, 2026

Uh oh!

matthewgrossman commented Nov 10, 2026

Uh oh!

matthewgrossman commented Nov 10, 2026

Uh oh!

matthewgrossman commented Nov 12, 2026

Uh oh!

matthewgrossman commented Nov 12, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

matthewgrossman commented Nov 10, 2026 •

edited

Loading