[matthewgrossman]

Summary

Moves the driver-agnostic sandbox lifecycle and API tests from e2e/rust to testkit so they run on every conformance row, moves the rest to the podman driver suite or crate tests, and deletes the legacy files it replaces. PR1 of the #5297 plan.

Related Issue

Closes #5425
Part of #5281

Changes

• New conformance suites under tests/suites/conformance/src/: image, labels, templates, settings, process, filesystem_policy. New tests in lifecycle.rs and file_transfer.rs. The suite table in tests/suites/conformance/README.md is updated.
• tests/suites/drivers/podman/tests/oci_identity.rs: supervisor container identity and capabilities.
• Crate tests: crates/openshell-cli/tests/doctor_integration.rs (uses a fake docker on PATH) and crates/openshell-gateway/tests/proxy_config_preflight.rs.
• testkit internals: Context, SandboxBuilder, Sandbox and CliSandboxBuilder share one Handles value (clients, CLI, cleanup), so CliSandboxBuilder owns its state like SandboxBuilder rather than borrowing cx. Both builders are #[must_use] types, and their setters take the same argument types.
• Shared test helpers, used by the tests in this PR:
  • cx.cli_sandbox(suffix) → CliSandboxBuilder (.image(), .policy_yaml(yaml), .args(), .command(); .create() detaches and adopts, .run() returns the attached CliOutput). It tracks the sandbox before the CLI runs and passes --no-tty.
  • SandboxBuilder::image(..), so both builders set an image the same way, and create_ready() on both builders: create, then wait for Ready.
  • On Sandbox: poll_exec(what, timeout, command, ready) reruns an exec every 250 ms until ready accepts its output; get_json() is sandbox get --output json; wait_absent(timeout) waits for the gateway to remove the sandbox by itself, then checks get and list.
  • On both ExecOutput and CliOutput: success(), failure() and expect_stdout(expected), through one shared check so their messages match and report stdout and stderr. ExecOutput.description names the exec and its sandbox. CliOutput::report_text() (moved from file_transfer.rs) flattens a wrapped CLI error report for matching.
  • testkit::YamlFile (from test(conformance): port the credential gating e2e tests #5318) replaces this PR's private PolicyFile. CliSandboxBuilder::policy_yaml uses it.
  • testkit::with_deadline(what, timeout, future): a named deadline shorter than a step's own.
  • std::io::Error, serde_json::Error, serde_yml::Error and tonic::Status convert into testkit::Error, so I/O and gRPC results take .context() directly. A converted tonic::Status keeps its code for error.grpc_code(), which replaces Error::from_status. The .map_err(Error::from_error) calls in testkit, tests/support and the features suites are gone.
  • Engine::build_test_image(cx, name, containerfile) in container-engine-support: builds localhost/openshell-test-<name>:<run id> and removes it on cleanup. Engine::build_image is now private, so callers can't skip the cleanup.
• Existing testkit API that changed, so the helpers are uniform: cx.expect_listed(&sandbox, phase) and cx.expect_absent(&sandbox) are now sandbox.expect_listed(phase) and sandbox.expect_absent(). The testkit README lists the current API.
• These helpers replace duplicate code: the PolicyFile copies and hand-written sandbox create/sandbox get calls in host-gateway and oci-image (host-gateway now creates its sandbox with cx.sandbox(..).image(..).policy(..).create_ready() and runs the client with sandbox.exec), the image build/cleanup in oci-image and podman oci_identity, and the create in podman userns. host-gateway drops its tempfile dependency and oci-image drops its serde_json dependency. The proposed CLI-delete cleanup helper was skipped because no test in this PR uses it.
• ExecOutput::success() also replaces the hand-written exit-code checks in the existing smoke, lifecycle, persistent_workspace, file_transfer and policy_advisor conformance tests. Polls and checks that expect a failure still read exit_code.
• The other helpers are used across the conformance suite, existing tests included. credential_gating.rs (from test(conformance): port the credential gating e2e tests #5318) uses create_ready() and poll_exec. templates.rs, settings.rs and labels.rs also get small local helpers (create_template, read_global_setting, listed_names).
• Deleted 10 legacy files and their references: the [[test]] entries, PODMAN_CI_TESTS and TESTING.md. Also removed the now-dead SandboxGuard::create_with_gateway_default.
• Stay in e2e/rust until their batches: edge_tunnel_e2e.rs (with --enable-websocket-tunnel true in e2e/with-{docker,podman}-gateway.sh) and ephemeral_cleanup.rs (on the Podman and VM lanes).
• Stay in e2e/rust until their blockers are fixed: sandbox_lifecycle::on_failure_policy_replaces_runtime_and_preserves_workspace (bug(vm): on-failure restart never completes because the preserved overlay fails e2fsck #5507) and settings_management.rs (test(testkit): separate destructive tests from non-destructive ones #5527).
• Shared files:
  • sandbox_lifecycle.rs keeps its other tests for PR4 (test(e2e): port CLI-driven exec, attach and SSH host identity tests from e2e/rust #5426), which is stacked on this PR. PR4 must keep the file for the on-failure restart test until bug(vm): on-failure restart never completes because the preserved overlay fails e2fsck #5507 is fixed.
  • The incoherent-config tests are removed from docker_corporate_proxy.rs and vm_corporate_proxy.rs. docker_corporate_proxy.rs keeps its proxy_settings_are_inserted_before_nested_driver_tables unit test, because the remaining tests still use the helper it covers. Whichever of PR1/PR6 merges last deletes docker_corporate_proxy.rs. vm_corporate_proxy.rs stays until PR9.

Outcome per legacy test

Legacy test	Outcome
sandbox_lifecycle::sandbox_stop_start_preserves_workspace	Merged: already covered by persistent_workspace_survives_stop_start. "Stop while connect is open" moves to PR4.
sandbox_lifecycle::sandbox_can_be_deleted_while_stopped	Merged: covered by lifecycle::stopped_sandbox_can_be_deleted
sandbox_lifecycle::sandbox_exec_outlives_startup_recovery_deadline	lifecycle::exec_outlives_startup_recovery_deadline
sandbox_lifecycle::detached_canonical_main_{exit_zero_reaches_completed,nonzero_exit_reaches_error}	lifecycle::main_exit_after_ready_sets_final_phase
sandbox_lifecycle::canonical_main_and_exec_receive_declared_environment + canonical_tty_main_uses_sandbox_environment	lifecycle::canonical_main_environment
sandbox_lifecycle::detached_main_exit_during_provisioning_is_classified_as_workload_result	lifecycle::main_exit_during_provisioning_is_workload_result
sandbox_lifecycle::on_failure_policy_replaces_runtime_and_preserves_workspace	Stays in e2e/rust until #5507 (on the VM driver the replacement never starts). The legacy test runs on the Docker and Kubernetes lanes.
ephemeral_cleanup::detached_ephemeral_{success,failure}_removes_sandbox_and_driver_resources	Gateway half added as lifecycle::detached_no_keep_sandbox_is_removed_after_main_exits; the legacy test stays for the driver-resource half
landlock::hard_requirement_accepts_enriched_device_path	filesystem_policy::hard_requirement_accepts_added_device_paths
landlock::{best_effort,hard_requirement}_rejects_replaced_policy_{path,ancestor} (4)	filesystem_policy::replaced_policy_path_rejects_next_launch
core_dump_hardening::sandbox_processes_disable_core_dumps	process::core_dumps_are_disabled
default_image::sandbox_from_default_image	Merged into lifecycle::default_command_sandbox_is_listed_ready
default_image::sandbox_from_explicit_nvidia_ubuntu_image	image::explicit_image_is_used
sandbox_labels::sandbox_labels_are_stored_and_filterable	labels::labels_are_stored_and_filter_list
sandbox_templates::sandbox_create_from_template_uses_reusable_workload	templates::sandbox_from_template_uses_its_workload
sandbox_templates:: get-after-delete, duplicate name, missing template (3)	templates::template_crud_errors
settings_management::settings_global_override_round_trip	Split: the sandbox scope is migrated as settings::sandbox_setting_round_trip; the global override stays in e2e/rust until #5527
upload_backpressure::large_upload_preserves_content_and_metadata_outside_workdir	Split: file_transfer::large_upload_backpressure + the outside-workdir metadata case in file_transfer
podman_oci_identity::podman_supervisor_runs_as_oci_identity_without_capabilities	drivers/podman oci_identity::supervisor_runs_as_oci_identity_without_capabilities
docker_preflight::doctor_check_fails_without_docker + doctor_check_output_shows_docker_label	doctor_check_fails_without_docker_on_path (+ doctor_check_fails_fast_when_docker_info_fails)
docker_preflight::doctor_check_passes_with_docker	doctor_check_passes_when_docker_info_succeeds
docker_preflight::doctor_check_error_includes_guidance	Dropped
docker_corporate_proxy::…rejects_incoherent_configuration + vm_corporate_proxy::…rejects_incoherent_configuration	Dropped → crate test proxy_config_preflight.rs (docker, podman, vm)
docker_corporate_proxy::proxy_settings_are_inserted_before_nested_driver_tables	Stays in docker_corporate_proxy.rs
edge_tunnel_e2e (2 tests)	Stays in e2e/rust until its batch
internet_network_perf::benchmark_complete_internet_path	Dropped (#5472)
live_internet_traffic_perf::benchmark_live_internet_agent_traffic	Dropped (#5472)

Decisions and defaults applied

• The drops follow decision D in test(e2e): decide which e2e/rust tests to port and in what order #5297, and the work ships as one PR.
• large_upload_backpressure runs on every conformance row.
• Conformance tests don't change gateway-wide settings. sandbox_setting_round_trip reads the global value of ocsf_json_enabled and fails with a hint if one is set, because it would hide the sandbox's value. The legacy test cleared it instead.
• The podman OCI identity test runs on all 8 userns rows, rootful included, and passes on each.
• The proxy preflight test checks both config preflight (category=malformed, file unchanged) and real openshell-gateway --config startup, because only startup names the offending keys.
• Landlock tests now run on every row, through exec rather than as the main command with CLI --policy YAML. Their policy sets no process identity, so they run as the image's USER. The default test image has no sandbox user. Rows without Landlock fail hard_requirement, and they should.
• Test deadlines fit inside the runner's 2-minute kill.
• The labels test waits for its sandboxes to be Ready. Deleting a Docker sandbox that is still starting can fail with a 409 "volume is in use" error (bug(docker): deleting a sandbox while it is starting fails with "volume is in use" #5459); a conformance test for that lands with the fix.
• proxy_config_preflight.rs compiles only when the Docker, Podman or VM driver is built in, because the gateway feature matrix also builds without them. The --no-keep test releases both sandboxes before waiting, with a 60 s limit.

What was dropped or moved

• Benchmarks deleted. internet_network_perf and live_internet_traffic_perf were manual, needed the live Internet, and nothing referenced them. test(perf): decide whether to bring back the live-Internet and data-path benchmarks #5472 tracks whether to bring one back.
• Doctor and proxy preflight moved to crate tests. They need no running gateway or sandbox, so they can't be conformance tests. docker_preflight::doctor_check_error_includes_guidance is dropped: the CLI doesn't print DOCKER_HOST when docker info fails. The legacy test passed only because its fake docker printed DOCKER_HOST itself, and the crate test already checks that the CLI shows docker's error.
• Stay in e2e/rust until their batches (test(e2e): decide which e2e/rust tests to port and in what order #5297, test: retire the old e2e test system onto testkit #5281). edge_tunnel_e2e checks the --enable-websocket-tunnel flag on real Docker and Podman gateways, which conformance can't vary. ephemeral_cleanup checks that drivers remove their containers, pods and VM state, which conformance can't see. Until the driver suites replace it, its gateway half also runs in conformance.
• Stay in e2e/rust until their blockers are fixed. The on-failure restart test failed on every VM row when ported, because the VM driver's replacement never starts (bug(vm): on-failure restart never completes because the preserved overlay fails e2fsck #5507). Ignoring it in conformance would have dropped its Docker and Kubernetes coverage, so the legacy test stays until the fix. The global settings override changes a setting for every sandbox on the gateway. Conformance tests don't change gateway-wide state, so it waits for test(testkit): separate destructive tests from non-destructive ones #5527 to decide how destructive tests are marked and where they run.

Testing

• CI: every E2E row is green, including the Docker, Podman (all 8 userns rows), Kubernetes and VM conformance rows. The k3s-ha conformance row needed re-runs for an unrelated flake (bug(e2e): file_transfer::upload_download_round_trip stalls on the k3s-ha conformance row #5520).
• mise run e2e:testkit on a local Docker gateway: the full conformance suite passes (the #[ignore]d bug(sandbox): on Kubernetes, deleting a sandbox waits out the pods' termination grace period #5025 test is skipped, as on main), plus the oci-image and host-gateway feature suites
• cargo test for the new crate tests on macOS: doctor_integration, proxy_config_preflight; proxy_config_preflight also builds under the CI feature matrix (no drivers, Kubernetes only, Docker)
• clippy -D warnings and fmt: openshell-server, -cli, -gateway, the conformance, podman and features suites, tests/support, testkit, e2e/rust
• mise run rust:lockfiles:check
• cargo check on e2e/rust with e2e, e2e-podman, e2e-docker and e2e-vm

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[matthewgrossman]

/ok to test c2eceee

[github-actions]

Label test:e2e applied for c2eceee. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[matthewgrossman]

/ok to test 45180d0

[matthewgrossman]

/ok to test e3b0f27

[matthewgrossman]

/ok to test 73e5bff

[matthewgrossman]

/ok to test caff38c

[matthewgrossman]

/ok to test 2cefa9a

[matthewgrossman]

/ok to test 1ce77a5

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5431.docs.buildwithfern.com/openshell

[matthewgrossman]

/ok to test a96df3a

[matthewgrossman]

/ok to test 034814b

[matthewgrossman]

/ok to test 1ed49ce

[matthewgrossman]

/ok to test 163fa4d

[matthewgrossman]

/ok to test 1cd665d

[matthewgrossman]

/ok to test 139dc5d

[elezar]

@matthewgrossman I pushed one commit on top of your branch (b013446be) that rolls two tests back out of this port:

• on_failure_policy_replaces_runtime_and_preserves_workspace is restored in e2e/rust/tests/sandbox_lifecycle.rs, and its conformance version and #[ignore] are removed. The legacy test ran on the Docker and Kubernetes lanes (never VM or Podman), so ignoring the port dropped that coverage. It moves to conformance once bug(vm): on-failure restart never completes because the preserved overlay fails e2fsck #5507 is fixed.
• settings_management.rs is restored unchanged, and global_setting_overrides_sandbox_scope is removed from conformance. It changes a gateway-wide setting, so it needs a serial test group (test(runner): apply per-family nextest config and a thread count to runner rows #5295) first. sandbox_setting_round_trip stays in conformance.

The README rows for lifecycle and settings are updated to match.

Please update the PR description:

• In the outcome table, change the rows for sandbox_lifecycle::on_failure_policy_replaces_runtime_and_preserves_workspace and settings_management::settings_global_override_round_trip. The first stays in e2e/rust until bug(vm): on-failure restart never completes because the preserved overlay fails e2fsck #5507. The second is split: the sandbox scope is migrated, and the global override stays in e2e/rust until test(runner): apply per-family nextest config and a thread count to runner rows #5295.
• Remove the paragraph about on_failure_policy_restarts_main_and_keeps_workspace being #[ignore]d, and the note that the global settings test depends on serial execution.
• Count settings_management.rs and the on-failure test among the files that stay, not the deleted legacy files.

Your local branch now has a new commit on top, so git pull is enough; no reset is needed this time.

[elezar]

Re-approving after the rebase onto current main (10e797e2b). The only conflict was in e2e/rust/Cargo.toml: I kept main's new supervisor_middleware_http_hooks test entry and dropped the sandbox_templates entry that this PR deletes. The rest of the diff is unchanged. cargo check passes on e2e/rust and the conformance suite. CI still needs to run on the new head.