[matthewgrossman]

Summary

Port the six credential gating e2e tests from #5264 into the conformance family, and remove the old e2e/rust test. This is the pilot port for #5296. The pieces the port needed that other tests will need too are shared: policy helpers and provider install in testkit, and a new tests/support/host-server crate for servers on the test host. The existing host-gateway feature test now uses them as well.

Related Issue

Closes #5317

Part of #5281. Pilot for #5296, which stays open. Ports the tests from #5264.

What the tests check, and what they needed

The #5264 tests check what the sandbox supervisor does with a provider credential when the sandbox sends traffic out:

• The gateway rejects a policy that binds a credential to an endpoint it can't inspect (L4-only or tls: skip), unless the endpoint opts in.
• A REST body sent upstream carries the credential placeholder, or the real secret when request_body_credential_rewrite is on.
• A WebSocket binary frame, which can't be inspected for the credential, isn't forwarded.
• Binding a provider with policy set adds its environment variable to a running sandbox, and unbinding removes it.

Porting them needed four things. Each was added as a generic helper, and host-gateway, an existing feature test that had its own copies, now uses the shared ones.

1. An upstream server that records what arrived. Most of the assertions are about what reached upstream, so the test needs a server that the sandbox can reach and that records what it received. New crate tests/support/host-server (host-server-support):
   • HttpRecorder::start(body) answers every request with 200 and body. wait_for_requests(n) returns the requests, head and body apart, once n have arrived. It reads Content-Length bodies, not chunked ones.
   • WebSocketRecorder::start() completes the WebSocket handshake. It then reads frames until a close frame, the end of the connection, or a 10s timeout. wait_for_handshake() waits for the handshake, and wait_until_finished() waits for the connection to end and reports whether a binary frame arrived.
   • Both listen on the test host, which sandboxes reach as host.openshell.internal, and stop when dropped. It lives in tests/support/, alongside the Keycloak fixture, because it's a service outside OpenShell that any suite can use. host-gateway replaces its hand-written server with HttpRecorder.
2. Policies with credential settings on network endpoints. Each test needs a slightly different endpoint (protocol: rest or websocket, tls: skip, body rewriting, credential_binding, allow_uninspected_credentials). Before this PR, tests wrote out a whole policy as a YAML string. New testkit module testkit::policy:
   • base() reads base-policy.yaml: the filesystem and process settings every sandbox needs, and no network access.
   • allow(&mut policy, name, host, port, binaries) adds an endpoint. Its Endpoint setters are named after the YAML fields, e.g. .rest(Access::Full).credential_binding(&provider).
   • to_yaml() renders the policy for CLI commands such as policy set.
   • A unit test checks that each helper builds exactly what its YAML parses to.
   • PYTHON_BINARIES names the paths python3 runs from in testkit::python_image() images, which differ between the workload image and the pullable uv image.
   • The sandbox builder can now combine a policy with an image and providers. Before, cx.sandbox(..).policy(..) rejected both, and these tests need all three.
3. A provider per test, cleaned up afterwards. Each test imports its own provider profile and creates a provider under a name unique to the run. New cx.import_profile(profile_yaml) imports a profile and returns its id, read from the YAML, and cx.create_provider(name, type, credentials) creates a provider. Each registers its cleanup before creating. cx.install_provider(name, profile_yaml, credentials) does both.
4. Small CLI and exec plumbing. testkit::YamlFile writes YAML to a temporary file for CLI commands that read one (profile import, policy set); host-gateway replaces its PolicyFile copy with it. ExecOutput::success() fails a test on a non-zero exec exit, like CliOutput::success().

Helpers for OpenShell itself (sandboxes, policies, providers, the CLI) go in testkit. Fixtures for services outside OpenShell go in tests/support/.

Changes

• conformance: add credential_gating, one file with the same six scenarios and unique names as test(e2e): split credential_gating into independent parallel tests #5264, built on the helpers above.
  • The tests assert on the request that reached the host server, not on verdict strings printed by the sandbox client. The REST client is a short urllib POST.
  • Dropped a client-side check: the WebSocket test no longer requires the client to see close code 1008. The original client counted a read timeout as "denied" too, so the check passed whether or not the supervisor closed the connection. The test now requires that the handshake reaches upstream, waits until upstream's connection has ended, and requires that no binary frame arrived on it.
  • The REST tests compare the body upstream exactly (the secret, or the placeholder between the client's prefix and suffix), and check that the secret appears nowhere in the request. Failure messages print the request with the secret redacted.
  • Dropped the "secret reached the sandbox" check, which could never fail.
  • The gateway's rejections are checked through the builder's create() error: the gRPC code must be FailedPrecondition, and the message keeps test(e2e): split credential_gating into independent parallel tests #5264's wording checks. testkit's Error now keeps the gateway's status code (grpc_code(), testkit::GrpcCode). The CLI is used for profile import, provider create and policy set --wait.
• test image: testkit::python_image() still defaults to openshell/e2e-python:dev. The Docker and Podman runners now build it for conformance, as the tmachine installers already load it. The Kubernetes and VM runners set OPENSHELL_TEST_PYTHON_IMAGE to a pinned public uv Python image unless it is already set. A gateway outside these lanes has to set OPENSHELL_TEST_PYTHON_IMAGE; running conformance against any gateway without an override is a follow-up. host_gateway runs python3 from PATH and allows PYTHON_BINARIES, so it works with either image, and no longer retries its connection for 20s: the credential tests never did, and the server listens before the sandbox is created.
• external Kubernetes driver lane: the Helm post-renderer now copies the chart's server.hostGatewayIP into the standalone driver's OPENSHELL_HOST_GATEWAY_IP. Before, only the in-process driver got it, so sandboxes on that lane couldn't resolve host.openshell.internal. The lane hadn't run a host-reachability test before. The renderer fails if the lane set the IP but the chart rendered no host.openshell.internal alias.
• removal: delete e2e/rust/tests/credential_gating.rs, its [[test]] entry, and its PODMAN_CI_TESTS entry.
• docs: the testkit README lists the new helpers. The conformance README and TESTING.md list the suite and state that conformance requires the machine running it to be reachable from sandboxes as host.openshell.internal.

Gaps found by the pilot (for #5296)

• Step logging: there is no per-step progress output. A slow step shows up only as the test's total time.
• Sandbox names: Podman caps sandbox names at 19 characters, which leaves 5 for cx.sandbox(suffix). testkit doesn't check this before sending the request.
• Error-text assertions: the admission test matches phrases in the gateway's error ("credentialed endpoint", "L4-only", "tls:"/"skip"), as test(e2e): split credential_gating into independent parallel tests #5264 does. That is stricter than the conformance README's guidance on asserting behavior over wording.
• Remote clusters: host servers run on the test host, so remote Kubernetes or OpenShift gateways can't reach them.
• Host lanes on tmachine: the Docker and Podman host lanes still run through e2e/rust/e2e-*.sh scripts, not tmachine.

Testing

• cargo fmt and cargo clippy --locked --all-targets -D warnings on testkit, support, features and conformance.

• testkit unit tests: 43 passed.

• check_cargo_lockfile_alignment.py and check-conformance-program.sh (22 tests) pass.

• Local Podman (macOS, e2e/with-podman-gateway.sh): openshell-conformance run credential_gating passed 6/6 in 33.7s. host_gateway passed.

  Test	Time
  policy_binding_requires_inspection_or_opt_in	2.1s
  policy_binding_rest_body_keeps_placeholder_without_rewrite	2.7s
  policy_binding_rest_body_rewrite_sends_secret_upstream	1.9s
  policy_binding_updates_provider_environment_live	22.4s
  profile_rest_endpoint_keeps_body_placeholder	2.8s
  profile_websocket_endpoint_denies_binary_frames	1.8s

• Not run locally: Docker (Docker Desktop requires sign-in), k3s, k3s-ha. CI covers these rows.

CI: copy-pr-bot doesn't sync draft PRs, so E2E runs only after a maintainer comments /ok to test <sha>. In #5264, the WebSocket test took 60–120s on Kubernetes. If k3s or k3s-ha hit the per-test timeout, I'll raise it here rather than change .config/nextest.toml (see #5295).

• Checks appropriate to the affected code and behavior pass
• Unit tests added/updated (if applicable)
• E2E tests added/updated (if applicable)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

Label test:e2e applied, but pull-request/5318 does not exist yet. A maintainer needs to comment /ok to test 1472ee7b123f994b562336888e2bb2fea87bb91d to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5318.docs.buildwithfern.com/openshell

[matthewgrossman]

/ok to test 566b39d

[matthewgrossman]

/ok to test 99bbed4

[github-actions]

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. cli — No explicit risk acceptance found

OPENSHELL_E2E_KUBERNETES_FEATURES=e2e,e2e-kubernetes mise run e2e:kubernetes previously excluded host-dependent credential tests but now runs them unconditionally through openshell-conformance.

• Impact: Remote Kubernetes/OpenShift operators and driver authors encounter failed connections to host.openshell.internal and conformance exit 1 even when their gateway satisfies the existing contract.
• Action: For v0.1.8, preserve the existing host-gateway opt-in or move these tests to a feature suite; the new warning does not address this patch-train break.
• PR: test(conformance): port the credential gating e2e tests #5318

Review details

• Execution: complete
• Candidate: PR #5318 (c29689d502b2329fb012721ec158f81b7e83c8e1)
• Baseline: cc4acdcfdeda9fdeeb0fe94a537d47ad8414eb3d (cc4acdcfdeda9fdeeb0fe94a537d47ad8414eb3d)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

[matthewgrossman]

/ok to test 61504a3

[matthewgrossman]

/ok to test 3a31220

[elezar]

Thanks for the port. The recorder-based assertions are a clear improvement on the old verdict strings, and keeping all six scenarios while moving them onto shared helpers is the right shape for the pilot.

I'd like two things changed before this merges, plus some smaller suggestions inline:

1. Drop 61504a338 (the default Python image change) and handle "conformance runs on any gateway" in a follow-up.
2. Give the WebSocket test a positive control, so the denial assertion can fail.

Remote-cluster behaviour (inline comment on TESTING.md) is fine as a tracked follow-up.

[matthewgrossman]

/ok to test 9ab1be1

[github-actions]

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. cli — Risk explicitly accepted

OPENSHELL_E2E_KUBERNETES_FEATURES=e2e,e2e-kubernetes mise run e2e:kubernetes previously excluded host-dependent credential tests but now runs them unconditionally through openshell-conformance run --junit <path>.

• Impact: Remote Kubernetes/OpenShift operators and driver authors now encounter failed connections to host.openshell.internal and conformance exit 1 despite satisfying the previously supported gateway contract.

• Action: The earlier host-gateway opt-in or feature-suite remediation remains necessary for v0.1.8; the documented restriction and accepted follow-up do not resolve this patch-train violation.

• PR: test(conformance): port the credential gating e2e tests #5318

• Acceptance: comment

Review details

• Execution: complete
• Candidate: PR #5318 (328cde47374d56cd8cbd425adc3dd8a2ad2d210c)
• Baseline: cc4acdcfdeda9fdeeb0fe94a537d47ad8414eb3d (cc4acdcfdeda9fdeeb0fe94a537d47ad8414eb3d)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

[matthewgrossman]

/ok to test 4ff621b

[matthewgrossman]

/ok to test e5e72bf

[matthewgrossman]

/ok to test e22087f

[matthewgrossman]

/ok to test aeec397

[matthewgrossman]

/ok to test 6ebaf4d

[matthewgrossman]

/ok to test 16e7f7e

[elezar]

Approving. Both of my requested changes are addressed (the default image is back to openshell/e2e-python:dev, and the WebSocket test has a positive control), and the six credential-gating tests passed on the Kubernetes workspace-managed, Kubernetes external-driver, VM and k3s-ha lanes in run 42763840495.

Remote clusters are still a known gap, tracked in #5409, where I added a proposal to move the four host-dependent tests to a features suite.

Nits, non-blocking:

• Use Error::from_status at the remaining from_error(&status) sites in context.rs (:217, :559, :576, :594), so grpc_code() is kept wherever a tonic::Status is mapped.
• HttpRecorder has no unit tests. A few covering Content-Length bodies and the chunked case would match what the WebSocket recorder has.