bug: stop sandbox times out in Kubernetes E2E on Agent Sandbox v1alpha1 #5457

User Story

I contribute tests and CI changes to OpenShell and run its Kubernetes E2E checks on my pull requests. I directly encountered a stop request on a Kubernetes sandbox timing out in an unrelated PR, and I need to know whether stopping a sandbox on the Agent Sandbox v1alpha1 backend is reliable so that a red Kubernetes check can be trusted as a real signal.

Problem Statement

In the Kubernetes E2E (Rust smoke) job on Agent Sandbox v1alpha1 (v0.4.6), sandbox_lifecycle::sandbox_can_be_deleted_while_stopped failed after 50 seconds. The gateway returned Internal error with stop sandbox failed: ServiceError: request timed out. 104 of the job's 105 tests passed, and the same Rust smoke on Agent Sandbox v1beta1 (v1.0.3) passed in the same workflow run. The failing run was on a pull request that changes no gateway, driver, SDK or Kubernetes code (it only adds tests and test environments for CDI on Docker and Podman).

I have seen it once. I don't know whether it is a flaky timeout in the stop path, a slow CI cluster, or a real defect.

Impact / Why This Matters

A failure here fails the aggregate Core E2E result check, which blocks merging until the job is re-run. If the stop path is intermittently slow, contributors cannot tell an unrelated red check from a regression in their own change, and each occurrence costs a full re-run of a roughly 15 minute job.

Acceptance Criteria

• The cause of the stop sandbox timeout on Agent Sandbox v1alpha1 is identified (stop path, request timeout, or CI capacity).
• Either the stop request completes reliably within the test's time budget, or the test's timeout is justified against the behavior of the stop path.
• Repeated runs of sandbox_can_be_deleted_while_stopped on v1alpha1 no longer fail with request timed out.

Reproduction Steps

I could not reproduce it deterministically. It was observed once in CI:

1. Run the Kubernetes E2E (Rust smoke) job for Agent Sandbox v1alpha1 v0.4.6 (kubernetes-e2e).
2. The test openshell-e2e::sandbox_lifecycle sandbox_can_be_deleted_while_stopped stops a sandbox and then deletes it.
3. The stop request timed out at 50 seconds.

Failing job: https://github.com/NVIDIA/OpenShell/actions/runs/42849338261/job/129952498006

Environment

• OpenShell: pull request pull-request/4461 at e7776d2ab, based on main at f3d976590
• Deployment: Kubernetes E2E in CI, Agent Sandbox v1alpha1 (v0.4.6)
• Passing in the same run: Agent Sandbox v1beta1 (v1.0.3) Rust smoke and the other Kubernetes jobs

Logs

FAIL [  50.014s] ( 99/105) openshell-e2e::sandbox_lifecycle sandbox_can_be_deleted_while_stopped
thread 'sandbox_can_be_deleted_while_stopped' panicked at tests/sandbox_lifecycle.rs:405:5:
sandbox stop should succeed (exit Some(1)):
Error:   × code: 'Internal error', message: "stop sandbox failed: ServiceError:
  │ request timed out"
Summary [ 250.680s] 105 tests run: 104 passed (3 slow), 1 failed, 5 skipped