bug(kubernetes): a ResourceQuota or LimitRange rejecting the workspace PVC leaves the sandbox in Provisioning for 300 s with an unrelated message #5491

User Story

Operators of shared Kubernetes clusters commonly put a ResourceQuota or LimitRange on the namespaces where sandboxes run. When one of them blocks the sandbox's workspace PVC, the person running openshell sandbox create waits five minutes and gets an error that does not mention storage. I found this by reading the Kubernetes driver and reproduced it on a local kind cluster; I have not seen it on a production cluster.

Problem Statement

Every sandbox gets a workspace PVC, created by the Agent Sandbox controller from the driver's volumeClaimTemplates. When a ResourceQuota or LimitRange rejects that PVC, openshell sandbox create waits the full provisioning timeout (300 s) and then fails with:

sandbox provisioning timed out after 300s. Last reported status:
ConfigurationPending: Waiting for effective configuration validation
before workload activation

The sandbox phase stays Provisioning the whole time, and the message says nothing about storage.

Where the cause is lost, from reading the code:

• is_terminal_failure_reason treats ReconcilerError as transient.
• While a runtime generation bootstraps, mark_sandbox_runtime_control_unavailable overwrites Ready with a generic dependency message.
• apply_configuration_readiness then overwrites it with ConfigurationPending.

Impact / Why This Matters

The cause is known to Kubernetes but hidden by OpenShell, so users cannot tell what to fix. The gateway's own status line points at the wrong thing (persistentvolumeclaim "workspace-..." not found; the claim is missing because it was rejected). Finding the real cause takes kubectl describe on the Sandbox. The failure is slow (5 minutes per attempt) and repeats for every create until the quota or limit is fixed.

Acceptance Criteria

• A persistent ResourceQuota or LimitRange rejection of the workspace PVC puts the sandbox in Error within about 30 s, and the CLI error names the quota or limit.
• An RBAC or namespace-terminating 403 is not reclassified as terminal.

Reproduction Steps

1. Run a gateway with the Kubernetes driver's default settings. That means workspace_mode = "shared" (every sandbox goes into the one configured namespace) and the default workspace PVC size of 2Gi (workspace_default_storage_size).
2. In that namespace, create a ResourceQuota with requests.storage: 1Gi.
3. Run openshell sandbox create --name s1 --no-keep -- true. After 300 s it fails with the message above.
4. The Sandbox CR shows the real cause: Ready=False, ReconcilerError: Error seen: persistentvolumeclaims "workspace-default--s1" is forbidden: exceeded quota: q, requested: requests.storage=2Gi, used: requests.storage=0, limited: requests.storage=1Gi.
5. The same happens with a ResourceQuota of persistentvolumeclaims: "0" and with a LimitRange of type PersistentVolumeClaim with max: storage: 1Gi.

I only ran shared mode. managed and operator modes share the same status path (the cluster-wide watcher calls the same functions), so I expect them to behave the same, with the quota set on the per-workspace namespace instead. That is from reading the code and was not run.

Related work

These are not duplicates of this issue; I list them to avoid overlap.

• openshell sandbox create timeout reports ConfigurationPending when the pod could not be scheduled #5230 and draft PR fix(cli): report scheduling failures in provisioning timeouts #5344 change the CLI timeout text to report a pod scheduling failure instead of ConfigurationPending. PR feat(kubernetes): surface pod scheduling status #4907 surfaces PodScheduled=False and FailedScheduling events. From their descriptions they report the scheduler's PodScheduled message. In the quota case that message is persistentvolumeclaim "workspace-..." not found, which is a symptom; the quota text is in the Ready condition. I have not run either PR.
• PR feat(driver-kubernetes): add empty_dir workspace volume with bounded size #4473 adds workspace_max_storage_size and a per-sandbox workspace size through driver_config. That is a gateway-side size cap. This issue is about quotas and limits set in the namespace by someone else.

Environment

• OpenShell: 0.1.9-dev.10+gf1ce69307 (built from f1ce69307, origin/main)
• OS: Fedora 44 (CLI host)
• Deployment: gateway as a pod on a local kind v0.27 cluster (Kubernetes v1.32.2), Agent Sandbox controller v1.0.10, Kubernetes compute driver, default settings

Logs

Error:   x sandbox provisioning timed out after 300s. Last reported status:
  | ConfigurationPending: Waiting for effective configuration validation
  | before workload activation