Repository navigation
Add OpenShell backend for sandbox tools - #73961
Merged
Merged
Conversation
zozo123
requested review from
Lee-W,
ashb,
gopidesupavan and
kaxil
as code owners
September 30, 2026 15:10
Contributor
|
Quickest fix: git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-leaseAutomated nudge — ignore if you're not ready to rebase. This comment is updated in place on future |
Add OpenShellSandboxBackend, a self-hosted backend that runs SandboxToolset sandboxes through an NVIDIA OpenShell gateway on Docker, Podman or Kubernetes. OpenShell confines each sandbox with Landlock and seccomp, gives it no network interface of its own and mediates egress in a per-sandbox supervisor, so the toolset's deny-all default and a hostname allowlist on port 443 are enforced by the runtime rather than approximated. The backend keeps the fail-closed contract itself instead of trusting the gateway. It reads the effective policy back after create, and before and after every command, and destroys the sandbox when the policy differs from the request, comes from a gateway-wide override, or can be widened by auto-approved proposals. It also works around gaps measured in the 0.1.2 SDK and gateway: an exec timeout that leaves the command running (a guest wrapper kills the command's session), client-side buffering of all output (events are streamed into per-stream caps), the 32 KiB argument limit (commands and file content go over stdin), the 19-character name limit, proxy and CA variables the supervisor silently rewrites (refused), and gateway restarts (reported as an unknown outcome, never as a success). Credentials are ambient, read from the OpenShell CLI's gateway registration, so no connection type is added. The extra states protobuf and grpcio floors that the SDK wheel under-declares, and the root pyproject carries a temporary exclude-newer override for the two-day-old release.
zozo123
force-pushed
the
add-openshell-sandbox-backend
branch
from
October 2, 2026 11:18
55126b4 to
ea743cd
Compare
zozo123
force-pushed
the
add-openshell-sandbox-backend
branch
from
October 3, 2026 17:38
6330b1f to
f48eb1b
Compare
openshell 0.1.2 is now outside the rolling 4-day window, so the per-package cutoff the pin comment scheduled for removal on 2026-10-03 is no longer needed.
Contributor
Author
|
Ran the OpenShell boundary checks on a Boat machine. Docker Desktop on macOS cannot. Its kernel is built without Landlock, and the supervisor exits before a sandbox is ready. Boat sandbox The eight steps in Drafted-by: Cursor agent |
Keep the rewritten sandbox docs and document OpenShell inside them.
zozo123
force-pushed
the
add-openshell-sandbox-backend
branch
from
October 6, 2026 16:12
e08fba7 to
ca6e80d
Compare
A single output line longer than the cap came back as no output at all, because the leading partial line was always dropped; it is now kept unless that would discard more than half the window, as in the Modal backend. A gateway drop during the not-ready retry escaped run_command as a private exception. The client lock is module level so the backend deep-copies like Modal's, results report applied_timeout, and the write_file docstring states the real reason it chunks.
Without it the bullet list above runs into the paragraph and the docs build fails on a warning treated as an error.
zozo123
force-pushed
the
add-openshell-sandbox-backend
branch
from
October 6, 2026 18:54
0a06661 to
3eef0b0
Compare
The guest wrapper's timer counted iterations of `sleep 1`, and each iteration forks under the sandbox's CPU quota, so on a busy sandbox the delay grew with the budget. A 120 s budget fired after 147 s with 32 busy processes, and a 300 s one after 367 s, past the gRPC deadline of budget plus 30 s; the backend then took the hang for a stalled gateway and destroyed the sandbox and its files over an ordinary timeout. The timer now fires once /proc/uptime has advanced by the budget. It is monotonic, has centisecond resolution so the timer never fires early, and is read with a shell builtin, so a starved timer is at most one sleep late instead of one per second of budget.
The wrapper ignored a failure to write the command to /tmp. With /tmp nearly full it ran the truncated prefix that did fit, and with /tmp full it ran an empty file; either way it reported that run's status, usually 0, so the model was told a command succeeded that never ran in full. A failed mktemp or cat now removes the staging directory and exits 125 with a fixed stderr line, and run_command raises a recoverable SandboxError for that pair. The stderr line is needed as well as the status because commands such as docker run, env and nohup exit 125 on their own errors, and those still reach the model as exit codes.
Three behaviours of the OpenShell backend had no test that failed without them: the policy read waiting out a restarting gateway, the refusal of a policy with network middlewares, and a hung file operation being a recoverable error that keeps the sandbox. Disabling any of them left the suite green, and the only test that noticed a missing retry did so by counting time.monotonic calls. The new tests cover the retry on UNAVAILABLE and DEADLINE_EXCEEDED, its bound at the recovery window, the middleware refusal and the hung read, and each fails when its behaviour is removed.
The rewrap of the teardown paragraph in the sandbox overview had dropped the blank line before the "Why not KubernetesPodOperator or the KubernetesExecutor?" title, so docutils folded the title into the paragraph without a warning and the section vanished from the page. The paragraph is back to its original wrapping with the blank line. On the backends page, the Modal intro still counted two backends that run on Kubernetes, and the OpenShell credentials paragraph gave a false reason for having no connection (Airflow does mask extras whose keys look sensitive) and compared the setup to ~/.modal.toml, which Modal now only falls back to. The paragraph now says where the credentials live and that the SDK reads them itself. The timeout sweep was described as rescanning until no process is left, in the docs, the backend docstring and the wrapper comment. It stops at the first pass that kills nothing, after at most 50 passes, so a command that keeps forking can outlast it. The docs also say that a command that cannot be staged in /tmp is reported and not run.
The timeout sweep forked a `cat` for every /proc entry, under the same CPU quota as the processes it was killing, so its cost grew faster than linearly with their number. With a 5 s budget and 128 busy processes on one CPU the wrapper returned 30 to 98 s after the budget, past the gRPC deadline of budget plus 30 s, and the backend then destroyed the sandbox over an ordinary timeout. The sweep now reads each stat line with the `read` builtin and strips the process name with the shortest match, falling back to the longest match only when a name contains a parenthesis; on a 300-byte stat line the longest match alone cost as much as the read. Measured in python:3.12-slim with --cpus=1 and a reaping init, as the sandbox has: 128 busy processes and a 5 s budget now return 1.5 to 3.6 s after the budget, and 256 with a 10 s budget 7 to 22 s, against 142 s before. The starved-timer test bounded the time after SIGCONT at 1.5 s, which also covered the forking sweep and failed with 300 processes in the namespace. It now stops the wrapper for its whole 6 s budget, so a timer that counts its own sleeps would be 5 s late, and allows 3 s; with 600 processes and 8 busy loops on 2 CPUs it measured at most 1.1 s. A new test covers a process whose name contains ") ".
The error for a command that could not be staged in /tmp told the model to free space by overwriting a large file there with write_file. That does not hold in the cases that matter: list_directory goes through the same wrapper and fails the same way, so the model cannot find the file; the space can be held by an unlinked spool file that a background process from an earlier command is still writing; and /tmp usually shares a filesystem with the workdir. The message also named /tmp twice. It now says the command was not run and gives the cause the shell printed, such as "No space left on device", and nothing else. The failure was recognised by searching stderr as capped at max_output_bytes, so under a cap shorter than the 50-byte staging line it came back as a plain exit 125. The stderr buffer now keeps at least that line's length for the check, while the text and the truncation flag still follow the cap, and the check requires stderr to end with the line. The staging test is split and parametrized so that the staging line with another exit status, and a cap below the line's length, are each covered by a case that fails without the matching half of the check.
Shortening the sweep sentence left "that keeps forking faster than the sweep. If the gateway" as a short line in the middle of the paragraph. The rest of the paragraph is rewrapped; no wording changes.
The module's autouse fixture patches time.sleep through the backend module, whose `time` is the global module, so every sleep in TestRunWrapper returned at once. The CPU-starvation test then never held the wrapper stopped, the wrapper ran its whole 6 s budget after the resume, and the test failed on Linux in the full suite; the settling waits in the two timeout tests did nothing either. The figure of at most 1.1 s in the earlier sweep commit came from a copy of the class that left the fixture out, so it does not describe this suite. TestRunWrapper now overrides the fixture with one that leaves sleep alone. In the Airflow CI image (python3.12) the whole file passes, and the starvation test sees the wrapper exit 0.02 to 0.04 s after the resume; with the old timer that counts its own sleeps it is 5.1 s and the test fails.
The staging check was tightened to require stderr to end with the wrapper's staging line, but no test failed when the line only appeared somewhere in stderr. A command can print that text and keep going, and exit 125 on its own, so a status of 125 with the line followed by more output has to stay the command's own result. That case is now in the parametrized test, which fails if the check looks for the line anywhere in stderr, and the test is renamed to match what its cases share.
The timeout sweep read only the first line of each /proc/<pid>/stat. The kernel writes the process name there unescaped, so a process whose name contains a newline, because it was run through such a file name or renamed with prctl, left its session field on a line the sweep never read, and it survived the timeout. The `cat` the sweep used before read the whole file and killed it. The sweep now joins every line of the file with the `read` builtin, so it still forks nothing per entry. The wrapper test for a name with a parenthesis is parametrized with a newline name, which survives the single-line read. In python:3.12-slim with --cpus=1 and an init, 128 busy processes and a 5 s budget returned 1.4 to 2.6 s after the budget, against 2.9 to 4.6 s with the single-line read, so the extra read per entry costs nothing measurable.
Main now targets Python 3.11, so ruff's UP017 asks for datetime.UTC instead of timezone.utc and fails the static checks.
Without setsid every command came back as exit 125 with a stderr line the model read as its own failure, so it retried a command it could never fix from inside a sandbox with no network. The wrapper now reports the missing binary with its own status and a private last stderr line, the same way it reports a staging failure, and run_command turns that into a terminal error naming the image requirement.
The SDK refreshes a stale OIDC token inside the call's auth interceptor and writes it back next to the gateway registration. On a registration mounted read-only, as a Kubernetes Secret or ConfigMap is, that write raises an OSError out of every call, which the backend reported as an opaque failure of whatever it was doing. Turning off the write-back would break refresh token rotation across workers, so the backend now recognises the failed write-back and fails the task with what to change, and the docs say that an OIDC registration must be writable and per worker and that mTLS is the supported setup for a read-only one.
The SDK ships .pyi stubs for its generated messages, but the backend typed the sandbox config, the sandbox, the spec and the workspace selector as Any, so mypy could not see a misspelled field on any of them. They now carry the stub types, imported for type checking only.
read_file and write_file pass the path as a command argument, and the gateway rejects an argument with a NUL in it or over 32 KiB of UTF-8 with INVALID_ARGUMENT, which the backend reports as terminal. A bad path is the model's to correct, so both operations now refuse it as a recoverable error before anything is sent. list_directory sends its path on stdin and is not affected.
The docs said the sandbox is destroyed when the gateway stops relaying a command. The backend only asks that same gateway to delete it and logs a failed delete, so the sandbox can survive and is left for the operator's reaper. Both the commands section and the portability note now say so.
The filesystem-policy test compared the read-write paths with the constant they come from and never looked at the read-only paths or include_workdir, so widening the policy left it green. It now spells out every path. The gateway-error test did not count exec calls, so retrying on any terminal status, rather than only on a sandbox that is not ready, also passed it.
The exit-code step only looked for the exit status, so a backend that lost stderr passed it, and the shell step only looked for 42, not the marker the first step wrote. Each step can now require several strings, and these two check what the OpenSandbox system test checks.
Only a failed write of the SDK's .oidc_token.* temp file means the registration is read-only. Nothing kept the mapping from catching any OSError, which would tell an operator with an unreadable mTLS key or a full disk to switch to mTLS. These cases now must keep the generic message.
The wrapper reported a missing setsid as status 127 with a fixed last stderr line, so a command that printed that line and exited 127 was indistinguishable from it and failed the task; a faked staging line likewise passed for an unrun command. run_command now sends a per-call nonce as the first stdin line, the wrapper reads it with the read builtin and appends it to both lines, and only a line carrying that nonce counts. The nonce stays out of the command's argv, environment and staged script, so printing the fixed text is no longer enough.
The refused-path test only covered paths over the limit, so an off-by-one or a stricter threshold still passed: the error message prints the constant, not the threshold that was checked. A path of exactly 32 KiB of UTF-8, and one with control characters but no NUL, must now reach the gateway for both read_file and write_file.
A full disk or a used-up quota fails the SDK's .oidc_token temp file on a writable registration too, and telling the operator to switch to mTLS for those would send them the wrong way.
Contributor
Author
|
Status at
Drafted-by: Claude Code (Opus 5.5) (no human review before posting) |
# Conflicts: # providers/common/ai/README.rst
A live run against an OpenShell 0.1.2 gateway on the Docker driver showed the workload keeps a loopback interface, and that a restarted gateway does not always bring a sandbox back, in which case the task fails. Only the Docker driver has been exercised, and on Kubernetes the cluster's CNI has to enforce NetworkPolicy for egress to stay behind the supervisor.
The live run measured a loopback-only interface on OpenShell's Docker driver. On Kubernetes the Agent Sandbox controller creates a separate workload Pod, so the claim is not known to hold there.
kaxil
reviewed
Oct 13, 2026
# Conflicts: # providers/common/ai/docs/sandbox/backends.rst # providers/common/ai/tests/system/common/ai/README.md
Three failure paths told the model something stronger than the backend knew. The gateway answers DEADLINE_EXCEEDED after about 10 seconds when its relay into the sandbox does not open, before the command starts, and that was taken for a hung command, so a healthy sandbox was deleted and the model saw a timeout for a command that never ran. A dropped connection to a gateway that stays up leaves the command running until the wrapper's timer fires, yet the model was told it had been stopped and could run it again. And a widened sandbox whose delete failed was still reported as destroyed, with nothing logged for the operator who has to reap it, since OpenShell has no server-side lifetime. The output-cap test now checks the buffer itself, because the returned text is sliced to the cap anyway and passed with the trim removed. Claude-Session: https://claude.ai/code/session_01W3CHee8ErP7UArBK3SKZUa
If the wrapper's kill on timeout broke, the backend would abandon the command and destroy the sandbox, and the next step would count sleepers in the replacement, where there are none, so the system test passed either way. Reading the file written at the start of the run in the same command shows the count comes from the sandbox that timed out. Claude-Session: https://claude.ai/code/session_01W3CHee8ErP7UArBK3SKZUa
The backend has only been verified on OpenShell's Docker driver, which OpenShell itself scopes to local development and single-machine gateways, but the overview listed it with Modal and OpenSandbox as a production and Kubernetes backend, contradicting the backend page. Claude-Session: https://claude.ai/code/session_01W3CHee8ErP7UArBK3SKZUa
An early DEADLINE_EXCEEDED can also come from a proxy capping stream duration after the command started, so only the gateway's own relay-open timeout is reported as not run; anything else is reported as possibly still running.
… variable names read_file and write_file now run the same egress check as run_command before and after the operation, and refuse a named pipe instead of blocking on it with nothing to end the wait. The command wrapper's shell variables are renamed under an _airflow_ prefix that SandboxSpec.env may not use, so an env key can no longer share a name with the variable holding the per-call nonce and receive it. list_directory fails the task when the sandbox was destroyed during the listing and reports a timed-out listing as recoverable. write_file copies one chunk at a time, and an abandoned command logs why its sandbox is being destroyed. Docs stop recommending OpenShell for Kubernetes, cover it in the network configuration section, and say when to pick it over OpenSandbox. Tests cover the /proc rescan deterministically and bind the wait before a not-ready retry.
…eview gaps BusyBox ash, the /bin/sh of Alpine images, never finishes the wrapper's timeout sweep, so the docs and docstring now ask for dash or bash. The policy re-check is described as running after each operation that completes, which is what the code does, and a test covers a policy widened during a file operation. A test checks every variable the wrapper assigns carries the reserved prefix. A listing the gateway stopped relaying no longer claims its sandbox was destroyed, and the docs drop the Linux-only gateway host claim and say a reaper is only needed when the delete itself fails.
The policy re-check runs after operations that succeed, not every one that ends, a sandbox left behind by a BusyBox timeout is only asked to be deleted, and a deny-all connection fails with Permission denied rather than being refused. Add colima to the spelling wordlist.
Resolve the conflicts with the Boat and Islo sandbox backends, which landed on main in the same files: extras, integrations and docs now list all three, and uv.lock is main's plus openshell.
kaxil
approved these changes
Oct 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
OpenShellSandboxBackend, a self-hostedSandboxToolsetbackend that runs sandboxes through anNVIDIA OpenShell gateway (verified on its Docker driver), behind a new
openshellextra. Not attachable in this version:SandboxSpec.owneris refused, as on OpenSandbox.Why another backend
OpenShell is self-hosted and Apache-2.0, so the system test runs on a laptop with no paid account. Its runtime
enforces the toolset's defaults natively: on the Docker driver the workload container has only a loopback
interface, Landlock and seccomp confine it, and a per-sandbox supervisor opens every connection against a policy, so deny-all and
a hostname allowlist on port 443 are enforced by the runtime, not approximated. No DNS query leaves a deny-all
sandbox, and the backend re-reads the effective policy before and after every command, not only at create,
because OpenShell policies can be widened on a running sandbox (by an admin, a gateway-wide policy, or
auto-approved proposals). That was measured below.
What it does
allow_egress_tobecomes one rule with anendpoint per host, port 443, any binary. Refused:
block_network=False,allow_egress_to_cidrs, and non-hostname entries.GetSandboxConfig. The sandbox isdestroyed and the task fails unless the policy admits exactly the requested hosts, comes from the sandbox
rather than a GLOBAL policy, keeps Landlock at
hard_requirement, and has neitherproposal_approval_mode=autonor agent proposals enabled. The same check runs before and after everycommand.
sandbox execcommand and everything it spawned NVIDIA/OpenShell#3159, closed not_planned),so a guest wrapper runs the command from stdin in its own session, spools output to
/tmp, times the budgeton
/proc/uptime(counting its own sleeps drifted late on a CPU-starved sandbox), and on timeout SIGKILLs theprocesses in the session, rescanning
/procuntil a pass kills nothing, at most 50 times. A command thatcould not be staged in
/tmpis not run and comes back to the model as a recoverable error; an imagewithout
setsidfails the task. The wrapper's own failure lines carry a per-call nonce, so a command cannotfake them or have its own exit read as one. The exit status comes from the wrapper's own process
status, never from a trailer the command could print. Each stream is capped on the worker. If the gateway
goes quiet past the budget + 30 s, the result is
timed_outandsandbox_terminated, and the backend asksthe gateway to delete the sandbox, logging a warning if that fails. A gateway restart is never reported as
a success: if the sandbox comes back, the command's outcome is reported as unknown; if not, the task fails.
1 MiB). Reads are raw bytes, capped in the guest, and an oversized read reports its real size. A path the
gateway would reject (a NUL, or over 32 KiB) comes back as a recoverable error before any exec.
OPENSHELL_*, arerefused. Everything else reaches commands as given (no login shell).
and per worker (the SDK writes refreshed tokens back), and mTLS is the setup for a read-only mount; no
connection type (following
Add Islo backend for sandbox tools #71672 (comment)).
openshell>=0.1.2,<0.2plusgrpcio>=1.78.0andprotobuf>=6.31.1. The wheel declares lowerfloors, but its generated code fails to import below these (measured).
Measured against a live OpenShell 0.1.2 gateway at
3b6e4ad57f(gateway on macOS with the Docker driver against colima: Ubuntu 24.04 VM, kernel 6.8.0-64, Landlock ABI 4,
Docker 28.4.0;
python:3.12-slim)true×20)sleep 300, 3 s and 5 s budgetssleep 300left in/proc/procsleep 120 & echo started/proc/net/devlists onlylo1.1.1.1on 443, 80 and 22 and to[2606:4700:4700::1111]:443fails with EACCES; UDP to8.8.8.8:53and1.1.1.1:53fails;getent hosts example.comanswers a synthetic 198.18.0.2example.com1.1.1.1:443andexample.org:443deniedtouchin/usr,/etc,/and the world-writable/var/tmpdenied;/tmp,/dev/shmand the workdir writabledocker restartmid-command (earlier run, on this PR's first revision of 2026-09-30)timed_out+sandbox_terminatedat 65.0 s; the delete timed out and was logged; the sandbox was still there after resumeLimitations
_stubfor exec and config reads (thepublic exec buffers all output), hence the
<0.2cap.when the policy is widened can use it. On Kubernetes, egress also depends on a NetworkPolicy the cluster's
CNI must enforce, which the check cannot see. Only the Docker driver was measured.
generate-certseffectivelynever expire; use your own PKI.
re-encrypted by the supervisor with its own CA, so clients with their own trust store need pointing at it.
Every denied connect becomes a draft proposal visible to operators.
created-by=airflowandairflow-created-at, and the docs include a reaper recipe.setsiditself survives a timeout, and one that keeps forking faster than thesweep can outlast it. Output is sent only when a command ends,
so proxies in front of the gateway need an idle timeout of at least the budget + 30 s. Writes over 768 KiB
are not atomic. BusyBox images are unsupported.
Tests
test_openshell.py, 146 tests; 132 pass on macOS, where the 14 real-/procwrapper tests are skipped,and they run in the Airflow CI image on Linux.
example_sandbox_toolset_openshell.pypassed atfaba6c2956against a live OpenShell 0.1.2gateway on its Docker driver (24.5 s).
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.