[surajsharan]

Summary

#1233 took jupyter_env's verification off the notebook kernel and pinned what that does not fix: E2B starts every command as a login shell and the kernel runs as root, so a profile the agent writes still decides what a failing verify command reports. A test in that PR asserts the hole rather than hiding it, and #1232 tracks closing it.

This closes it for jupyter_env, as the first environment. On #1232 @burtenshaw asked which task-declared paths may be submitted and which setup and test files must come exclusively from the verifier; this implements the answer I gave there. @zuraiz-anjum then reproduced two of the attacks it refuses, against coding_tools_env, and is taking that environment next with the same layout. The design has not been signed off, so treat the shape as proposed rather than agreed, and the RFC note below stands.

A task passes submit_paths to reset. With paths declared, submitting verifies in a second sandbox:

1. A fresh sandbox from the same template.
2. The task's own setup commands, before any of the agent's work exists.
3. The declared paths copied out of the agent's sandbox into /submission as data. It is not the working directory and not on PATH or PYTHONPATH.
4. The verify commands, the reward-file clear and the reward-file read, all belonging to this sandbox alone.
5. The sandbox is killed.

The order is what makes it a boundary rather than more hardening: setup runs before the submission arrives, so nothing the agent submits can replace the task's test files, and the reward file exists only where the agent has never been.

submit_paths defaults to empty, and an empty list keeps the current in-place behaviour untouched, so every existing task scores exactly as it does today. The opt-in is also why the extra sandbox per scored rollout is acceptable.

What is refused

Paths are relative to /home/user. Refused with a reason recorded in state.submission_refused: an absolute path, one that leaves the workspace with .., a symlink, a socket, a fifo, a device, or a submission past 200 files or 10 MiB. Symlinks are reported rather than followed because following one copies whatever it points at.

Files that a shell or a test runner reads on the way to the first assertion are left behind even when a declared directory contains them: .bash_profile, .bashrc, .profile, sitecustomize.py, *.pth, conftest.py, pytest.ini, tox.ini, pyproject.toml. Declaring one of those exactly still submits it, since naming it is the task author's decision rather than an accident.

One decision worth flagging

Because the submission lands in /submission, a task that opts in writes its verify commands against that path. The alternative, copying into the workspace at the same relative paths after setup, would keep existing verify commands working unchanged, at the cost of letting a declared path collide with a file the task's own setup created. I chose the separate tree because that collision is exactly what the boundary is for, and because it matches Harbor's layout, where /tests is the verifier's and the agent's work is not. Say the word if adoption cost matters more and I will switch it.

Closes #1232 for jupyter_env. coding_tools_env is @zuraiz-anjum's next, with the same layout.

Type of Change

• Bug fix
• New feature
• Documentation

Alignment Checklist

• I have read .claude/docs/PRINCIPLES.md and this PR aligns with our principles
• I have checked .claude/docs/INVARIANTS.md and no invariants are violated
• I have run bash .claude/hooks/lint.sh and tests and addressed all issues

RFC Status

The implementation is environment-local: a reset kwarg, a second sandbox and one new module under envs/jupyter_env/server/. No core API, and nothing for another environment to import. The pattern is the part others would copy, so if you would rather have an RFC first, this PR is its reference implementation and I am happy to write it up.

Test Plan

New tests in tests/envs/test_jupyter_environment.py, driving the real JupyterEnvironment and E2BSandbox against two stand-in sandboxes: a kernel whose namespace persists between cells, a process runner that starts each command as bash -l -c with HOME set the way envd does, and a file API over each sandbox's own temporary directory.

• The case Run jupyter_env verification outside the notebook kernel #1233 pinned at 1.0: a cell writes trap 'exit 0' EXIT into the profile, and the episode now scores 0.0, because the verifier never reads that sandbox. Both tests live in the file, so the before and after sit next to each other.
• @zuraiz-anjum's four cases from Verify sandbox environments outside the agent's sandbox #1232, with their names: a wrong answer scores 0.0, a right answer 1.0, a wrong answer plus a conftest.py whose pytest_runtest_makereport hookwrapper marks every outcome passed scores 0.0 with the refusal recorded, and a rewritten copy of the task's test file does not replace the one setup wrote.
• Removing the exclusion makes the conftest.py case score 1.0 again, so the test fails for the reason it claims.
• An absolute path, ../outside.txt and work/../../outside.txt are refused while the rest of the submission still scores.
• A symlink to /etc/passwd is refused and never reaches /submission.
• With no declared paths, verification does not start a second sandbox at all and the reward is unchanged.
• Setup failing in the verifier sandbox is an error with no reward, not a score.

pytest tests/envs/test_jupyter_environment.py → 40 passed. Full suite with CI's dependency set and GITHUB_ACTIONS=true: 4132 passed, 132 skipped, 0 failed. ruff check and ruff format --check on src/ and tests/ pass, usort check flags only the two files it flags on main, and scripts/sync_env_docs.py --check passes.

Out of scope

• Not run against a live E2B sandbox, since I have no account, so the per-rollout cost of creating and setting up the second sandbox is unmeasured. That number decides whether this is opt-in per task or per training run.
• The copy uses find -printf, which is GNU find, as E2B's code-interpreter template has. A template with BusyBox find would need a different listing command, and the stand-in tests cannot catch that.
• terminus_env and coding_tools_env still verify in the agent's kernel. @zuraiz-anjum has the coding_tools_env port, and Update Terminus env harness and sandbox backends #729 rewrites terminus_env's sandbox.

Claude Code Review

N/A

[zuraiz-anjum]

Thanks for carrying the case names over, that makes the coding_tools_env port easy to line up.

One gap in EXCLUDED_NAMES: pytest reads its configuration from more files than the three listed, and each of these turns a failing run into exit 0 just like pytest.ini does. With pytest 9.2, a directory holding one failing test and one of these files, run as python -m pytest:

file	content	exit
none		1
setup.cfg	[tool:pytest] addopts = --co -q	0
.pytest.ini	[pytest] addopts = --co -q	0
pytest.toml	[pytest] addopts = ["--co", "-q"]	0
.pytest.toml	[pytest] addopts = ["--co", "-q"]	0

--co only collects, so nothing runs and pytest reports success. setup.cfg and .pytest.ini work on any recent pytest; the two TOML names are newer. Adding all four next to pytest.ini, tox.ini and pyproject.toml keeps the rule the same: a file the test runner reads before the first assertion stays behind unless the task names it. .bash_login belongs with the shell files for the same reason, since a login shell reads the first of .bash_profile, .bash_login and .profile.

I'll use the same list in coding_tools_env so the two stay aligned.

[surajsharan]

Good catch, and your table holds on pytest 9.1.1 as well as 9.2. One failing test in a directory, python -m pytest:

file	exit
none	1
setup.cfg with [tool:pytest]	0
.pytest.ini	0
pytest.toml	0
.pytest.toml	0

Pushed all four, plus .bash_login, which a login shell reads when there is no .bash_profile. The list is now the login files, the Python import hooks, and every file pytest takes options from, with a parametrized test pinning it so it cannot quietly shrink, and asserting that an exact declaration still submits the file.

Thanks for measuring it rather than listing it.