Verify jupyter_env in a sandbox the agent never touched

docs/source/environments/jupyter.md

@@ -105,7 +105,43 @@ E2B starts each command as a login shell, so a profile the agent writes decides
 what a verify command reports: an `exit 0` or an `EXIT` trap in
 `/root/.bash_profile` makes a failing command look like a passing one. A test
 pins that behaviour rather than leaving it to be found in a reward curve, and
-#1232 tracks verification outside the agent's sandbox, which is what closes it.
+`submit_paths` closes that, per environment and per task.
+
+## Verifying outside the agent's sandbox
+
+A task that passes `submit_paths` to `reset` is verified in a second sandbox the
+agent never touches. The list defaults to empty, and an empty list keeps the
+behaviour above, so existing tasks score exactly as before.
+
+With paths declared, submitting runs this order, which is what makes it a
+boundary rather than hardening:
+
+1. A fresh sandbox is created from the same template.
+2. The task's `setup` commands run there, before any of the agent's work exists.
+3. The declared paths are copied out of the agent's sandbox into `/submission`,
+   which is not the working directory and is not on `PATH` or `PYTHONPATH`.
+4. The verify commands run, and the reward file is cleared and read there.
+5. The sandbox is killed.
+
+Paths are relative to `/home/user`. An absolute path, one that leaves the
+workspace with `..`, a symlink, a socket, a fifo, a device, or a submission over
+200 files or 10 MiB is refused, and the reason is recorded in
+`state.submission_refused`. Files that a shell or a test runner reads before the
+first assertion are left behind even when a declared directory contains them:
+the login files a shell reads (`.bash_profile`, `.bash_login`, `.profile`,
+`.bashrc`), what Python runs before an import (`sitecustomize.py`, `*.pth`,
+`conftest.py`), and every file pytest takes options from (`pytest.ini`,
+`.pytest.ini`, `pytest.toml`, `.pytest.toml`, `pyproject.toml`, `setup.cfg`,
+`tox.ini`), since any of them can carry `addopts = --co` and report success
+without running a test. Declaring one of those exactly still submits it, since
+naming it is the task author's decision.
+
+Because the submission is no longer in the workspace, a task that opts in writes
+its verify commands against `/submission`. The task's own test files keep coming
+from its `setup` commands, so nothing the agent submits can replace them.
+
+The cost is one extra sandbox per scored rollout, which is why it is opt-in
+rather than the default.
 
 Only a command that ran and exited decides a verify result. If the sandbox
 itself fails, because it cannot be reached, the command cannot be started, or

envs/jupyter_env/README.md

@@ -120,7 +120,43 @@ E2B starts each command as a login shell, so a profile the agent writes decides
 what a verify command reports: an `exit 0` or an `EXIT` trap in
 `/root/.bash_profile` makes a failing command look like a passing one. A test
 pins that behaviour rather than leaving it to be found in a reward curve, and
-#1232 tracks verification outside the agent's sandbox, which is what closes it.
+`submit_paths` closes that, per environment and per task.
+
+## Verifying outside the agent's sandbox
+
+A task that passes `submit_paths` to `reset` is verified in a second sandbox the
+agent never touches. The list defaults to empty, and an empty list keeps the
+behaviour above, so existing tasks score exactly as before.
+
+With paths declared, submitting runs this order, which is what makes it a
+boundary rather than hardening:
+
+1. A fresh sandbox is created from the same template.
+2. The task's `setup` commands run there, before any of the agent's work exists.
+3. The declared paths are copied out of the agent's sandbox into `/submission`,
+   which is not the working directory and is not on `PATH` or `PYTHONPATH`.
+4. The verify commands run, and the reward file is cleared and read there.
+5. The sandbox is killed.
+
+Paths are relative to `/home/user`. An absolute path, one that leaves the
+workspace with `..`, a symlink, a socket, a fifo, a device, or a submission over
+200 files or 10 MiB is refused, and the reason is recorded in
+`state.submission_refused`. Files that a shell or a test runner reads before the
+first assertion are left behind even when a declared directory contains them:
+the login files a shell reads (`.bash_profile`, `.bash_login`, `.profile`,
+`.bashrc`), what Python runs before an import (`sitecustomize.py`, `*.pth`,
+`conftest.py`), and every file pytest takes options from (`pytest.ini`,
+`.pytest.ini`, `pytest.toml`, `.pytest.toml`, `pyproject.toml`, `setup.cfg`,
+`tox.ini`), since any of them can carry `addopts = --co` and report success
+without running a test. Declaring one of those exactly still submits it, since
+naming it is the task author's decision.
+
+Because the submission is no longer in the workspace, a task that opts in writes
+its verify commands against `/submission`. The task's own test files keep coming
+from its `setup` commands, so nothing the agent submits can replace them.
+
+The cost is one extra sandbox per scored rollout, which is why it is opt-in
+rather than the default.
 
 Only a command that ran and exited decides a verify result. If the sandbox
 itself fails, because it cannot be reached, the command cannot be started, or

envs/jupyter_env/models.py

@@ -38,6 +38,9 @@ class JupyterState(State):
     setup_results: List[CommandResult] = Field(default_factory=list)
     verify_results: List[CommandResult] = Field(default_factory=list)
     verify_commands: List[str] = Field(default_factory=list)
+    setup_commands: List[str] = Field(default_factory=list)
+    submit_paths: List[str] = Field(default_factory=list)
+    submission_refused: List[str] = Field(default_factory=list)
     submitted_answer: Optional[str] = None
     last_reward: Optional[float] = None
     reward_override_ignored: Optional[str] = None

envs/jupyter_env/server/e2b_sandbox.py

@@ -203,6 +203,10 @@ def write_file(self, filename: str, content: bytes) -> None:
         """Upload a file into the sandbox filesystem."""
         self._sbx.files.write(filename, content)
 
+    def read_file(self, filename: str) -> bytes:
+        """Read a file out of the sandbox filesystem."""
+        return self._sbx.files.read(filename, format="bytes")
+
     def kill(self) -> None:
         """Terminate the sandbox. Safe to call multiple times."""
         try:

envs/jupyter_env/server/jupyter_environment.py

@@ -22,9 +22,11 @@
 from openenv.core.env_server.types import Action, Observation
 
 try:
+    from . import submission
     from .e2b_sandbox import E2BSandbox
     from .notebook_tracker import NotebookTracker
 except ImportError:  # pragma: no cover
+    from server import submission
     from server.e2b_sandbox import E2BSandbox
     from server.notebook_tracker import NotebookTracker
 
@@ -223,7 +225,7 @@ def reset(
             )
 
         try:
-            self._sandbox = E2BSandbox(api_key=api_key)
+            self._sandbox = self._new_sandbox()
         except Exception as exc:  # noqa: BLE001
             self._state = self._JupyterState(
                 episode_id=episode_id or str(uuid4()),
@@ -251,6 +253,8 @@ def reset(
             kwargs.get("verify", kwargs.get("verify_scripts", []))
         )
         self._state.verify_commands = verify_commands
+        self._state.setup_commands = setup_commands
+        self._state.submit_paths = _coerce_commands(kwargs.get("submit_paths", []))
 
         if setup_commands:
             self._sandbox.run_shell("mkdir -p /home/user/logs/verifier")
@@ -390,29 +394,83 @@ def _run_shell_commands(self, commands: Iterable[str]):
             for command in commands
         ]
 
+    def _new_sandbox(self) -> E2BSandbox:
+        return E2BSandbox(api_key=os.environ["E2B_API_KEY"])
+
+    def _copy_submission(self, verifier: E2BSandbox) -> list[str]:
+        """Copy the declared paths into the verifier sandbox, return refusals."""
+        refused: list[str] = []
+        found: list[submission.SubmissionFile] = []
+        for declared in self._state.submit_paths:
+            reason = submission.check_declared_path(declared)
+            if reason:
+                refused.append(reason)
+                continue
+            listing = self._sandbox.run_command(submission.listing_command(declared))
+            files, refusals = submission.parse_listing(declared, listing.stdout)
+            found.extend(files)
+            refused.extend(refusals)
+        found, capped = submission.apply_caps(found)
+        refused.extend(capped)
+
+        for entry in found:
+            content = self._sandbox.read_file(
+                f"{submission.WORKSPACE}/{entry.relative_path}"
+            )
+            verifier.write_file(
+                f"{submission.SUBMISSION_DIR}/{entry.relative_path}", content
+            )
+        return refused
+
+    def _verify_in_fresh_sandbox(self) -> dict[str, Any]:
+        """Verify the submission in a sandbox the agent has never touched.
+
+        The order is what makes it a boundary: the task's setup runs before the
+        submission arrives, the submission lands outside the workspace, and the
+        verify commands and the reward file belong to this sandbox alone.
+        """
+        verifier = self._new_sandbox()
+        try:
+            setup_results = [
+                _command_result_from_cell_result(command, verifier.run_command(command))
+                for command in self._state.setup_commands
+            ]
+            failed = [result.command for result in setup_results if not result.success]
+            if failed:
+                raise RuntimeError(
+                    f"setup failed in the verifier sandbox: {failed[0]!r}"
+                )
+
+            self._state.submission_refused = self._copy_submission(verifier)
+            return self._score(verifier)
+        finally:
+            verifier.kill()
+
     def _run_verify_commands(self) -> dict[str, Any]:
         if not self._sandbox:
             return {"passed": 0, "total": 0, "reward": None}
+        if self._state.submit_paths:
+            return self._verify_in_fresh_sandbox()
+        return self._score(self._sandbox)
 
+    def _score(self, sandbox: E2BSandbox) -> dict[str, Any]:
         # Verification runs as separate sandbox processes, not in the notebook
         # kernel, so nothing a cell does to the kernel changes how verify
         # commands run or what they report.
-        self._sandbox.run_command("mkdir -p /home/user/logs/verifier")
+        sandbox.run_command("mkdir -p /home/user/logs/verifier")
         # The override is documented as verify-written, so drop anything the
         # policy wrote there during the episode before verification starts.
-        cleared = self._sandbox.run_command(_CLEAR_REWARD_FILE).success
+        cleared = sandbox.run_command(_CLEAR_REWARD_FILE).success
         verify_results = [
-            _command_result_from_cell_result(
-                command, self._sandbox.run_command(command)
-            )
+            _command_result_from_cell_result(command, sandbox.run_command(command))
             for command in self._state.verify_commands
         ]
         self._state.verify_results = verify_results
 
         passed = sum(1 for result in verify_results if result.success)
         total = len(verify_results)
         reward, self._state.reward_override_ignored = _read_reward_override(
-            self._sandbox, cleared
+            sandbox, cleared
         )
         if reward is None and total:
             reward = passed / total

envs/jupyter_env/server/submission.py

@@ -0,0 +1,142 @@
+"""Which of the agent's files may be handed to a verifier, and which may not.
+
+Verification in a fresh sandbox (#1232) copies the paths a task declares in
+`submit_paths` out of the agent's sandbox into `/submission` there. The
+submission is data: it is not the working directory and not on `PATH` or
+`PYTHONPATH`. A task's own tests normally run against it, though, so a file a
+shell or a test runner reads on its way to the first assertion is code rather
+than data, and those are left behind.
+"""
+
+from dataclasses import dataclass
+from typing import List, Optional, Tuple
+
+SUBMISSION_DIR = "/submission"
+WORKSPACE = "/home/user"
+
+# Read before anything a verify command asserts: a shell's login files, Python's
+# import-time hooks, and a test runner's configuration and fixtures.
+EXCLUDED_NAMES = frozenset(
+    {
+        # A login shell reads the first of these that it can.
+        ".bash_profile",
+        ".bash_login",
+        ".profile",
+        ".bashrc",
+        # Python runs these on the way to importing anything.
+        "sitecustomize.py",
+        "conftest.py",
+        # Every one of these can carry `addopts = --co`, which collects the
+        # tests without running them and reports success.
+        "pytest.ini",
+        ".pytest.ini",
+        "pytest.toml",
+        ".pytest.toml",
+        "pyproject.toml",
+        "setup.cfg",
+        "tox.ini",
+    }
+)
+
+# A submission is an answer, not a dataset.
+MAX_FILES = 200
+MAX_TOTAL_BYTES = 10 * 1024 * 1024
+
+
+@dataclass
+class SubmissionFile:
+    """A regular file to copy, as a path relative to the workspace."""
+
+    relative_path: str
+    size: int
+
+
+def check_declared_path(path: str) -> Optional[str]:
+    """Refuse a `submit_paths` entry, or return `None` to accept it.
+
+    Args:
+        path (`str`):
+            The entry as the task declared it.
+
+    Returns:
+        `str` or `None`: why the entry was refused, or `None` when it is usable.
+
+    An excluded name is accepted here on purpose: declaring one exactly is the
+    task author choosing to submit it, while sweeping a directory that contains
+    one is an accident, which [`excluded_file`] refuses.
+    """
+    if not path:
+        return "empty path"
+    if path.startswith("/"):
+        return f"not relative to {WORKSPACE}: {path!r}"
+    if ".." in path.split("/"):
+        return f"leaves the workspace: {path!r}"
+    return None
+
+
+def excluded_file(relative_path: str) -> Optional[str]:
+    """Refuse a file found under a declared directory, or return `None`."""
+    name = relative_path.rsplit("/", 1)[-1]
+    if name in EXCLUDED_NAMES or name.endswith(".pth"):
+        return f"runs before verification: {relative_path!r}"
+    return None
+
+
+def listing_command(declared_path: str) -> str:
+    """Build the command that lists the regular files under a declared path.
+
+    `-printf` is GNU find, which E2B's code-interpreter template has. The type
+    letter comes before the size so a symlink, socket, fifo or device is
+    reported rather than read: following one copies whatever it points at.
+    """
+    target = f"{WORKSPACE}/{declared_path}"
+    return f"find '{target}' -maxdepth 8 -printf '%y\\t%s\\t%P\\n' 2>/dev/null || true"
+
+
+def parse_listing(
+    declared_path: str, stdout: str
+) -> Tuple[List[SubmissionFile], List[str]]:
+    """Turn one [`listing_command`] result into files to copy and refusals.
+
+    Args:
+        declared_path (`str`):
+            The entry being expanded, used to build workspace-relative paths.
+        stdout (`str`):
+            Output of [`listing_command`].
+
+    Returns:
+        `tuple` with:
+            - `list[SubmissionFile]`: the regular files to copy.
+            - `list[str]`: one reason per refused entry.
+    """
+    files: List[SubmissionFile] = []
+    refusals: List[str] = []
+    for line in stdout.splitlines():
+        kind, _, rest = line.partition("\t")
+        size, _, below = rest.partition("\t")
+        if kind == "d":
+            continue
+        # find prints an empty %P for the path it was given, so an empty one
+        # means the declared path is itself the file.
+        relative = f"{declared_path}/{below}" if below else declared_path
+        if kind != "f":
+            refusals.append(f"not a regular file: {relative!r}")
+            continue
+        reason = excluded_file(relative) if below else None
+        if reason:
+            refusals.append(reason)
+            continue
+        files.append(SubmissionFile(relative_path=relative, size=int(size)))
+    if not files and not refusals:
+        refusals.append(f"nothing to submit at {declared_path!r}")
+    return files, refusals
+
+
+def apply_caps(files: List[SubmissionFile]) -> Tuple[List[SubmissionFile], List[str]]:
+    """Keep a submission inside `MAX_FILES` and `MAX_TOTAL_BYTES`."""
+    if len(files) > MAX_FILES:
+        return [], [f"more than {MAX_FILES} files declared: {len(files)}"]
+    total = sum(entry.size for entry in files)
+    if total > MAX_TOTAL_BYTES:
+        return [], [f"submission is larger than {MAX_TOTAL_BYTES} bytes: {total}"]
+    return files, []