[kashif]

Summary

Adds an OR-Tools env for training models on hard OR problems, following LLMs Can Design Near-Optimal OR Algorithms. Problems are job shop, RCPSP, CVRP, CVRP with time windows, weighted tardiness, nurse rostering and course timetabling, generated at random sizes from the standard benchmark designs. At level 1 the agent submits a solution; at level 2 it submits a solve(instance) that is scored on hidden instances. The reward places the objective between a naive baseline (0) and the OR-Tools reference (1). There are also check tools for feedback before submitting, an exporter to Harbor tasks, and tests that cross-check the pure-Python verifiers against OR-Tools.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation
• New environment
• Refactoring

Alignment Checklist

Before submitting, verify:

• I have read .claude/docs/PRINCIPLES.md and this PR aligns with our principles
• I have checked .claude/docs/INVARIANTS.md and no invariants are violated
• I have run /pre-submit-pr (or bash .claude/hooks/lint.sh and tests) and addressed all issues

RFC Status

• Not required (bug fix, docs, minor refactoring)
• RFC exists: #___
• RFC needed (will create before merge)

Test Plan

PYTHONPATH=src:envs pytest tests/envs/test_ortools_env.py

I've also been running GRPO on it with TRL (Qwen3-1.7B, level 2 CVRP) and checking the completions; the TRL example lives in TRL.

Claude Code Review

N/A

---

Note

Medium Risk
The PR adds untrusted-code execution and a sandbox with documented containment gaps; risk is mitigated by private-deployment guidance, isolation tests, and no automatic Spaces release.

Overview
Introduces ortools_env, a new OpenEnv environment for training and evaluating agents on hard operations-research tasks (job shop, RCPSP, CVRP/CVRPTW, tardiness, rostering, timetabling). Episodes follow a two-level design: level 1 submits a solution for one instance; level 2 submits a solve(instance) algorithm scored on hidden instances. Reward is normalized between a naive baseline and an OR-Tools reference via MCP tools (run_python, check_solution / test_algorithm, submit_*).

The implementation adds problem modules (generators, loaders, pure-Python verifiers, reference solvers), a subprocess sandbox with import blocking and container UID/interpreter separation, build_dataset / Harbor export, docs, an example episode script, and broad pytest coverage. SKIP_HF_DEPLOYMENT and SECURITY.md document that the sandbox is not ready for hostile public deployment.

CI gains a test-ortools-env job: locked uv sync, regression tests, production Docker build, and offline container checks for sandbox isolation (including an expected failure for detached-child escape).

^{Reviewed by Cursor Bugbot for commit 07f3da5. Bugbot is set up for automated code reviews on this repo. Configure here.}

[burtenshaw]

Review of the draft (head a4275f7). Lint, usort, ruff format and sync_env_docs.py --check are clean. All 37 tests in tests/envs/test_ortools_env.py pass on Linux semantics. Structure matches echo_env, there are no cross-env imports, the client only imports models, rewards are computed in the env, and reset is not exposed as an MCP tool. I pushed nothing.

Blocking

1. The ortools import block can be bypassed, so an agent can score 1.0 by calling the reference solver (envs/ortools_env/sandbox.py:23-34). The blocker is a sys.meta_path entry inside the agent's own process, so one line removes it:

   import sys
   sys.meta_path[:] = [m for m in sys.meta_path if type(m).__name__ != "_Blocker"]
   from ortools.sat.python import cp_model  # works with allow_ortools=False

   I ran submit_algorithm at level 2 (tardiness, allow_ortools=False) with a solve that does this, puts envs/ on sys.path and calls tardiness.reference. Reward was 1.0000 on every eval instance. The same trick imports problems and server. The module docstring says this is not a security boundary. That is fine for isolation, but here the block is what keeps the reward honest. One fix is to run sandboxed code with an interpreter or venv that does not have ortools or the env source installed, for example a second venv in the Dockerfile that has only numpy and scipy.

2. CI never runs the env tests. tests/envs/test_ortools_env.py:11 uses importorskip("ortools"), and ortools is not a root dependency, so the whole file is skipped in test (3.x). envs/ortools_env/ also has no uv.lock, which echo_env and the other envs have, and the Dockerfile falls back to an unlocked uv sync. Please add a lock and a CI job like test-thinkingbox-env.

Non-blocking

• sandbox.py:48-53: RLIMIT_AS raises in preexec_fn on macOS, so 11 tests fail locally there. Linux-only is fine for the container, but please skip those tests or note it.
• sandbox.py:99-113: the timeout kills only the direct child. A run_python call can start a detached process that keeps solving after the call returns and writes to the persistent workdir, which gets around compute_budget_s. Running the child in a new session and killing the process group would close most of this.
• Scope: export_harbor.py, harbor_verify.py (it copies score) and build_dataset.py add about 350 lines next to the env itself. A follow-up PR for the Harbor export would keep this one smaller.
• I tried malformed submissions (empty, wrong shape, strings, wrong types). They all score -1 through evaluate and score, and NaN clamps to -1. I found no way to get reward without a feasible solution apart from item 1.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[sergiopaniego]

Thanks for this, really nice env. The two-level setup (solution vs solve(instance) on hidden instances) and the reward normalized against a naive baseline are a great fit for RL with verifiable rewards, and the follow-ups to the first review (sandbox user + separate interpreter, lockfile, test-ortools-env job, SECURITY.md) are solid. Reviewed at 5096fda6.

Bugbot findings

Thanks for 1372634, it fixes Bugbot 1 (episode ending with reward=None when the reference raises) and 4 (Harbor level 1 scoring against the editable /app/instance.json), with tests for both. The other two still reproduce at head and I'd fix them before merge:

2. Level-2 tool timeout does not budget the baseline, which is a 60s CP-SAT solve for rostering and timetabling.
3. Timetabling generate can raise for some (seed, size) pairs with no retry, so reset fails.

Other comments

• Keep it out of automatic Spaces releases. SECURITY.md says the sandbox is not ready for hostile submissions, but the env has no SKIP_HF_DEPLOYMENT file, so the next prepare_hf_deployment.sh --all release would publish it as a public Space under openenv/. I'd add envs/ortools_env/SKIP_HF_DEPLOYMENT (as posttrain_env and thinkingbox_env do) plus a line in the README/docs page saying it is excluded from Spaces releases and should run locally or on private infra, one container per session, without network and with Docker limits (--pids-limit, --memory), as the CI job does. That keeps the env usable for training now without blocking on the sandbox work. Per-submission PID namespace / cgroup isolation could then be a follow-up issue, probably at the OpenEnv level since every env that runs model code has the same gap.
• Results and example. Thanks for sharing the preliminary GRPO + LoRA run (Qwen3-4B, tardiness n40, level 2), going from about -0.6 to 0.8 mean reward in 50 steps is a nice sign that the reward is learnable. Looking forward to the longer runs. When they're ready, it'd be great to add a plot to the README/docs, ideally with a held-out eval before/after training, a simple heuristic as a point of comparison (e.g. ATC for tardiness), and one of the harder problems (CVRP or job shop) where the env should have the most headroom. Other envs ship a runnable script in examples/ (e.g. qed_math_inference.py, pelican_svg_grpo.py, posttrain_env_agent.py). A small one here, e.g. an agent playing one level-1 episode through the MCP tools, plus a link to the TRL GRPO example, would help people get started. The GRPO script itself can stay in TRL.
• Docker in the README. Since the real isolation (sandbox user and separate interpreters) only exists inside the container, it'd be worth adding a short "Building and running the Docker image" section to the README. Right now the build command only appears in the CI workflow.
• Naive baseline wording. The README says the naive solution "ignores the instance data", but for rostering and timetabling it is a CP-SAT feasibility solve on the instance. Worth adjusting the sentence, and maybe noting that this baseline depends on solver time and threads, so precomputing it with build_dataset is recommended for stable rewards.
• Harbor and ortools. In Harbor tasks nothing stops the agent from installing ortools. Probably fine for measuring frontier agents, but worth stating so those numbers are not compared directly with allow_ortools=False runs.
• Scope. +1 to splitting the Harbor export (export_harbor.py, harbor_verify.py) into a follow-up. It would make this PR easier to land.
• PR description. The test plan mentions Qwen3-1.7B on CVRP, while the run above is Qwen3-4B on tardiness, and it only lists test_ortools_env.py. Worth updating it with the results, the sandbox test and the CI job, and a short note on the sandbox status from SECURITY.md.

[burtenshaw]

NEEDS AUTHOR CHANGES. Reviewed head 0974449.

Add envs/ortools_env/SKIP_HF_DEPLOYMENT before merging. The file is absent, and running the existing discovery function prints "discover_all_envs selected: ortools_env". scripts/prepare_hf_deployment.sh:319 checks the marker before scripts/prepare_hf_deployment.sh:323 adds an environment to bulk deployment. The script defaults to private Spaces at scripts/prepare_hf_deployment.sh:110; using --public publishes the selected environments. envs/ortools_env/SECURITY.md:3 retains the hostile-submission deployment hold, and tests/containers/check_ortools_sandbox.py:96 retains an expected failure for detached children. Add the requested README and docs instructions for local or private infrastructure, one container per session, disabled network access and container PID/memory limits.

Suggestion: extend the production container checks to verify source and reference isolation with allow_ortools=True, then add a live-server test with synthetic hidden-instance/reference canaries and orchestration endpoint isolation. tests/containers/check_ortools_sandbox.py:27 calls the sandbox helper without starting the server. Its full denial assertions run with allow_ortools=False at tests/containers/check_ortools_sandbox.py:30; the allow_ortools=True branch only checks a successful OR-Tools import at tests/containers/check_ortools_sandbox.py:60. envs/ortools_env/SECURITY.md:49 already records the endpoint-isolation requirement. This is a coverage suggestion, not another demonstrated bypass.

Validation: 65 environment tests and 20 sandbox tests passed locally. Timetabling reset passed 2,000 cases across seeds 0 through 199, all five sizes and both levels. The dedicated CI job at this head passed 85 pytest cases and three production-container checks, with one expected failure for detached-child cleanup.

[burtenshaw]

Rechecked at dd1f8c34. The remaining item from our review is resolved: envs/ortools_env/SKIP_HF_DEPLOYMENT makes scripts/prepare_hf_deployment.sh --all skip the environment (line 319), and the README and docs now describe local or private deployment with one container per session, an internal network, and PID and memory limits. The baseline wording is corrected as well. Nothing further from our side.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 07f3da5. Configure here.}