Measure OpenEnv's reward-hack defenses and prototype a Karotte bridge #1357

Owner: @thegovind

Summary

Karotte is Preference Model's MIT-licensed framework for hardening RL environments by default. Agents run as an unprivileged student user, grading data is root-only, and the harness kills every student process and copies the submission into root-only custody before grading. It tests its firewall at startup and applies runtime-dependent resource limits, with the strongest guarantees on VMs; Docker has no memory cap by default.

Its blog compares Karotte, Harbor, HUD, AgentEnv, verifiers and Inspect on five reward hacks: reading the answer key, tampering with grading, unusual answer files (FIFO or symlink), exhausting memory/processes/disk, and looking answers up online. Harbor 0.23 failed four of five. OpenEnv is absent from this vendor-authored comparison.

OpenEnv gaps

harbor_env (#1036) is now the recommended black-box path. src/openenv/harbor/rollout.py runs Harbor's Trial with the default VerifierConfig(). It likely inherits Harbor's results, depending on the backend; we have not measured it.

The shared-sandbox verification problem in the five native agentic envs (jupyter_env, terminus_env, coding_tools_env, opencode_env and pi_env) is already tracked in #1232, following #1199 / #1200.

tests/test_validation/integration/test_runtime_cli.py marks RFC 008's runtime.network_policy, runtime.host_containment, runtime.resource_bounds, runtime.episode_isolation and runtime.oracle_containment as PENDING. semantic.canary_floor has no grader. statistical.adversarial_floor is reserved; RFC 008 says its procedure is "not yet specified anywhere". Karotte's scripted attacks could seed that panel. They need no inference, so they can run locally.

External-agent bridge

With agent: "external", Karotte fetches each assistant message through long-poll GET /api/internal/get_message and removes it through POST /api/internal/delete_message. It runs tool calls as the student and streams events back through append_transcript and update_run_state.

This maps to OpenEnv reset()/step(): the trainer generates every token (white-box), and Karotte's tools and judges keep their protections. The proposed karotte_env server would also implement Karotte's backend API.

reset() returns the system prompt and step instructions, which arrive as message_added events. Since the protocol does not carry tool schemas, the bridge has to build them from Karotte's built-in tools and the definitions in src/environment/tools/. step() returns tool results or step rewards with any next-step instructions. done waits for terminal update_run_state.

A local prototype drove the real MessageLoopAgent, BackendSource, BackendClient and stream_transcript_to_backend from reset()/step() and passed 10/10 checks, including 4 concurrent episodes. The runner and tool server were emulated, with no sandbox. A run against a real Karotte image is still needed.

The bridge can run outside the sandbox with no Karotte changes. Two upstream additions would help: tool schemas in task_started and a way to firewall an extra port. The port setting would let the bridge run inside the all-in-one image needed for deployment on HF Spaces.

sequenceDiagram
    participant T as Trainer (TRL environment_factory)
    participant E as karotte_env (OpenEnv server + backend API)
    participant K as Karotte harness (VM/container root)
    participant M as MCP tools (student)
    participant J as Judge

    T->>E: reset()
    E->>K: Start run agent=external with backend_uri
    K->>E: append_transcript system prompt and instructions
    K->>E: GET get_message long-poll
    E-->>T: System prompt and instructions

    loop Turns with tool calls
        T->>E: step(assistant message)
        E-->>K: Assistant message
        K->>E: POST delete_message
        K->>M: Call tools as student
        M-->>K: Tool results
        K->>E: append_transcript tool events
        K->>E: GET get_message long-poll
        E-->>T: Tool observation
    end

    Note over T,K: Final task step
    T->>E: step(final message without tool calls)
    E-->>K: Final message
    K->>E: POST delete_message
    K->>K: Kill student processes + collect_submission
    K->>J: Grade submission in custody
    J-->>K: Score
    K->>E: append_transcript scoring event
    K->>E: append_transcript task_completed
    K->>E: update_run_state terminal
    E-->>T: reward and done

Tasks

• Run the five attacks against harbor_env (Docker backend) and against coding_tools_env. Record results in Karotte's table format and report native verification findings in Verify sandbox environments outside the agent's sandbox #1232.
• Prototype envs/karotte_env over the external-agent protocol. Run example-task through TRL's environment_factory against a real Karotte image. Replay the blog's symlink attack through step() and check for reward 0 and misbehavior in scoring metadata.
• Try Karotte's --proxy path through the Harbor capture proxy (RFC 006/012) to produce a TrainingTrace.
• Draft an RFC 008 amendment to implement the pending security graders and semantic.canary_floor, and specify a scripted adversarial panel seeded with these attacks.
• Ask Karotte upstream to include tool schemas in task_started and support configurable blocked ports or a plugin hook for in-sandbox bridges. Discuss the proposed Sandbox class in Turn Sandbox into a class that runc, gVisor and VM implement preferencemodel/karotte#25 as a seam for OpenEnv providers.
• Share the measurements with Preference Model and ask for an OpenEnv column in their comparison.

Related

• Harbor integration: serve Harbor task datasets through OpenEnv as trainable environments #1036, Verify sandbox environments outside the agent's sandbox #1232, Sandbox envs take their reward from a file the agent can write #1199 / Only accept reward.txt written during verification #1200
• RFC 008, RFC 006, RFC 012
• Karotte, docs, backend protocol, comparison blog