[burtenshaw]

This PR replaces #471 and references that blocked review thread.

It carries forward the session harness runtime, collection pipeline, env adapters, BrowserGym examples, and the latest review fixes from #471. The original PR currently has green checks but is blocked by a stale changes-requested review state.

Local verification before opening this replacement:

• PYTHONPATH=src:envs:examples uv run pytest tests/core/test_harness_runtime.py tests/core/test_harness_collect.py tests/envs/test_browsergym_harness.py tests/scripts/test_browsergym_harness_eval_examples.py tests/test_cli/test_collect.py tests/envs/test_openspiel_collect.py
• uv run usort format src/ tests/ && uv run ruff format src/ tests/ && uv run ruff check src/ tests/
• Targeted ruff checks for modified env/example files

Original PR: #471

[greptile-apps]

Greptile Summary

This PR introduces the session harness runtime (MCPHarnessAdapter, StepEnvSessionAdapter, SessionMCPBridge), a collection pipeline (CollectRunner, RolloutSerializer, EpisodeRecord), an openenv collect CLI command, and harness adapters for BrowserGym, OpenSpiel, and ReasoningGym. The dual-API boundary invariant \u2014 agents see only MCP tool calls while Gym reset/step/state stay in the orchestration layer \u2014 is correctly implemented and enforced by RESERVED_TOOL_NAMES checks at both the session and bridge level.

• Harness runtime: StepEnvSessionAdapter bridges agent tool calls to client.step(); MCPHarnessAdapter drives the white-box MCP loop with token/logprob accumulation; build_harness_rollout_func produces a TRL-compatible rollout function.
• Collect pipeline: CollectRunner orchestrates repeated rollouts into an append-only JSONL dataset with resume support, Rich progress display, and optional HF Hub push.
• ALIGNMENT FLAG: build_harness_rollout_func passes a trainer object into model_step_builder, coupling training-infrastructure state into the session loop in a way that differs from the evaluation path. This trade-off should be documented in RFC 005. Principle at stake: \u201cMinimize lifecycle deltas\u201d. Suggested reviewer: @darktex

Confidence Score: 3/5

The harness runtime and collection pipeline are well-structured, but two defects in the core loop need fixing before this is safe to merge for production use.

The max_tool_calls_per_turn truncation leaves tool-call IDs in the assistant message without matching tool-result entries, causing OpenAI-compatible APIs to reject the subsequent request. Separately, CollectResult.episode_ids returns all planned IDs rather than only those written to results.jsonl, so downstream consumers loading episodes by ID will hit missing-record errors. Both issues sit in central code paths shared by all three env adapters.

src/openenv/core/harness/init.py (tool-call truncation) and src/openenv/core/harness/collect.py (CollectResult.episode_ids semantics and _tasks iterator exhaustion)

Important Files Changed

Filename	Overview
src/openenv/core/harness/init.py	Core harness runtime with MCPHarnessAdapter, StepEnvSessionAdapter, and SessionMCPBridge. Dual-API boundary correctly enforced. One bug: max_tool_calls_per_turn truncation leaves orphaned tool-call IDs in the assistant message.
src/openenv/core/harness/collect.py	CollectRunner, RolloutSerializer, EpisodeRecord, and HF Hub helpers. Two issues: episode_ids contains all planned IDs not just collected ones; _tasks iterator is exhausted after the first run() call.
envs/browsergym_env/harness.py	BrowserGymSessionFactory with ast.parse-based action conversion and environment-forwarding verify builder. No issues found.
envs/openspiel_env/harness.py	OpenSpielSessionFactory with TTT board renderer. Clean implementation.
envs/reasoning_gym_env/harness.py	ReasoningGymSessionFactory with single answer tool and random seed fallback. No issues found.
src/openenv/cli/commands/collect.py	New openenv collect CLI command. system_prompt is passed redundantly to both LLMClient and build_model_step but is harmless. Otherwise clean.
src/openenv/core/llm_client.py	OpenAI and Anthropic clients with MCP schema converters. No issues found.
tests/core/test_harness_runtime.py	Comprehensive unit tests covering the full harness lifecycle and reward invariants. No issues found.

Sequence Diagram

sequenceDiagram
    participant TRL as TRL Trainer / CollectRunner
    participant HF as HarnessAdapter (MCPHarnessAdapter)
    participant SA as StepEnvSessionAdapter (ResourceSession)
    participant GC as Gym Client (step/reset)
    participant LLM as LLM (ModelStep)

    TRL->>SA: session_factory.create(task, episode_id)
    SA->>GC: client.reset(seed, episode_id, task_name)
    GC-->>SA: StepResult (initial observation)
    SA-->>TRL: ResourceSession

    TRL->>HF: run_white_box(model_step, session, limits)
    HF->>SA: session.initial_messages()
    SA-->>HF: user message with observation

    loop Each harness turn (max_turns)
        HF->>LLM: model_step(messages, tools, sampling)
        LLM-->>HF: ModelStepResult (tool_calls, logprobs)
        HF->>SA: session.call_tool(name, arguments)
        Note over SA,GC: Agent sees only tool calls, never reset/step/state
        SA->>GC: client.step(action)
        GC-->>SA: StepResult (observation, reward, done)
        SA-->>HF: ToolResult (data, done, metadata.reward)
        HF->>HF: append tool result message
        alt "done=True"
            HF-->>TRL: HarnessRolloutResult
        end
    end

    TRL->>SA: session.verify(transcript, final_state)
    SA-->>TRL: VerifyResult (env_reward forwarded)
    Note over TRL: _resolve_env_reward enforces rewards-in-env invariant
    TRL->>SA: session.close()

Loading

Prompt To Fix All With AI

Fix the following 4 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 4
src/openenv/core/harness/__init__.py:531-534
**Orphaned tool-call IDs when `max_tool_calls_per_turn` truncates**

`assistant_message` is built from `step_result.response.to_message_dict()`, which embeds **all** tool-call IDs returned by the model. Only the first `max_tool_calls_per_turn` tool calls then get a corresponding `role="tool"` result message. On the next iteration, the messages list is handed back to the LLM with N+M tool-call entries in the assistant turn but only N matching tool-result entries — the OpenAI API (and most OpenAI-compatible servers) treat this as a validation error and reject the request outright. The bug is dormant when the option is `None` (default), but fires whenever a caller sets `max_tool_calls_per_turn` and the model returns more calls than the limit.

### Issue 2 of 4
src/openenv/core/harness/collect.py:341-348
**`episode_ids` in `CollectResult` contains all planned IDs, not only collected ones**

`planned_ids` is the full range generated before the loop, so `CollectResult.episode_ids` will contain IDs for skipped, dropped, and failed episodes as well as collected ones. Any consumer that iterates `result.episode_ids` expecting a list of episodes present in `results.jsonl` will attempt to load records that don't exist there. Concretely, with `num_episodes=10`, `num_skipped=3`, `num_dropped=2`, and `num_failed=1`, only 4 IDs are in the file but all 10 appear in the field.

### Issue 3 of 4
src/openenv/core/harness/collect.py:215-228
**`_tasks` iterator is exhausted after the first `run()` call**

`CollectRunner.__init__` converts the `tasks` argument to an iterator once and stores it in `self._tasks`. If `run()` is called a second time (which the class docstring explicitly invites: "A single runner instance can be reused across runs"), every `_next_task()` call will return `None` because the iterator is spent — silently destroying task-to-episode alignment without any warning. Either the raw iterable should be stored and re-iterated on each `run()` call, or the docstring should be corrected to say the runner is stateful and must not be re-used when `tasks` is provided.

### Issue 4 of 4
src/openenv/core/harness/collect.py:328-334
**`avg` is recomputed on every collected episode**

`sum(rewards) / len(rewards)` re-traverses the growing `rewards` list on each iteration. An incremental running sum would be O(1) per step instead of O(n).

```suggestion
                self._serializer.write_episode(record)
                num_collected += 1
                rewards.append(record.reward)
                running_sum = running_sum + record.reward if num_collected > 1 else record.reward
                avg = running_sum / num_collected
                progress.update(
                    task_id, advance=1, collected=num_collected, avg_reward=avg
                )
```

_{Reviews (1): Last reviewed commit: "Merge branch 'main' into feature/harness..." | Re-trigger Greptile}

[greptile-apps]

Orphaned tool-call IDs when max_tool_calls_per_turn truncates

assistant_message is built from step_result.response.to_message_dict(), which embeds all tool-call IDs returned by the model. Only the first max_tool_calls_per_turn tool calls then get a corresponding role="tool" result message. On the next iteration, the messages list is handed back to the LLM with N+M tool-call entries in the assistant turn but only N matching tool-result entries — the OpenAI API (and most OpenAI-compatible servers) treat this as a validation error and reject the request outright. The bug is dormant when the option is None (default), but fires whenever a caller sets max_tool_calls_per_turn and the model returns more calls than the limit.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/openenv/core/harness/__init__.py
Line: 531-534

Comment:
**Orphaned tool-call IDs when `max_tool_calls_per_turn` truncates**

`assistant_message` is built from `step_result.response.to_message_dict()`, which embeds **all** tool-call IDs returned by the model. Only the first `max_tool_calls_per_turn` tool calls then get a corresponding `role="tool"` result message. On the next iteration, the messages list is handed back to the LLM with N+M tool-call entries in the assistant turn but only N matching tool-result entries — the OpenAI API (and most OpenAI-compatible servers) treat this as a validation error and reject the request outright. The bug is dormant when the option is `None` (default), but fires whenever a caller sets `max_tool_calls_per_turn` and the model returns more calls than the limit.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

episode_ids in CollectResult contains all planned IDs, not only collected ones

planned_ids is the full range generated before the loop, so CollectResult.episode_ids will contain IDs for skipped, dropped, and failed episodes as well as collected ones. Any consumer that iterates result.episode_ids expecting a list of episodes present in results.jsonl will attempt to load records that don't exist there. Concretely, with num_episodes=10, num_skipped=3, num_dropped=2, and num_failed=1, only 4 IDs are in the file but all 10 appear in the field.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/openenv/core/harness/collect.py
Line: 341-348

Comment:
**`episode_ids` in `CollectResult` contains all planned IDs, not only collected ones**

`planned_ids` is the full range generated before the loop, so `CollectResult.episode_ids` will contain IDs for skipped, dropped, and failed episodes as well as collected ones. Any consumer that iterates `result.episode_ids` expecting a list of episodes present in `results.jsonl` will attempt to load records that don't exist there. Concretely, with `num_episodes=10`, `num_skipped=3`, `num_dropped=2`, and `num_failed=1`, only 4 IDs are in the file but all 10 appear in the field.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

_tasks iterator is exhausted after the first run() call

CollectRunner.__init__ converts the tasks argument to an iterator once and stores it in self._tasks. If run() is called a second time (which the class docstring explicitly invites: "A single runner instance can be reused across runs"), every _next_task() call will return None because the iterator is spent — silently destroying task-to-episode alignment without any warning. Either the raw iterable should be stored and re-iterated on each run() call, or the docstring should be corrected to say the runner is stateful and must not be re-used when tasks is provided.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/openenv/core/harness/collect.py
Line: 215-228

Comment:
**`_tasks` iterator is exhausted after the first `run()` call**

`CollectRunner.__init__` converts the `tasks` argument to an iterator once and stores it in `self._tasks`. If `run()` is called a second time (which the class docstring explicitly invites: "A single runner instance can be reused across runs"), every `_next_task()` call will return `None` because the iterator is spent — silently destroying task-to-episode alignment without any warning. Either the raw iterable should be stored and re-iterated on each `run()` call, or the docstring should be corrected to say the runner is stateful and must not be re-used when `tasks` is provided.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

avg is recomputed on every collected episode

sum(rewards) / len(rewards) re-traverses the growing rewards list on each iteration. An incremental running sum would be O(1) per step instead of O(n).

Suggested change

	self._serializer.write_episode(record)
	num_collected += 1
	rewards.append(record.reward)
	avg = sum(rewards) / len(rewards)
	progress.update(
	task_id, advance=1, collected=num_collected, avg_reward=avg
	)
	self._serializer.write_episode(record)
	num_collected += 1
	rewards.append(record.reward)
	running_sum = running_sum + record.reward if num_collected > 1 else record.reward
	avg = running_sum / num_collected
	progress.update(
	task_id, advance=1, collected=num_collected, avg_reward=avg
	)

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/openenv/core/harness/collect.py
Line: 328-334

Comment:
**`avg` is recomputed on every collected episode**

`sum(rewards) / len(rewards)` re-traverses the growing `rewards` list on each iteration. An incremental running sum would be O(1) per step instead of O(n).

```suggestion
                self._serializer.write_episode(record)
                num_collected += 1
                rewards.append(record.reward)
                running_sum = running_sum + record.reward if num_collected > 1 else record.reward
                avg = running_sum / num_collected
                progress.update(
                    task_id, advance=1, collected=num_collected, avg_reward=avg
                )
```

How can I resolve this? If you propose a fix, please make it concise.