[rycerzes]

Summary

Adds an RFC for training the policy driving a black-box agent harness (Pi, OpenCode, Claude Code, …) inside OpenEnv by intercepting its LLM calls at an OpenAI-compatible boundary and recording a token-faithful trace — (prompt_ids, response_ids, loss_mask, response_logprobs, reward) — that TRL consumes via rollout_func.

This turns RFC 005's CLIHarnessAdapter.run_white_box stub (currently NotImplementedError) into a real seam. It is the installed-agent training path — the counterpart to the external-agent pattern TRL's Harbor integration already covers.

Design-only RFC PR — no runtime behavior changes.

What's in it

• Ownership boundary with TRL (OpenEnv owns interception + trace; TRL owns generation, weight sync, advantages, IS correction).
• 15 design decisions with rationale/trade-offs, each backed by cross-framework evidence (NVIDIA Polar/ProRL, verifiers, AReaL, Agent Lightning, rLLM) and recent literature (TIM, SAO, TITO, loss-mask/provenance work).
• Token-level trace contract, TITO discipline via TRL's chat_template_utils, prefix-break forking, session-ID-as-API-key auth, synthetic SSE replay, weight-sync fencing, proxy-enforced budgets.
• Mermaid diagrams: architecture, prefix-break forking, request lifecycle.

Relationship to other work

• Tracking issue: Agentic RL with black-box harnesses in OpenEnv via LLM-call interception #940
• Extends RFC 005 (rfcs/005-agentic-harnesses.md) — reconciles with it, does not parallel it.
• Supersedes draft [Draft] core: harness-rollout contract + runnable example for online RL with TRL #864 (message-level contract) with the token-level contract.

Note on numbering

Filed as 006, but 006 is also used by open PRs #794 and #624 — happy to renumber to whatever's free at merge time.

cc @burtenshaw @adithya-s-k @sergiopaniego

---

Note

Low Risk
Changes are limited to RFC and governance markdown; no runtime or training behavior is modified in this diff.

Overview
This PR is documentation-only: it adds rfcs/006-agentic-rl-harness-interception.md and updates GOVERNANCE.md and rfcs/README.md so RFC 006 is the agentic RL / harness interception spec, not the older “multi-component rewards” pointer.

The new RFC reframes the topic as the design rationale and contract for the capture stack already landed in #1036 (core/harness/capture/), reconciled with RFC 012. It documents the token capture contract (canonical prompt_token_ids / sampled ids + generation-time logprobs → (input_ids, loss_mask, logprobs, reward)), ownership vs TRL, and 19 design decisions, including corrections where implementation diverged from an earlier draft (OpenEnv assembles graphs without a tokenizer, prefix-linked graphs not chains, structural roles + ATIF, sandbox dials out to a trainer-side proxy).

Governance now links RFC 006 to this file (status In review), and the reward/eval persona bullet points eval authors at RFC 004 rubrics instead of RFC 006 multi-component rewards. Open work called out in the RFC includes consolidating opencode_env / pi_env onto core capture and TRL consuming capture/contract.py converters instead of re-tokenizing in _turns_from_trace.

^{Reviewed by Cursor Bugbot for commit 15fb7b2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[sergiopaniego]

linking huggingface/trl#6420 for considering it as it's directly related with this one.

[sergiopaniego]

@rycerzes solid RFC. One alignment point on the TRL seam: the harness-training path is being implemented in TRL as AsyncGRPOTrainer + HarnessRolloutWorker (huggingface/trl#6420), rather than GRPOTrainer's rollout_func as described here. I validated that branch end-to-end (OpenCode agents in remote HF sandboxes, in-sandbox proxy trace, GRPO), so it's working. Just flagging so you can take it into account and align the RFC's TRL integration section with it.

[Copilot]

Pull request overview

Adds a new design RFC describing an “installed-agent” RL training path for black-box CLI harnesses by intercepting OpenAI-compatible LLM traffic and emitting token-faithful traces consumable by TRL.

Changes:

• Introduces RFC 006 describing an interception server + trace recorder architecture for token-level rollouts.
• Defines the proposed trace contract (prompt_ids, response_ids, loss_mask, response_logprobs, reward) and key design decisions (TITO, prefix-break forking, session-as-API-key, etc.).
• Documents intended integration points with existing OpenEnv harness seams (CLIHarnessAdapter.run_white_box, build_harness_rollout_func) and TRL’s rollout_func.

Alignment Review Notes (per skill)

• Automated Checks
  • Lint: Not run (no facility to execute .claude/hooks/lint.sh in this review environment)
  • Debug code: Not run (no facility to execute .claude/hooks/check-debug.sh in this review environment)
• Open RFCs context (relevant)
  • RFC 005: In Review (rfcs/005-agentic-harnesses.md) — this RFC explicitly extends it.
  • Other RFCs in Draft/In Review exist but no direct conflict identified from this doc-only change.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[sergiopaniego]

Following up on my earlier note about the TRL seam. Now that huggingface/trl#6420 is merged, the installed-agent path in TRL is concretely AsyncGRPOTrainer + HarnessRolloutWorker (not GRPOTrainer.rollout_func / run_white_box as the RFC describes), so it would be worth aligning the integration section on that. Two more inputs from running it end to end (OpenCode + Pi, local subprocess and remote HF sandboxes):

1. Remote / managed-sandbox interception topology. The architecture assumes the harness can reach the trainer-side interception server. For agents in remote, network-isolated sandboxes (HF Sandbox, E2B) that needs spelling out: the sandbox can egress HTTPS but has no easy inbound. The shape we validated keeps the server trainer-side/colocated and exposes it to the sandbox through an egress tunnel. The harness only gets OPENAI_BASE_URL = tunnel/rollout/<sid>/v1 plus the per-rollout bearer, and nothing runs in the sandbox. Two properties. (a) Weight-sync fencing (D12) stays a local call (an in-sandbox recorder would make it distributed). (b) Less crosses the wire: the token-faithful data (ids, logprobs) is recorded trainer-side and never reaches the sandbox, only the OpenAI-dialect request and completion text do. Only the base-URL host differs between local and remote, so it generalizes cleanly. Happy to write this up as a subsection.

2. Where the TITO buffer should live. For context on the status quo: in the merged worker path, _turns_from_trace reconstructs each turn's prompt framework-side with apply_chat_template, while OpenEnv's proxy records only raw (messages, generation-time completion_token_ids, logprobs). That reconstruction is already faithful for prefix-preserving templates (vLLM re-renders per stateless call too, so both sides land on the same ids, consistent with the 18/19 you cite). TITO's real value is turning that into a guarantee and handling prefix-breaks (compaction, sub-agents) via forking, which is needed. The thing to weigh is placement. D3 puts TITO in openenv.core via trl.chat_template_utils, which couples core to a TRL internal. Since the reconstruction already lives framework-side today and OpenEnv stays a raw recorder (any trainer can consume the same trace), keeping the token-buffer on the framework side would preserve OpenEnv's framework-agnosticism, and the contract stays HarnessTrace. The counter-argument is buffer locality (verifiers/Polar assemble at the interception point), a real trade-off, just flagging it for a conscious call.

We have validated real OpenCode/Pi harnesses that could plug into the interception server, happy to help wire them up when it exists.

For reference, the in-sandbox vs trainer-side placement in a remote-sandbox run:

In-sandbox (what the merged path does today):
  [sandbox]  agent  →  in-sandbox proxy  →(tunnel)→  vLLM  [trainer host]
             the proxy records the trace inside the sandbox, trainer reads it back afterwards

Trainer-side (this RFC):
  [sandbox]  agent  →(tunnel)→  interception server  →  vLLM  [trainer host, localhost]
             the server records the trace on the trainer host, live, and owns the token buffer

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

rfcs/006-agentic-rl-harness-interception.md:7

• RFC numbering: RFC 010 explicitly states that numbers 006–009 are reserved for the self-improving-gym RFC family (rfcs/010-echo-env-token-world-model.md:8-11). Using RFC ID: 006 here (and the filename/README index) creates an in-repo inconsistency. Consider renumbering this RFC (and renaming the file + README entry), or updating RFC 010’s reservation note if that plan has changed.

**Created**: 2026-07-11
**Authors**: @rycerzes, @sergiopaniego
**RFC ID**: 006

rfcs/README.md:96

• This RFC index entry uses 006-..., but RFC 010 notes that IDs 006–009 are reserved (rfcs/010-echo-env-token-world-model.md:8-11). If that reservation still stands, this entry (and the RFC filename/header) should be renumbered to an unreserved ID to avoid conflicting documentation.

- [005-agentic-harnesses.md](./005-agentic-harnesses.md) - Agentic Harness Integration (OpenClaw, Claude Code, etc.)
- [006-agentic-rl-harness-interception.md](./006-agentic-rl-harness-interception.md) - Agentic RL through Harness Interception: token-faithful traces for TRL

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

rfcs/006-agentic-rl-harness-interception.md:6

• RFC ID 006 is currently also used by open PRs #624 and #794 (both add different rfcs/006-*.md files). Keeping this as 006 will force a renumber + file rename + README index update during merge conflict resolution; it’s safer to pick an unused RFC ID now (or mark the ID as TBD) to avoid collisions and broken cross-references.

**Created**: 2026-07-11
**Authors**: @rycerzes, @sergiopaniego
**RFC ID**: 006

[rycerzes]

@sergiopaniego Pushed a revision covering all three points.

• Seam — now AsyncGRPOTrainer + HarnessRolloutWorker; run_white_box is the secondary mode, rollout_func a sync bridge.
• Buffer — your call. My "verifiers and Polar assemble at the interception point" claim was wrong: Polar merges post-hoc in prefix_merging.py, verifiers in graph.py, TRL in _chain_to_sequences. None buffers in the proxy. OpenEnv is a recorder only.
• D3 adds one thing: inject return_token_ids and record the engine's canonical prompt_token_ids. Interstitials then come from slicing the next call's prompt after the EOT token (as Polar does) — nothing re-rendered, no tokenizer in core.
• That implies one TRL change: _turns_from_trace reading entry["prompt_token_ids"] instead of re-rendering, with a fallback. It currently passes neither chat_template nor chat_template_kwargs where _sample_turn passes both, so the training-template substitution is skipped and our enable_thinking=False injection is invisible. Divergent prompts make _chain_to_sequences FORK every turn and pair input_ids with logprobs from a different rendering.
• Topology — D18 + a section from your description, now grounded in huggingface_hub.Sandbox.proxy_url_for. Since that port proxy carries wss:// and streamed responses, the trainer can dial in to a thin in-sandbox relay rather than needing a publicly reachable address, so I wrote that up as the preferred transport with an outbound tunnel as the alternative — which of the two did you actually use? Also kept the in-sandbox proxy as a supported mode rather than legacy; it's the simpler deployment and both modes can share the recorder and TraceEntry.
• D12 fixed — the fence is TRL's WeightTransferClient.pause()/.resume() on a VLLM_SERVER_DEV_MODE=1 server, not trl vllm-serve (no pause endpoint there).

One question: D16 puts TraceEntry + the loop-owning session protocol in openenv.core.harness. Does that match your "contract stays HarnessTrace"? I made TraceEntry what OpenEnv emits and HarnessTrace the consumer-built target, since a framework-side buffer means we can't be emitting a loss_mask.

Yes please on wiring up the OpenCode/Pi harnesses once the server exists.

[rycerzes]

@sergiopaniego Rescoped the RFC against #1036 rather than patching it — it was arguing for a component that now exists, so it is now the rationale and contract spec for core/harness/capture/ plus what is left open. Status Draft → In Review.

Four decisions moved, three against what I argued last time.

• D3 — taking back the concession I made you. I agreed OpenEnv is a recorder only. It assembles now, and my reasoning was wrong in two places. The dependency argument assumed assembly means rendering; it is prompt_ids[len(parent.end_ids):] and concatenate along a path — no tokenizer, nothing added to core. And "everyone assembles post-hoc" is a claim about staging, not which repository: Polar merges inside Polar, verifiers inside verifiers. What decides placement is that retries, subagents and dead branches are only observable across all calls of a rollout, which a flat trace has already destroyed — the thing the agent_turn_fn TODO(@openenv) is complaining about. Framework-agnosticism survives through capture/contract.py shipping converters, not through refusing to compute.
• D4 was wrong. It is prompt_ids + sampled_ids vs the next prompt_ids, and a graph — not prompt-to-prompt chains, which cannot recover the interstitial span at all.
• D17 superseded. My call_kind-from-boundary-heuristics is the family that was tried and dropped. Structural roles plus the ATIF cross-check instead — the only validation in the stack that does not share our own assumptions.
• D18 — you were right, and my open question answers itself. I had preferred trainer-dials-in over Sandbox.proxy_url_for; the outbound tunnel you described is what shipped, with measured numbers behind it. RFC now says sandbox-dials-out.

New D19 for the four wire dialects, plus the hazard I had missed: processed_logprobs is taken after the logit processors, so a harness sampling with top_p<1 yields logprobs over a truncated, renormalised distribution while a full-vocabulary recompute gets different numbers for the same tokens.

One question, on the example: terminus-2 on terminal-bench@1.0 (host-side, no forward needed, and every path has n_tools == 0 — the branch that has already caused a mislabelling), or claude-code (rewrites its system prompt mid-run and opens a second root, so the graph sells itself)?

Still yes on wiring up the OpenCode/Pi harnesses once consolidation lands.

[rycerzes]

@sergiopaniego @adithya-s-k @burtenshaw, reconciled this with merged #1036 and RFC 012: D2 now adopts RFC 012's eval/train split, and D3, D13 and D16 are updated to match main. It's now the rationale and contract record for core/harness/capture.

Could you both take a final look for merge?

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[adithya-s-k]

@rycerzes @sergiopaniego the links for the Environments have to be updated to use the latest FineEnvs releases https://huggingface.co/collections/FineEnvs/smoldataenvs

[sergiopaniego]

lgtm with the changes requested by @adithya-s-k 🚀

[adithya-s-k]

i think it would make sense to merge both these RFCs(006 and 012) into a single one as there is an overlap
https://github.com/huggingface/OpenEnv/blob/main/rfcs/012-harbor-capture-providers.md

@sergiopaniego @burtenshaw how do we go about this ?

[sergiopaniego]

i think it would make sense to merge both these RFCs(006 and 012) into a single one as there is an overlap https://github.com/huggingface/OpenEnv/blob/main/rfcs/012-harbor-capture-providers.md

@sergiopaniego @burtenshaw how do we go about this ?

mmm we can actually combine both RFCs, you're right. 2 options: merge this one as is and then you can open a follow-up PR, or we can fold those sections into 006 here. Which one would you prefer @adithya-s-k?

Also @rycerzes, GOVERNANCE.md still lists RFC 006 as "multi-component rewards" (#623), so those two lines will need an update once this lands as 006

[adithya-s-k]

we can get this merged and i will create a consolidated PR, and i will be a pushing a few more things for harbor x openenv

[rycerzes]

Sounds good. Merging this as it is, with a consolidated 006 + 012 follow-up from @adithya-s-k, works for me.

@sergiopaniego, I've updated GOVERNANCE.md in this PR (8345daa). RFC 006 now points to this RFC in both the in-flight list and the RFC table. The reward/eval authors row still asked people to adopt the multi-component reward schema, but #623, #624 and #625 were closed as not planned, so it now says to expose reward components through the rubric tree (RFC 004), which is shipped and is what RFC 008's rubric checks read.

@adithya-s-k, could you review and re-approve?