[jimmy-holdout]

Fixes #1551

Summary

There are three commits, one per environment in the issue. qed_math_env no longer sends the answer-mode reference solution to the agent. reasoning_gym_env and finqa_env stop scoring once the episode has ended. Later step() calls return the episode's final observation again, with the same reward, until reset().

• qed_math_env: reset() leaves ProblemObservation.reference_solution empty and get_problem leaves it out in both modes. Proof mode already worked this way. Grading still reads the reference from the server-side problem and the gold cache. The docstrings (the class, reset(), mcp_server.py), the reference_solution field description and the README now say the reference is not sent.
• reasoning_gym_env: the observation of the first scored answer is kept, and later step() calls return a copy of it until reset(), without scoring or counting a step. The episode reward therefore stays the scored answer's reward. That holds for the session's default verify(), which reads the last step, for _resolve_env_reward, which reads the last non-None tool reward, and for any loop that reads the last step. The existing "no current entry" branch is unchanged.
• finqa_env: step() and step_async() keep the terminal observation when an answer is submitted or max_steps is reached, and reset() clears it. Any later step, including ListToolsAction, returns a copy of it: no tool runs, nothing is scored, and step_count does not change. A submission on the last allowed step is still scored. ground_truth stays in the submit observation, which now comes only at the end of the episode.

Behaviour changes for callers:

• Code that read reference_solution from the qed_math_env reset observation or the get_problem payload in answer mode now gets "" or no key.
• In reasoning_gym_env and finqa_env, a step() after done now gets the final observation again instead of a new score or a tool result. This applies to the step() / WebSocket path. The production /mcp route calls tools directly and never issued a step reward.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation
• New environment
• Refactoring

Alignment Checklist

Before submitting, verify:

• I have read PRINCIPLES.md and this PR aligns with our principles
• I have checked INVARIANTS.md and no invariants are violated
• I have run the lint hook (lint.sh) and the tests, and addressed all issues

lint.sh stops at its ruff format --check src/ tests/ envs/ step on main too. It lists the same 47 files on main and on this branch: 46 under envs/ and src/openenv/core/harness/README.md. Three of them are READMEs edited here (qed_math, reasoning_gym, finqa). In those, ruff flags code blocks that this PR does not change. usort check reports only the two known files. ruff format --check on the changed Python files, and ruff check src/ tests/, pass. scripts/sync_env_docs.py --check passes. The three doc stubs were regenerated with --fix.

RFC Status

• Not required (bug fix, docs, minor refactoring)
• RFC exists: #___
• RFC needed (will create before merge)

Test Plan

New tests. The ones that check the fix fail on the parent commit and pass on their own commit (qed 2, reasoning_gym 2, finqa 6). The honest-path controls pass on both.

• tests/envs/test_qed_math_environment.py: test_answer_mode_hides_reference_solution (reset observation and get_problem) and test_auto_detected_answer_mode_hides_reference_solution (a row without a rubric). Control: test_answer_mode_correct_answer_still_scores (\boxed{4} still gets reward 1.0). test_answer_mode_includes_reference_solution is replaced.
• tests/envs/test_reasoning_gym_environment.py: wrong, then the revealed answer keeps reward 0.0; correct, then an extra step keeps 1.0. Control: reset() scores the next question normally. This file importorskips reasoning_gym, which the CI job does not install, so these tests run only where reasoning-gym is installed.
• tests/envs/test_finqa_environment.py: new TestEpisodeEnd, with synthetic data like TestResetSeed, so it runs in CI. Fix tests cover a submit after a submit, a submit after max_steps, and a ListToolsAction after a correct submit (reward stays 1.0 and step_count is unchanged), each through step() and step_async(). Controls: a submit on the last allowed step (1.0) and reset().

Runs:

• The CI test command from test.yml, run on a fresh clone with the CI job's extra packages plus reasoning-gym (Python 3.13): 4344 passed, 110 skipped on this branch, and 4331 passed, 110 skipped on main. The touched test files: 153 passed for qed (5 integration tests deselected), 30 passed for reasoning_gym, and 79 passed, 16 skipped for finqa. With the downloaded FinQA data linked at envs/finqa_env/data, all 95 finqa tests pass, including the existing data-dependent TestEnvironment.
• The repro scripts from the issue, before and after. Each env was run in-process under uvicorn. The reasoning_gym_env and finqa_env images were also built from their Dockerfiles and gave the same rewards. The finqa image was driven with raw /ws messages.

Case	main	this branch
qed: reference in reset observation / get_problem	full worked solution	"" / absent
qed: honest \boxed{66} / wrong \boxed{-12345}	1.0 / 0.0	1.0 / 0.0
reasoning_gym: step("x"), then the revealed answer	0.0, then 1.0	0.0, then 0.0 (not rescored)
reasoning_gym harness: wrong, then revealed (verify() / resolved reward)	1.0 / 1.0	0.0 / 0.0
reasoning_gym harness: correct, then one extra call (verify() / resolved reward)	0.0 / 0.0	1.0 / 1.0
finqa: submit_answer("0"), then the returned ground_truth	0.0, then 1.0	0.0, then 0.0 (not rescored)
finqa: submit after FINQA_MAX_STEPS=3 ended the episode	1.0	0.0 (not rescored)
finqa: honest first submission	1.0	1.0

---

Note

Medium Risk
Changes evaluation/training reward semantics and removes answer-mode reference exposure; callers depending on post-done rescoring or client-side gold answers need to adapt.

Overview
Fixes reward leakage and double-scoring across three environments.

qed_math_env no longer sends the ground-truth reference to the agent. reset() / ProblemObservation always expose reference_solution as empty, and get_problem drops it for proof and answer modes alike. Server-side grading and the gold cache are unchanged; docs and field descriptions now state the reference stays server-side only.

reasoning_gym_env and finqa_env treat the first terminal step as final: they cache the ending observation and return a deep copy on later step() calls (including finqa_env step_async / WebSocket) without re-scoring, running MCP tools, or bumping finqa_env step_count. reset() clears the cache. Submit on the last allowed FinQA step is still scored once.

Callers that read answer-mode references from QED or submit again after done=True (e.g. using revealed ground_truth or correct_answer) will see different rewards than before. READMEs and synced env docs describe the new semantics; tests cover hiding references, post-done steps, and async FinQA paths.

^{Reviewed by Cursor Bugbot for commit 3bdfdba. Bugbot is set up for automated code reviews on this repo. Configure here.}

[burtenshaw]

No blocking issues on 9920bc0e.

What I ran:

• The three env suites (tests/envs/test_qed_math_environment.py, test_reasoning_gym_environment.py with reasoning-gym installed, and test_finqa_environment.py): 262 passed, 16 skipped, 5 deselected. With each env's server file restored to its parent commit, the new fix tests fail (qed 2, reasoning_gym 2, finqa 6) and the controls pass.
• The harness and collect tests that resolve rewards (tests/core/test_harness_collect.py, test_harness_rollout_backcompat.py, test_harness_runtime.py, tests/envs/test_openspiel_collect.py, tests/test_cli/test_collect.py): 164 passed.
• reasoning_gym_env under uvicorn, with ReasoningGymSessionFactory behind SessionMCPBridge and seed 1. A wrong answer followed by the revealed answer gives step rewards [0.0, 0.0], verify() 0.0 and _resolve_env_reward 0.0. The parent commit gives [0.0, 1.0], 1.0 and 1.0. A correct answer followed by one extra call gives [1.0, 1.0], 1.0 and 1.0. The parent gives [1.0, 0.0], 0.0 and 0.0.
• finqa_env under uvicorn with FINQA_MAX_STEPS=3, driven over the WebSocket session. Resubmitting the returned ground_truth scores 0.0, and step_count stays at 1. A submission after the max-steps end also scores 0.0. The parent commit scores both 1.0. call_tool("submit_answer") over /mcp returns only the confirmation string, so that route never scored.
• A qed_math_env answer-mode row served with create_app, with its gold string searched for in 16 outputs: the WS reset, /mcp get_problem, get_grading_guidelines and submit_proof, WS step for get_problem and submit_proof, state, HTTP /reset, /state, /schema, /metadata, /web/reset, /web/step, /web/state, /web/metadata and the Task API. None contain it. On the parent commit, 6 of them do.
• Consumers: QEDMathEnv._as_problem_observation (envs/qed_math_env/client.py:79) builds ProblemObservation(**payload), and the field defaults to "", so the missing key is safe. Nothing in examples, docs or integration code reads reference_solution. Every caller of reasoning_gym_env or finqa_env in the repo stops at done.

Follow-ups, not blocking:

1. qed_math_env keeps scoring after done, which is the bug this PR fixes in the other two envs. submit_proof_payload (envs/qed_math_env/server/qed_math_environment.py:1348-1376) grades every call and does not check whether the episode has ended. On this branch, with an answer-mode row whose gold is 73441, step_async(submit_proof("\boxed{1}")) returns reward 0.0 with done=True. A second step_async(submit_proof("\boxed{73441}")) then returns reward 1.0 with done=True. With the reference hidden, answer mode allows only blind re-guessing. In proof mode the grader prompt contains the reference (envs/qed_math_env/server/rubric.py:438 and :454-455) and the grader feedback goes back to the agent, so a second submission can use it. A fix must let multi_step problems keep scoring until their own done. A follow-up issue is enough for this PR.
2. Optional: envs/thinkingbox_env/server/thinkingbox_environment.py:211 already keeps the terminal observation as _terminal_observation. Using that name instead of _final_observation in both envs keeps the repeated pattern aligned, as CLAUDE.md asks.