feat(rl): shared choice-aware parser/decode layer for evaluation and reward (phase 7) - #9
Merged
Merged
Conversation
added 2 commits
August 19, 2026 14:29
…reward (phase 7) - agent.task.parser: new never-raise choice-aware layer (check_stage1_choices / check_stage2_choices -> ChoiceParseResult) validating the choice protocol (exact count, uniqueness, known choice ids / local id 1..5) and decoding to canonical category_ids; canonical_view() feeds the existing ParseResult contract. - evaluation: evaluate_stage1_choices / evaluate_stage2_choices now consume the shared layer directly (no inline decode / re-serialize). - rl reward: reward_stage1_choices / reward_stage2_choices / reward_for_choice_result apply the unchanged reward table after decode; RewardResult gains constraint_valid (additive) so valid-but-wrong is distinguishable from invalid. RewardConfig and reward values unchanged. - tests: tests/task/test_parser_choices.py (17) + tests/rl/test_rl_reward_choices.py (16, incl. eval<->reward validity agreement on identical outputs).
- Evaluation module: Phase 6 = prompt-facing choice protocol (PromptChoiceRegistry, global Stage 1 ids, local Stage 2 ids); Phase 7 = shared choice-aware parser/decode layer consumed by evaluation and reward, completing the task-level choice-to-canonical reward contract. - RL landing status: remove 'pending merge to master' for the merged prompt-choice-identity work; state the two phases as merged on master.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Phase 7 closes the task-level choice-to-reward contract: model outputs that speak choice ids (from PR #8) now flow through one shared, never-raising choice-aware parser that decodes them to canonical
category_idbefore the existing evaluation / reward logic.Changes
Shared choice-aware layer —
src/agent/task/parser.pycheck_stage1_choices: JSON/schema → exactly 5 → unique → choice_id ∈ prompt catalog → decode to canonicalcategory_idtuplecheck_stage2_choices: JSON/schema → answer ∈"1".."5"(strict, no numeric coercion) → positional decode against the canonical candidatesChoiceParseResult:format_valid/constraint_valid/decoded/ model-leveloutput/errors, pluscanonical_view()→ the PR6ParseResultcontract so reward keeps a single table implementationcheck_stage1/2_output): never raises on model output, programming errors still raiseEvaluation —
src/agent/evaluation/classification.pyevaluate_stage1_choices/evaluate_stage2_choicesnow consume the shared layer directly (removed the inline parse/decode + re-serialize round-trip);predictionis the decoded canonical resultRL reward —
src/agent/training/rl/reward.pyreward_stage1_choices/reward_stage2_choices/reward_for_choice_result: decode via the shared layer, then the unchanged reward table — correct → 1.0, valid wrong →stage1_valid_miss0.3 /stage2_partial0.5, invalid / malformed → 0.0, never raisesRewardResultgainsconstraint_valid(additive) so "valid but wrong" is distinguishable from "invalid";RewardConfigand all reward values unchangedTests (+33)
tests/task/test_parser_choices.py(17): valid decode, unknown/duplicate/wrong count, malformed JSON, answer 0/6/unknown, never-raise battery, canonical_view, programming errorstests/rl/test_rl_reward_choices.py(16): correct/wrong/invalid/malformed reward, config unchanged, and evaluation ↔ reward validity agreement on identical outputs (parametrized, both stages)Constraints respected
Verification
233 passed, 2 skipped(skips = verl-compat, run in CI)