Skip to content

feat(rl): shared choice-aware parser/decode layer for evaluation and reward (phase 7) - #9

Merged
xianyu9n merged 2 commits into
masterfrom
feat/rl-choice-aware-parser
Aug 19, 2026
Merged

feat(rl): shared choice-aware parser/decode layer for evaluation and reward (phase 7)#9
xianyu9n merged 2 commits into
masterfrom
feat/rl-choice-aware-parser

Conversation

@xianyu9n

@xianyu9n xianyu9n commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Overview

Phase 7 closes the task-level choice-to-reward contract: model outputs that speak choice ids (from PR #8) now flow through one shared, never-raising choice-aware parser that decodes them to canonical category_id before the existing evaluation / reward logic.

raw model output
  → check_stage1_choices / check_stage2_choices   (shared, never raises)
  → ChoiceParseResult.decoded  = canonical category_ids
  → existing evaluate_* / reward tables            (unchanged)

Changes

Shared choice-aware layer — src/agent/task/parser.py

  • check_stage1_choices: JSON/schema → exactly 5 → unique → choice_id ∈ prompt catalog → decode to canonical category_id tuple
  • check_stage2_choices: JSON/schema → answer ∈ "1".."5" (strict, no numeric coercion) → positional decode against the canonical candidates
  • ChoiceParseResult: format_valid / constraint_valid / decoded / model-level output / errors, plus canonical_view() → the PR6 ParseResult contract so reward keeps a single table implementation
  • Same contract style as PR6 (check_stage1/2_output): never raises on model output, programming errors still raise

Evaluation — src/agent/evaluation/classification.py

  • evaluate_stage1_choices / evaluate_stage2_choices now consume the shared layer directly (removed the inline parse/decode + re-serialize round-trip); prediction is the decoded canonical result

RL reward — src/agent/training/rl/reward.py

  • New reward_stage1_choices / reward_stage2_choices / reward_for_choice_result: decode via the shared layer, then the unchanged reward table — correct → 1.0, valid wrong → stage1_valid_miss 0.3 / stage2_partial 0.5, invalid / malformed → 0.0, never raises
  • RewardResult gains constraint_valid (additive) so "valid but wrong" is distinguishable from "invalid"; RewardConfig and all reward values unchanged

Tests (+33)

  • tests/task/test_parser_choices.py (17): valid decode, unknown/duplicate/wrong count, malformed JSON, answer 0/6/unknown, never-raise battery, canonical_view, programming errors
  • tests/rl/test_rl_reward_choices.py (16): correct/wrong/invalid/malformed reward, config unchanged, and evaluation ↔ reward validity agreement on identical outputs (parametrized, both stages)

Constraints respected

  • No prompt / candidate-policy / canonical ground-truth / registry / candidate-metadata changes
  • No parquet re-export, no token-stats recompute, no reward-value changes
  • No training run

Verification

  • Full suite: 233 passed, 2 skipped (skips = verl-compat, run in CI)

曾立宏 added 2 commits August 19, 2026 14:29
…reward (phase 7)

- agent.task.parser: new never-raise choice-aware layer
  (check_stage1_choices / check_stage2_choices -> ChoiceParseResult)
  validating the choice protocol (exact count, uniqueness, known
  choice ids / local id 1..5) and decoding to canonical category_ids;
  canonical_view() feeds the existing ParseResult contract.
- evaluation: evaluate_stage1_choices / evaluate_stage2_choices now
  consume the shared layer directly (no inline decode / re-serialize).
- rl reward: reward_stage1_choices / reward_stage2_choices /
  reward_for_choice_result apply the unchanged reward table after
  decode; RewardResult gains constraint_valid (additive) so
  valid-but-wrong is distinguishable from invalid. RewardConfig and
  reward values unchanged.
- tests: tests/task/test_parser_choices.py (17) +
  tests/rl/test_rl_reward_choices.py (16, incl. eval<->reward
  validity agreement on identical outputs).
- Evaluation module: Phase 6 = prompt-facing choice protocol
  (PromptChoiceRegistry, global Stage 1 ids, local Stage 2 ids);
  Phase 7 = shared choice-aware parser/decode layer consumed by
  evaluation and reward, completing the task-level choice-to-canonical
  reward contract.
- RL landing status: remove 'pending merge to master' for the merged
  prompt-choice-identity work; state the two phases as merged on master.
@xianyu9n
xianyu9n merged commit 6daefe8 into master Aug 19, 2026
3 checks passed
@xianyu9n
xianyu9n deleted the feat/rl-choice-aware-parser branch August 19, 2026 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant