Repository navigation
RFC 006 proposal: standard multi-component rewards with scalar compatibility #623
Description
Summary
I want to propose RFC 006 to standardize multi-component rewards while keeping the existing scalar reward contract intact.
Problem
Today we optimize on a single scalar reward, which works for training but makes reward design/debugging harder in complex agentic tasks (e.g., coding with safety/process constraints). Teams can manually encode these pieces into one score, but there is no common schema to inspect and compare components across environments.
Proposal
Introduce a standard component representation (e.g., success/progress/penalty/shaping/binary) emitted in observation metadata, while preserving observation.reward as the scalar optimization target.
Why now
This is a core reward-contract extension that touches environment authoring, observability, and training diagnostics. It should be aligned via RFC before implementation.
Draft RFC
A full draft is prepared in rfcs/006-multi-component-rewards.md and is submitted via PR for review.
Related RFCs
- RFC 001 (abstractions / boundaries)
- RFC 002 (environment-computed rewards)
- RFC 004 (rubrics)
- RFC 005 (harness boundary and reward opacity)
Activity
adithya-s-k commentedon Aug 19, 2026
Closing this. I'm not going to pursue multi-component rewards as a standalone OpenEnv core-reward-contract extension right now. The direction it was solving for — inspectable, weightable reward components rather than one opaque scalar — is being carried on the training-framework side instead: the reward contract there already takes a scalar / per-step list / named-component
dictand weights it (environment-owned reward,reward_weights), which covers the observability/debugging goal without changing OpenEnv'sobservation.rewardcontract.Rather than leave a ratified-but-unimplemented RFC lingering, I'd rather close and re-file a tighter proposal if/when OpenEnv itself needs a first-class component schema. Closing the RFC PR (#624) alongside this; leaving the exploratory
repl_envimplementation (#625) as-is for reference.