This directory covers how reinforcement learning is applied to language models for code generation: specifically, using execution feedback (test pass/fail, runtime errors, linting) as reward signals rather than token-prediction loss alone. Token-level metrics like BLEU don't correlate reliably with functional correctness, which motivates RL approaches.
The PAPERS.md here is generated by ../../../tools/arxiv-collector/; it lists papers returned by arXiv queries on RL for code generation, filtered to 2022 onward.
PPO is the standard baseline for RL fine-tuning of code models. DPO (Rafailov et al., 2023, arXiv:2305.18290) and GRPO (Shao et al., 2024, arXiv:2402.03300) are more recent alternatives with lower infrastructure overhead.
For a survey of RL methods applied to code LLMs, see: Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Survey (Dec 2024, arXiv:2412.20367).
The most direct reward signal is unit-test pass rate. Secondary signals include execution errors, static analysis scores (Bandit, Semgrep), and code quality metrics (cyclomatic complexity, style adherence). Reward hacking (e.g., generating code that satisfies test harnesses but is otherwise incorrect) is a known problem; diverse test suites and KL penalties to a reference policy help.
Code generated during RL training must execute in isolated environments. Without sandboxing, a model can overwrite files, exhaust resources, or issue destructive API calls during exploration. Containers with resource limits (CPU, memory, network) and static analysis filters are standard practice.
- HumanEval: 164 function-level problems with unit tests (OpenAI).
- MBPP: ~1,000 crowd-sourced Python problems.
- APPS: 10,000 problems spanning interview to competition difficulty.
- SWE-bench: real GitHub issues requiring multi-file edits.
- LiveCodeBench: regularly refreshed to limit benchmark contamination.