Mechanistic Effect of RL Post-Training on Pretrained Policies

Characterize how reinforcement learning with verifiable rewards changes the behavior of pretrained large language models for reasoning by determining whether RL primarily sharpens existing action preferences or discovers new, previously low-probability solution modes, and identify the conditions under which each behavior occurs.

Background

There are competing views on how RL affects pretrained reasoning models: some studies suggest RL mainly sharpens existing preferences, while others argue RL composes new skills. The lack of step-level supervision and massive action spaces in natural language make this mechanistic question difficult to resolve.

A precise characterization of how RL reshapes the inherited policy—at the level of distributions over actions and the structure of reasoning traces—would clarify when RL provides genuine discovery versus amplification and guide post-training strategies.