Formalizing why POPE improves exploration on hard problems

Develop a formal theoretical framework that explains why Privileged On-Policy Exploration (POPE) improves exploration on hard problems for large language models, identifying the mechanisms by which conditioning on oracle-solution prefixes and instruction-following lead to learning that transfers from guided to unguided problem settings.

Background

POPE conditions on prefixes of oracle solutions to guide on-policy rollouts for hard reasoning problems, then trains on a mixture of guided and unguided prompts. Empirically, the learned behavior transfers to unguided problems, improving solvability where standard on-policy RL fails.

The paper provides empirical evidence and a conceptual mental model but no formal analysis of why this transfer occurs. The authors suggest that instruction-following and the induced overlap of internal states between guided and unguided rollouts are key, and explicitly call for a theoretical account.