Can outcome-reward RL learn operator-like reasoning procedures from a finite, fixed prompt set?

Determine whether outcome-reward reinforcement learning applied to large language models under a finite, fixed prompt distribution and bounded training rollout lengths can learn an operator-like reasoning procedure that composes across iterations to solve problems, thereby avoiding harmful distribution shift when test-time horizons exceed those seen during training.

Background

The paper analyzes why standard outcome-reward RL for LLM reasoning, typically trained on a fixed set of prompts with a bounded rollout length, struggles to extrapolate to much longer test-time horizons. When models reason beyond training budgets, distribution shift arises because generation proceeds from conditional distributions not encountered during training.

The authors note that if a model had learned a true "operator"—a reasoning procedure that composes across iterations—this distribution shift would be less problematic. However, they explicitly state uncertainty about whether RL, under common training constraints, can learn such operators from a finite, fixed prompt set, motivating their alternative decoding and training approach (Reasoning Cache).