Maximum Likelihood Reinforcement Learning

Abstract: Reinforcement learning is the method of choice to train models in sampling-based setups with binary outcome feedback, such as navigation, code generation, and mathematical problem solving. In such settings, models implicitly induce a likelihood over correct rollouts. However, we observe that reinforcement learning does not maximize this likelihood, and instead optimizes only a lower-order approximation. Inspired by this observation, we introduce Maximum Likelihood Reinforcement Learning (MaxRL), a sampling-based framework to approximate maximum likelihood using reinforcement learning techniques. MaxRL addresses the challenges of non-differentiable sampling by defining a compute-indexed family of sample-based objectives that interpolate between standard reinforcement learning and exact maximum likelihood as additional sampling compute is allocated. The resulting objectives admit a simple, unbiased policy-gradient estimator and converge to maximum likelihood optimization in the infinite-compute limit. Empirically, we show that MaxRL Pareto-dominates existing methods in all models and tasks we tested, achieving up to 20x test-time scaling efficiency gains compared to its GRPO-trained counterpart. We also observe MaxRL to scale better with additional data and compute. Our results suggest MaxRL is a promising framework for scaling RL training in correctness based settings.

• The paper introduces MaxRL as a framework that optimizes likelihood over binary correctness, addressing core RL limitations in tasks like code generation and reasoning.
• It employs a Maclaurin expansion for the log-likelihood to derive an unbiased REINFORCE-style estimator that reweights gradients based on pass@k performance.
• Empirical results demonstrate that MaxRL outperforms standard RL and GRPO across diverse tasks, including ImageNet classification, maze navigation, and large-scale mathematical reasoning.

Maximum Likelihood Reinforcement Learning: A Principled Bridge Between Supervised and RL Optimization

Introduction and Formalization

The paper "Maximum Likelihood Reinforcement Learning" (2602.02710) introduces MaxRL, a general framework for training models in correctness-based, sampling-intensive regimes where binary rewards are provided by an external verifier. It targets a core limitation in mainstream RL post-training for tasks such as code generation, navigation, and mathematical reasoning: despite relying on an implicit likelihood over successful generations, standard RL objectives do not truly optimize likelihood, but only its first-order (expected reward) approximation.

Formally, for a latent generation model $m_\theta(z|x)$ with deterministic decoding $y=f(z)$ and correctness verification, traditional RL maximizes expected pass rate $p_\theta(x)$: $$J_{\text{RL}}(\theta) = \mathbb{E}_{x\sim\rho}[p_\theta(x)]$$ In contrast, the correct end-to-end principle is maximum likelihood over binary correctness, i.e.,

$$J_{\text{ML}}(\theta) = \mathbb{E}_{x\sim\rho}[\log p_\theta(x)]$$

The log-likelihood objective yields gradient reweighting that places maximal learning focus on hard (low-pass-rate) examples. This discrepancy induces fundamentally divergent optimization trajectories, evidenced empirically in a variety of domains.

MaxRL: Objective Derivation and Estimation

MaxRL exploits the Maclaurin expansion for the log-likelihood: $\log p = -\sum_{k=1}^\infty \frac{(1-p)^k}{k}$ and its gradient: $$\nabla_\theta J_{\text{ML}}(x) = \sum_{k=1}^\infty \frac{1}{k} \nabla_\theta\, \text{pass@}k(x)$$ Thus, likelihood optimization becomes a harmonic mixture of pass@k gradients, naturally recovering RL (pass@1) as a special case and making explicit the tradeoff between computational budget and objective fidelity.

MaxRL defines a truncated series parameterized by sampling compute $T$: $$J^{(T)}_{\text{MaxRL}}(x) = -\sum_{k=1}^T \frac{(1-p)^k}{k}$$ with an unbiased REINFORCE-style sample-based estimator that averages score functions only over successful trajectories. Critically, increasing the number of rollouts $N$ increases the truncation order $T=N$, so additional sampling not only reduces variance but moves the objective closer to exact likelihood. This estimator is provably unbiased for $J^{(N)}_{\text{MaxRL}}$.

Unifying View: Weight Functions

All objectives considered—standard RL, GRPO, MaxRL, and ML—can be written as: $$\nabla_\theta J = \mathbb{E}_x[w(p_\theta(x)) \nabla_\theta p_\theta(x)]$$ MaxRL’s weight function interpolates between RL ($w(p)=1$) and ML ($w(p)=1/p$), with intermediate $T$: $w_T(p) = \frac{1-(1-p)^T}{p}$ GRPO, while using Z-normalization, does not approach ML weighting even with increased rollouts. Thus, only MaxRL admits objective-level approximation improvement as compute scales.

Experimental Analysis

Differentiable Regime: ImageNet Classification

In differentiable settings (ResNet-50 on ImageNet), direct ML training (cross-entropy) is feasible. MaxRL, as rollouts per sample increase, tracks exact ML and outperforms RL and GRPO, which fail to significantly improve due to low initial pass rates and insufficient gradient signals.

Figure 1: On ImageNet, MaxRL closely matches ML training with sufficient rollouts, whereas RL stagnates due to vanishing gradients on hard examples.

Non-Differentiable Regimes: Maze Navigation and Reasoning

Infinite Data Regime

In procedurally generated mazes, MaxRL achieves superior scaling with increased rollouts, decisively outperforming RLOO and GRPO both in pass@1 and pass@k. Even with lower rollout budgets, MaxRL achieves higher coverage compared to competing methods trained with an order of magnitude more samples.

Figure 2: MaxRL maintains efficient scaling with increased rollout count, outperforming GRPO and RLOO in reasoning tasks.

Scarce Data Regime: GSM8K

In reinforcement post-training over a fixed dataset, competitive baselines demonstrate accelerated overfitting, with pass@k collapsing as training extends. MaxRL, despite slower early dynamics, consistently retains coverage and converges to higher final performance across metrics.

Figure 3: GSM8K experiments illustrate MaxRL's resilience to overfitting and coverage collapse compared to GRPO and RL.

Large-Scale LLM Mathematical Reasoning

MaxRL, when deployed on billion-parameter LLMs (Qwen3-1.7B/4B) for post-training on advanced mathematical datasets, Pareto-dominates GRPO on coverage across four benchmarks. Notably, it enables up to $20\times$ inference speedup given a perfect verifier, by reducing pass@k degradation and maintaining high diversity, which translates directly to real-world efficiency gains in domains with verifiable outputs.

Figure 4: MaxRL delivers Pareto improvements and major speedup in Qwen3 post-training, with stable coverage for large k.

Analysis of Optimization Dynamics

MaxRL’s reweighting intensifies gradients on hard, low-pass-rate examples, in precise correspondence to cross-entropy in classical ML. This leads to broader exploration of the data manifold and increased fraction of prompts with at least one correct rollout during training.

Figure 5: MaxRL induces high gradient norms for hard prompts, matching the behavior of cross-entropy.

Figure 6: MaxRL consistently solves a larger fraction of problems during training compared to alternatives.

Theoretical and Practical Implications

Theoretical impact: The recognition that standard RL objectives approximate a lower-order expansion of the ideal ML criterion elucidates a broad spectrum of RL failures in compositional generation and structured reasoning tasks. MaxRL closes this gap without compromising sampling-based feasibility, and can, in theory, be generalized further to other reward structures.

Practical impact: MaxRL’s scaling behavior fundamentally changes the cost-performance tradeoff in self-improvement, program synthesis, mathematical reasoning, and navigation tasks. It eliminates the empirical tradeoff between pass@1 and pass@k observed in prior RLVR protocols, and enables more efficient inference—even in high-dimensional or data-constrained regimes.

Contradictory claims: The work shows that pass@k degradation (mode collapse and coverage loss), previously attributed to RL or optimization artifacts, is largely a consequence of objective selection, not fundamental limitations of RL as an algorithmic class.

Open Directions

MaxRL is presently developed for binary correctness settings; generalizations to continuous rewards, off-policy learning, and multi-turn sequential decision-making (including PPO/actor-critic variants), are open and promising. Direct expansion to preference-based RLHF, reinforcement from textual feedback, and reward modeling protocols could leverage similar analytic decompositions to correct objective misspecification. Further research can investigate MaxRL in domains with complex verification, reward ambiguity, or hierarchical correctness.

Conclusion

Maximum Likelihood Reinforcement Learning is a principled, computationally-controllable synthesis of supervised and reinforcement objectives. It uniquely addresses coverage collapse, sample efficiency, and optimization dynamic limitations endemic to standard RL-based post-training and unlocks substantial improvements across large-scale reasoning tasks. Beyond binary correctness, its analytic machinery could serve as a template for future robust objective design in scalable RL and LLM post-training.