FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization

Abstract: We present Future-KL Influenced Policy Optimization (FIPO), a reinforcement learning algorithm designed to overcome reasoning bottlenecks in LLMs. While GRPO style training scales effectively, it typically relies on outcome-based rewards (ORM) that distribute a global advantage uniformly across every token in a trajectory. We argue that this coarse-grained credit assignment imposes a performance ceiling by failing to distinguish critical logical pivots from trivial tokens. FIPO addresses this by incorporating discounted future-KL divergence into the policy update, creating a dense advantage formulation that re-weights tokens based on their influence on subsequent trajectory behavior. Empirically, FIPO enables models to break through the length stagnation seen in standard baselines. Evaluated on Qwen2.5-32B, FIPO extends the average chain-of-thought length from roughly 4,000 to over 10,000 tokens and increases AIME 2024 Pass@1 accuracy from 50.0% to a peak of 58.0% (converging at approximately 56.0\%). This outperforms both DeepSeek-R1-Zero-Math-32B (around 47.0%) and o1-mini (approximately 56.0%). Our results suggest that establishing dense advantage formulations is a vital path for evolving ORM-based algorithms to unlock the full reasoning potential of base models. We open-source our training system, built on the verl framework.

• The paper introduces FIPO, a novel RL algorithm that incorporates future-KL divergence to refine token-level credit assignment for improved reasoning.
• It re-weights token advantages to overcome reasoning bottlenecks, enabling longer chain-of-thought responses and enhanced task accuracy.
• Empirical results reveal a 6.0% Pass@1 improvement on AIME tasks, validating FIPO's efficiency in scaling deep reasoning capabilities.

FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization

Introduction

The paper introduces Future-KL Influenced Policy Optimization (FIPO), a reinforcement learning algorithm aimed at enhancing the reasoning capabilities of LLMs. FIPO addresses limitations observed in standard GRPO frameworks, which rely on outcome-based rewards leading to coarse-grained credit assignment. These traditional methods fail to distinguish critical reasoning steps, resulting in reasoning bottlenecks and performance ceilings. FIPO incorporates discounted future-KL divergence into policy updates, creating dense advantage formulations that re-weight tokens based on their subsequent trajectory influence. The empirical analysis presented in the paper demonstrates FIPO's ability to overcome length stagnation observed in traditional reinforcement learning models, significantly extending chain-of-thought (CoT) reasoning lengths and improving task accuracy.

Figure 1: FIPO vs. Baselines Performance Comparison on AIME2024. FIPO demonstrates that pure RL training alone is sufficient to not only outperform other pure RL baselines, but also surpass o1-mini.

Related Work

The paper builds on previous reinforcement learning methodologies for LLMs aimed at improving reasoning capabilities. The introduction of the RLVR framework by DeepSeek-R1 and the development of proprietary systems such as OpenAI's o-series have marked significant advancements. However, specific algorithms and training protocols can obscure the understanding of how reinforcement learning fundamentally enhances reasoning depth. Open-source initiatives like DAPO represent attempts to replicate such systems transparently, but GRPO's reliance on uniform token rewards imposes notable constraints.

In contrast, while recent methodologies like PPO introduce complexity through critic models for granular advantage estimation, FIPO achieves similar granular supervision without this complexity by refining GRPO directly. This research aims to bridge the gap between existing reinforcement algorithms by demonstrating that dense advantage formulations can effectively unlock deep reasoning capabilities in base models, thus challenging the notion that sophisticated critic models are necessary.

Methodology

Probability Shift: $\Delta \log p$

Central to FIPO is the concept of probability shift $\Delta \log p$, which serves as the atomic unit for credit assignment. This mechanism captures policy drift at a token level by measuring the log-space difference between current and previous policy probabilities. Positive shifts indicate reinforcement, while negative shifts signal suppression, enabling distinction between influential and trivial reasoning steps.

Future-KL Estimation

FIPO defines Future-KL as the cumulative probability shift from a token's current position until the end of its sequence. This forward-looking metric aggregates the drift throughout the trajectory, estimating a modified KL divergence that accounts for future influence. With an explicit focus on proximal signals facilitated by a soft decay window, Future-KL prioritizes local coherence and categorically excludes extreme value signals that destabilize optimization.

Advantage Re-weighting

The core innovation of FIPO re-weighs standard advantage estimations using calculated future influence weights, effectively amplifying positive advantage reinforcement for trajectories initiated by stable anchors. This modulation of policy updates is dimensionally controlled via influence weight clipping, ensuring numerical stability and preventing extreme log-probability shifts from introducing variance.

Experimentation and Results

In practice, the experimental setup utilizes DAPO's datasets for evaluation, ensuring rigorous comparison between baselines and FIPO on mathematical reasoning tasks like AIME 2024. The results signify a systematic improvement, with FIPO achieving approximately 6.0% higher Pass@1 metrics than DAPO.

Analysis reveals that FIPO not only facilitates increasing response length but also consistently bridges reasoning depth for problem-solving tasks of high difficulty. The dynamics of training show sustained upward trends in length-weighted mean advantages, establishing positive reinforcement cycles where longer responses lead to gains.

Figure 2: Dynamics of response length and performance scaling during training. Longer CoT is key to breaking performance barriers.

Figure 3: Analysis of training reward and length-weighted advantages. FIPO establishes a positive reinforcement cycle where longer responses increasingly yield advantages.

Conclusion

The research introduces FIPO as an efficient reinforcement learning approach to overcome intrinsic limitations in GRPO frameworks. By transforming sparse outcome-based rewards into dense token-specific supervision, FIPO successfully extends reasoning capabilities, pushing models beyond traditional length limitations. Validated empirically, FIPO extends average CoT length considerably, demonstrating that dense formulations can be realized effectively within existing GRPO frameworks. Through open-sourcing training code, the paper contributes scalable methodologies empowering further research advancements in large-scale reasoning models. Future avenues include improving computational resource efficiency, expanding task generalization, and exploring broader data distributions to refine dense advantage systems.