$\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving

Abstract: Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning performance of LLMs by increasing test-time compute. However, even after extensive RLVR training, such models still tend to generate unnecessary and low-quality steps in their chain-of-thought (CoT), leading to inefficient overthinking and lower answer quality. We show that when the initial direction or quality of the CoT is suboptimal, the model often fails to reach the correct answer, even after generating several times more tokens than when the initial CoT is well-initialized. To this end, we introduce Reinforcement Learning with Re-solving (Re$^2$), in which LLMs learn to flexibly abandon unproductive reasoning paths and restart the solution process when necessary, rather than always committing to a final answer. Re$^2$ applies pure reinforcement learning without any preliminary supervised fine-tuning, successfully amplifying the rare redo behavior in vanilla models from only 0.5% to over 30%. This leads to substantial performance gains over standard RLVR under the same training compute budget, and also demonstrates notable improvements in test-time performance as the number of samples increases.

• The paper introduces a pure RL approach (Re²) that enables LLMs to abandon unproductive reasoning paths and restart from scratch.
• It employs prefix group sampling and resolve reward estimation to optimize chain-of-thought trajectories, yielding significant accuracy improvements.
• Experimental results demonstrate consistent performance gains across multiple benchmarks, outperforming existing RLVR methods.

Reinforcement Learning with Re-solving (Re$^{2}$): A New Paradigm for LLM Reasoning

Problem Statement and Motivation

Post-training reinforcement learning (RL) with verifiable rewards (RLVR) is the prevailing method for enhancing the reasoning capacity of LLMs, mainly by incentivizing longer and more elaborate chain-of-thought (CoT) traces. While this has led to improvements in domains such as mathematical and scientific reasoning, RLVR-trained LLMs remain susceptible to inefficient, low-quality, and overextended reasoning. A critical phenomenon uncovered in this work is that once the initial reasoning path deviates from an optimal direction, LLMs find it exceedingly challenging to recover, even if allowed to generate significantly more tokens. This is due to the lack of explicit mechanisms for abandoning unproductive paths—a key limitation when compared to human problem-solving processes.

The Re$^{2}$ Framework

To address this, the paper introduces Reinforcement Learning with Re-solving (Re$^{2}$), a pure RL approach that explicitly endows LLMs with the ability to terminate unproductive reasoning and restart from scratch (i.e., re-solve). The core idea is to augment the RL objective with an action that allows the model to either commit to an answer or initiate a fresh reasoning trajectory when uncertainty or confusion is detected. This re-solving action is associated with a reward computed as the expected probability of success from restarting, empirically estimated from out-of-group completions during RL training.

Re$^{2}$ is distinguished by two main characteristics:

1. No Supervised Pre-finetuning: Unlike prior work using imitation learning or RL from supervised trajectories, Re$^{2}$ solely relies on RL, which amplifies the frequency of redo behavior from a negligible baseline to over 30%.
2. Granular Exploration and Trajectory Management: By sampling multiple partial reasoning prefixes and suffix continuations per query, Re$^{2}$ generates a rich set of possible pathways. Rewards are explicitly computed to reflect the rational trade-off between continuing a CoT versus restarting, and policy updates leverage mean-normalized advantages computed within each prefix group.

Empirical Analysis

Through comprehensive analysis using Qwen2.5-7B-Instruct and DeepScaleR-1.5B-Preview, it is shown that:

• Longer CoTs are negatively correlated with accuracy (for fixed problems), particularly when early reasoning is flawed.
• When incorrect prefixes are continued, recovery is rare—the earlier the error, the steeper the drop in downstream accuracy.
• Allowing re-solving transitions LLMs from the "always guess" regime (frequent incorrect final answers from uncertain paths) to the "restart when in doubt" regime, leading to more reliable and accurate solutions.

Experimental Results

Re$^{2}$ is evaluated on five challenging benchmarks: AIME24, AIME25, AMC23, GSM8K, and GPQA-Diamond, covering advanced mathematics, general scientific reasoning, and arithmetic. Several model sizes (3B–14B) and types (base, instruct, reasoning-optimized) are considered. Results demonstrate that Re$^{2}$ consistently outperforms both the underlying vanilla models and recent RLVR frameworks such as DAPO on every benchmark under the same training and inference compute budgets.

Representative Results (accuracy gains relative to DAPO):

• Qwen2.5-14B-Base: +5.5% accuracy on average across tasks.
• DeepSeek-R1-Distill-Llama-8B: +4.4%.
• Notable gains on the hardest mathematical competitions (AIME24/25; e.g., +10.3% on AIME24 for Qwen2.5-14B-Base).

Moreover, Re$^{2}$ unlocks new properties under test-time scaling: as the number of samples increases, accuracy continues to improve beyond the saturation point of prior RLVR methods. This effect is attributed to the increased probability of sampling at least one successful re-solving trajectory.

Training and Methodological Advances

• Prefix Group Sampling: For each input, multiple truncated prefixes are created, and each is extended with multiple suffix continuations, substantially increasing the space explored.
• Resolve Reward Estimation: Out-of-group suffix completions are used to empirically estimate the recourse reward for resolving, directly integrating observable model success rates rather than relying on value function approximators or heuristics.
• Behavioral Analysis: Re$^{2}$ rapidly learns to invoke the resolve action early in training, with the fraction of "re-solve" moves peaking and then stabilizing as the model differentiates more sharply between promising and unproductive reasoning.

Theoretical and Practical Implications

The fundamental implication is that test-time compute scaling alone (i.e., "more tokens at inference") is insufficient for robust reasoning unless accompanied by flexible trajectory management via re-solving. Re$^{2}$ demonstrates that RLVR systems benefit from explicitly modeling the meta-cognitive capacity to abandon and restart—obviating the need for complex backtracking heuristics or external value guidance. This not only improves pass@1 metrics, but more importantly raises the effective upper bound of LLM reasoning under practical compute limitations.

From a practical standpoint, Re$^{2}$ introduces only a modest overhead during RL rollouts due to the prefix/suffix sampling structure (∼11% increase in rollout time vs. DAPO), while yielding substantial improvements in both training and inference robustness. The method is compatible with a variety of architectures and token budgets and can be readily adapted for edge-optimized models or hardware-aware configurations, as shown on OpenPangu-Embedded-1B.

Limitations and Future Directions

While Re$^{2}$ significantly enhances LLM reasoning, the framework does introduce complexity in the management of resolve probabilities and may require multiple recursions to reach a correct answer in exceptionally hard tasks. Future work should investigate adaptive policies for triggering re-solve, applications to non-text modalities, and extensions toward search-intensive or multi-step tool use scenarios.

Conclusion

Re$^{2}$ establishes a new RL-based paradigm for LLM reasoning, where the model learns not only to generate longer CoTs, but to strategically restart when necessary, akin to human problem solvers. Experimental evidence across diverse and challenging benchmarks demonstrates that Re$^{2}$ yields consistent, statistically significant improvements over both vanilla and RLVR-trained LLMs in both mathematical and scientific domains. This approach opens promising directions for future research on flexible, self-aware reasoning in generative models and the development of LLMs with truly reliable and controllable "thinking" dynamics (2603.07197).