Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL

Abstract: Large reasoning models (LRMs) are proficient at generating explicit, step-by-step reasoning sequences before producing final answers. However, such detailed reasoning can introduce substantial computational overhead and latency, particularly for simple problems. To address this over-thinking problem, we explore how to equip LRMs with adaptive thinking capabilities: enabling them to dynamically decide whether or not to engage in explicit reasoning based on problem complexity. Building on R1-style distilled models, we observe that inserting a simple ellipsis ("...") into the prompt can stochastically trigger either a thinking or no-thinking mode, revealing a latent controllability in the reasoning behavior. Leveraging this property, we propose AutoThink, a multi-stage reinforcement learning (RL) framework that progressively optimizes reasoning policies via stage-wise reward shaping. AutoThink learns to invoke explicit reasoning only when necessary, while defaulting to succinct responses for simpler tasks. Experiments on five mainstream mathematical benchmarks demonstrate that AutoThink achieves favorable accuracy-efficiency trade-offs compared to recent prompting and RL-based pruning methods. It can be seamlessly integrated into any R1-style model, including both distilled and further fine-tuned variants. Notably, AutoThink improves relative accuracy by 6.4 percent while reducing token usage by 52 percent on DeepSeek-R1-Distill-Qwen-1.5B, establishing a scalable and adaptive reasoning paradigm for LRMs. Project Page: https://github.com/ScienceOne-AI/AutoThink.

• The paper introduces AutoThink, a multi-stage RL framework that enables LLMs to decide dynamically when to engage in detailed reasoning.
• It uses an ellipsis prompt and structured reward shaping to improve accuracy by 6.4% while reducing token usage by 52% on benchmark tests.
• The method tailors reasoning depth to task complexity, reducing computational overhead and offering a scalable solution for resource-constrained environments.

Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL

The paper "Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL" focuses on enhancing the reasoning efficiency of LLMs by equipping them with the ability to dynamically decide whether to engage in detailed reasoning based on the complexity of the problem. This is achieved through a multi-stage reinforcement learning (RL) framework named AutoThink, which optimizes the reasoning policies via structured reward shaping.

Introduction to Adaptive Reasoning

Recent advancements in large reasoning models (LRMs) have been primarily focused on generating explicit, step-by-step reasoning sequences to improve accuracy in solving complex tasks. However, this approach can result in significant computational overhead due to unnecessary detailed reasoning on simpler problems. To mitigate this, the authors propose a mechanism for adaptive reasoning within LLMs, enabling them to balance accuracy and computational efficiency dynamically.

Figure 1: Overview of AutoThink Compared to Prior Reasoning Paradigms.

Method: Ellipsis Prompt and Multi-Stage RL Framework

The core of the proposed method involves a minimal prompting scheme using an ellipsis ("...") to trigger stochastic switching between thinking and no-thinking modes in R1-style models. This allows the models to decide autonomously whether explicit reasoning is necessary. The multi-stage RL framework refines this behavior over three stages:

1. Stage 1: Stabilizes the dual-mode behavior by balancing rewards for thinking and no-thinking responses to prevent mode collapse.
2. Stage 2: Reinforces reliable reasoning within each mode by allowing free evolution of reasoning policies, focused on enhancing accuracy.
3. Stage 3: Introduces length-aware pruning to encourage concise reasoning, ensuring tokens are used efficiently.

Figure 2: Accuracy and token usage with standard and ellipsis prompts.

Results and Analysis

Experiments demonstrated that AutoThink achieves better accuracy-efficiency trade-offs compared to existing methods. For instance, on the DeepSeek-R1-Distill-Qwen-1.5B model, AutoThink improved relative accuracy by 6.4% while reducing token usage by 52%. These improvements were consistent across different mathematical benchmarks.

Figure 3: Distribution of Reasoning Behaviors Across Models and Reasoning Modes.

The results underscore the effectiveness of the ellipsis prompt in activating adaptive reasoning while the RL framework ensures that reasoning depth is aligned with task difficulty.

Conclusion

The paper presents a novel approach to reasoning efficiently in LLMs by introducing an adaptive reasoning framework based on dynamic prompt adjustments coupled with a multi-stage RL training paradigm. The method promises significant improvements in computational efficiency without sacrificing accuracy, making it an attractive solution for deploying reasoning models in resource-constrained environments.

Overall, AutoThink offers a scalable and versatile mechanism for reasoning adaptation, capable of integrating seamlessly into existing R1-style models to enhance their utility in various practical applications. Future work could explore budget-aware CoT generation to further refine the control over reasoning depth and adaptively manage computational resources.