How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1

Abstract: Deep Research agents tackle knowledge-intensive tasks through multi-round retrieval and decision-oriented generation. While reinforcement learning (RL) has been shown to improve performance in this paradigm, its contributions remain underexplored. To fully understand the role of RL, we conduct a systematic study along three decoupled dimensions: prompt template, reward function, and policy optimization. Our study reveals that: 1) the Fast Thinking template yields greater stability and better performance than the Slow Thinking template used in prior work; 2) the F1-based reward underperforms the EM due to training collapse driven by answer avoidance; this can be mitigated by incorporating action-level penalties, ultimately surpassing EM; 3) REINFORCE outperforms PPO while requiring fewer search actions, whereas GRPO shows the poorest stability among policy optimization methods. Building on these insights, we then introduce Search-R1++, a strong baseline that improves the performance of Search-R1 from 0.403 to 0.442 (Qwen2.5-7B) and 0.289 to 0.331 (Qwen2.5-3B). We hope that our findings can pave the way for more principled and reliable RL training strategies in Deep Research systems.

• The paper shows that using Fast Thinking prompts improves model stability and boosts accuracy (e.g., Qwen2.5-7B accuracy from 0.403 to 0.422).
• It demonstrates that F1 rewards with action-level penalties prevent answer avoidance and outperform traditional EM-based rewards.
• The analysis reveals that REINFORCE outperforms PPO and GRPO, delivering superior performance with fewer search actions in deep research agent training.

RL Training Practices for Deep Research Agents: Systematic Dissection and Unified Baselines

Overview of Deep Research Agent Training and Core Dimensions

This paper delivers a systematic analysis of RL training for Deep Research agents, decomposing the RL pipeline into prompt templates, reward functions, and policy optimization algorithms. The authors perform controlled ablations isolating each factor, leading to robust observations regarding stability, efficiency, and final accuracy in knowledge-intensive tasks such as open-domain QA and multi-hop reasoning. A new baseline, Search-R1++, is established based on the findings, which surpasses previous methods on standard benchmarks.

Figure 1: (a) RL training pipeline for Deep Research; (b) Decoupled overview of prompt template, reward function, and policy optimization for systematic analysis.

Prompt Templates: Reasoning Structure Versus Training Efficacy

A granular exploration into prompt design disconfirms the previously popular intuition that richer and longer reasoning traces yield stronger models in RL-driven research agents. The Slow Thinking template, which enforces frequent explicit reasoning via > tags, is systematically compared against the minimalist Fast Thinking template that prioritizes direct search and answer actions with minimal intermediate reasoning. Empirical results demonstrate a robust and consistent negative correlation between the amount of explicit reasoning and model accuracy across single-hop and multi-hop QA benchmarks.

The instability endemic to the Slow Thinking template is quantified: training collapses manifest as runaway increases in <think> tags and excessive response length, culminating in degenerate rollouts dominated by vacuous reasoning segments and decision paralysis.

Figure 2: (a) Training score evolution under Fast/Slow Thinking templates; (b) Response length growth; (c) Surge in <think> tags coincident with training collapse.

Switching to the Fast Thinking template improves average accuracy from 0.403 to 0.422 (Qwen2.5-7B) and ensures stable convergence. This is attributed to the Fast Thinking template limiting spurious reasoning expansion, focusing optimization more tightly on search and answer decision boundaries.

Reward Function: EM, F1, and the Role of Action-Level Penalties

The reward signal critically shapes agent strategies. The standard transition from strict EM-based rewards to F1-based objectives is found to induce instability and a notable mode collapse: policies optimize via answer refusal rather than solving the underlying task, as evidenced by a sharp drop in answer rates but constant accuracy among answered instances.

Figure 3: Overall accuracy, accuracy over answered samples, and answer rate (shaded area) under F1 reward, illustrating answer avoidance-induced collapse.

The collapse under F1 supervision is resolved by introducing lightweight action-level penalties that discourage omission of search and answer actions (F1+ reward). This stabilizes policy learning and enables F1-optimized agents to outperform EM-trained ones across evaluation metrics: for example, EM-based agents achieve 0.422 average EM whereas F1+ models reach 0.429.

Policy Optimization Algorithms: Comparative Analysis

A head-to-head comparison among PPO, GRPO, and classic REINFORCE (with Fast Thinking prompts and controlled rewards) demonstrates that:

• GRPO exhibits the least stability; training frequently collapses, especially in long-context, multi-hop regimes.
• REINFORCE converges to the best accuracy and lowest search action count, with more adaptive reasoning trajectories.
• PPO produces more stable learning than GRPO but is less efficient than REINFORCE, often learning unadaptively high rates of search invocation even for single-hop questions.

The paper attributes REINFORCE's advantage to its lack of dependence on high-variance group baselines or high-bias critic estimates, both problematic in the sparse reward, long-sequence setting of Deep Research.

Unified Baseline: Search-R1++ and Empirical Gains

Leveraging the above insights, Search-R1++ is composed of:

• The Fast Thinking prompt template,
• F1+ reward (F1 with action-level penalties),
• REINFORCE policy optimization.

This configuration decisively advances state-of-the-art on all QA benchmarks for both Qwen2.5-7B and Qwen2.5-3B, improving average EM accuracy from 0.403 to 0.442 and from 0.289 to 0.331, respectively. Notably, Search-R1++ preserves performance even on small models where other approaches collapse.

Discussion: Practical and Theoretical Implications

The systematic disentangling of Deep Research RL training elucidates that:

• Excessive explicit reasoning (Slow Thinking) harms both stability and generalization in the long-horizon RL setting of Deep Research.
• Outcome-only rewards (F1/EM) are inadequate due to the risk of degenerate avoidance behaviors; minimal action supervision is required for robust training.
• Classic REINFORCE, when carefully configured, still outperforms newer, more complex algorithms under sparse RL regimes.

These findings prompt a shift in research focus: from adding complexity (longer chains, richer tools, advanced optimizers) toward principled design of prompt, reward, and policy optimizers that reflect the actual learning dynamics and desired behaviors of agent systems. There is strong evidence that deliberate, unified design outpaces ad hoc complexity in the scaling and deployment of advanced Deep Research agents.

Conclusion

This study presents the most comprehensive ablation to date of RL training ingredients in Deep Research. The findings are immediately actionable: strict normalization of prompt template, reward structure, and policy optimization is necessary to enable stable, interpretable, and high-performing agents for long-horizon knowledge-intensive reasoning. The Search-R1++ baseline provides compelling empirical evidence of such best practices. These insights generalize to other agentic LLM scenarios, highlighting the importance of systemic modular analysis for RL-driven LLM systems.