Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Reward

Abstract: LLMs exhibit remarkable problem-solving abilities, but struggle with complex tasks due to static internal knowledge. Retrieval-Augmented Generation (RAG) enhances access to external information, yet remains limited in multi-hop reasoning and strategic search due to rigid workflows. Recent advancements in agentic deep research empower LLMs to autonomously reason, search, and synthesize information. However, current approaches relying on outcome-based reinforcement learning (RL) face critical issues such as conflicting gradients and reward sparsity, limiting performance gains and training efficiency. To address these, we first propose Atomic Thought, a novel LLM thinking paradigm that decomposes reasoning into fine-grained functional units. These units are supervised by Reasoning Reward Models (RRMs), which provide Atomic Thought Rewards (ATR) for fine-grained guidance. Building on this, we propose Atom-Searcher, a novel RL framework for agentic deep research that integrates Atomic Thought and ATR. Atom-Searcher uses a curriculum-inspired reward schedule, prioritizing process-level ATR early and transitioning to outcome rewards, accelerating convergence on effective reasoning paths. Experiments on seven benchmarks show consistent improvements over the state-of-the-art. Key advantages include: (1) Atom-Searcher scales computation at test-time. (2) Atomic Thought provides supervision anchors for RRMs, bridging deep research tasks and RRMs. (3) Atom-Searcher exhibits more interpretable, human-like reasoning patterns.

• The paper introduces a reinforcement learning framework that decomposes reasoning into atomic thoughts for precise reward assignment.
• It employs a hybrid reward strategy combining fine-grained atomic thought rewards with outcome-based metrics, resulting in significant benchmark improvements.
• The approach enhances interpretability and computational scaling, demonstrating superior performance in both in-domain and out-of-domain QA tasks.

Atom-Searcher: Fine-Grained Atomic Thought Reward for Agentic Deep Research

Introduction

Atom-Searcher introduces a reinforcement learning (RL) framework for agentic deep research that addresses the limitations of outcome-based RL in LLMs for complex reasoning and information synthesis tasks. The core innovation is the "Atomic Thought" paradigm, which decomposes reasoning into minimal, functionally coherent units, enabling fine-grained reward assignment via a Reasoning Reward Model (RRM). This approach mitigates gradient conflicts and reward sparsity, two major bottlenecks in prior RL-based agentic research systems. Atom-Searcher demonstrates consistent improvements over state-of-the-art (SOTA) baselines across seven in-domain and out-of-domain benchmarks, with additional benefits in interpretability and test-time computational scaling.

Framework Overview

Atom-Searcher operates in two main phases: (1) supervised fine-tuning (SFT) on an atomic thought-annotated dataset to endow the policy LLM with the ability to generate atomic thoughts, and (2) RL optimization using a hybrid reward that combines fine-grained atomic thought rewards (ATR) from an RRM with outcome-based rewards. The reward aggregation follows a curriculum-inspired, linearly decaying schedule, prioritizing process-level ATR early in training and gradually shifting focus to outcome rewards as the model's reasoning aligns with correct answers.

Figure 1: Overview of Atom-Searcher, illustrating atomic thought dataset construction, SFT, RRM-based ATR computation, and hybrid reward RL optimization.

Atomic Thought Paradigm

Atomic Thoughts are defined as the minimal, irreducible units of reasoning within an LLM's trajectory, encapsulated in dedicated tags (e.g., <atom-think>). Unlike manual decomposition, Atom-Searcher incentivizes the model to autonomously induce atomic thoughts, allowing for task-specific adaptation. This decomposition provides explicit supervision anchors for the RRM, enabling precise credit assignment to intermediate reasoning steps.

The agentic deep research trajectory is modeled as a finite-horizon MDP, with actions comprising atomic thought generation, search invocation, and answer generation. The state includes the retrieved content and action history, and the reward function integrates both ATR and outcome-based F1 metrics.

Reward Modeling and Aggregation

The RRM evaluates each atomic thought in the reasoning trajectory, producing a vector of fine-grained scores. These are aggregated (e.g., via averaging or weighted sum) to yield the ATR for the trajectory. The final reward for RL is a convex combination of ATR and the outcome reward, with the ATR weight $\alpha$ decaying linearly over training steps:

$$\alpha = 0.5 \times \left(1 - \frac{T}{T_{MAX}}\right)$$

$$R = \begin{cases} \alpha R_{atom} + (1-\alpha) R_{f1} & \text{if format is correct} \ -1 & \text{if format is incorrect} \end{cases}$$

This dynamic aggregation ensures strong process-level supervision during early exploration and reduces noise as the model converges.

RL Optimization and Implementation

Atom-Searcher employs Group Relative Policy Optimization (GRPO) for policy updates, leveraging a reference policy and multiple rollouts per prompt. Loss masking is applied to exclude externally retrieved content from the optimization objective, focusing updates on model-generated reasoning and search queries. To prevent entropy collapse, a sliding-window-based entropy regulation mechanism dynamically adjusts policy temperature based on recent entropy trends.

Implementation uses Qwen2.5-7B-Instruct as the backbone, with Qwen3-30B-A3B as the RRM. Training is conducted with a batch size of 512, 32 prompts per step, and 16 rollouts per prompt, supporting up to 10 tool calls per rollout.

Empirical Results

Atom-Searcher is evaluated on four in-domain (NQ, TQ, HotpotQA, 2Wiki) and three out-of-domain (MuSiQue, Bamboogle, PopQA) QA benchmarks. It consistently outperforms both prompt-based and RL-based baselines, including DeepResearcher and Search-R1, with notable gains:

• In-domain: Atom-Searcher achieves best results on TQ, HotpotQA, and 2Wiki, with improvements of 4.3%, 2.5%, and 12.1% over the next-best method, respectively. On average, it surpasses DeepResearcher by 8.5% across in-domain tasks.
• Out-of-domain: Atom-Searcher leads on MuSiQue and PopQA, with 1.8% and 3.7% improvements, and is within 0.4% of the best on Bamboogle. The average out-of-domain gain over DeepResearcher is 2.5%.

Ablation studies reveal that direct RRM supervision without atomic thought decomposition yields minimal benefit, while the full Atom-Searcher framework provides significant gains, confirming the necessity of atomic thought structuring for effective fine-grained reward modeling.

Interpretability and Reasoning Behavior

Case studies demonstrate that Atom-Searcher produces more interpretable, human-like reasoning traces, with explicit problem analysis, hypothesis generation, risk assessment, and strategic planning. It also triggers more search calls and generates longer, more detailed responses compared to DeepResearcher.

Figure 2: Case study comparing the reasoning behavior of Atom-Searcher (bottom) and DeepResearcher (top), highlighting deeper, more structured reasoning in Atom-Searcher.

Token frequency analysis further supports this, with Atom-Searcher responses dominated by tokens related to observation, action, hypothesis, and risk analysis, in contrast to the more generic tokens in DeepResearcher outputs.

Figure 3: Word cloud of token frequency in Atom-Searcher (a) and DeepResearcher (b) responses, showing greater focus on structured reasoning in Atom-Searcher.

Test-Time Scaling

Atom-Searcher demonstrates effective test-time scaling, generating 3.2x more response tokens, 2.6x more think tokens, and 1.24x more tool calls per response than DeepResearcher, without explicit incentives for longer outputs. This indicates enhanced exploration and information synthesis capabilities, critical for complex research tasks.

Prompts and Reward Model Design

The RRM is prompted with dedicated templates to assess both atomic thoughts and overall thought processes, ensuring alignment between reward signals and the atomic structure of reasoning.

Figure 4: Prompt for RRM to assess the Atomic Thoughts.

Figure 5: Prompt for RRM to assess the Thought Process.

Implications and Future Directions

Atom-Searcher advances the state of agentic deep research by enabling fine-grained, interpretable, and efficient reasoning in LLMs. The atomic thought paradigm provides a principled foundation for process-level supervision, which can be extended to other domains requiring complex, multi-step reasoning and tool use. The curriculum-inspired reward aggregation strategy offers a general template for integrating intermediate and outcome-based rewards in RL for LLMs.

Potential future directions include:

• Scaling atomic thought annotation and SFT to larger, more diverse datasets.
• Exploring alternative aggregation functions and adaptive weighting schedules for reward integration.
• Extending atomic thought decomposition to multi-agent and multimodal reasoning settings.
• Investigating the transferability of atomic thought structures across domains and tasks.

Conclusion

Atom-Searcher demonstrates that fine-grained, atomic thought-level reward modeling, combined with dynamic reward aggregation and robust RL optimization, yields substantial improvements in agentic deep research. The framework not only achieves SOTA performance across diverse benchmarks but also enhances interpretability and computational scalability, providing a strong foundation for future research in agentic LLMs and autonomous reasoning systems.