Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Reward Design

Abstract: This paper investigates Reinforcement Learning (RL) approaches to enhance the reasoning capabilities of LLM agents in long-horizon, multi-turn scenarios. Although RL algorithms such as Group Relative Policy Optimization (GRPO) and Proximal Policy Optimization (PPO) have been widely applied to train multi-turn LLM agents, they typically rely only on sparse outcome rewards and lack dense intermediate signals across multiple decision steps, limiting their performance on complex reasoning tasks. To bridge this gap, we present the first systematic study of \textit{turn-level reward design} for multi-turn RL algorithms and agent applications. By integrating turn-level rewards, we extend GRPO and PPO to their respective multi-turn variants, enabling fine-grained credit assignment. We conduct case studies on multi-turn reasoning-augmented search agents, where we carefully design two types of turn-level rewards: verifiable and LLM-as-judge. Our experiments on multi-turn search tasks demonstrate that incorporating well-designed turn-level rewards enables RL algorithms to significantly outperform baseline methods with trajectory-level rewards. Both training and validation reward curves illustrate that our method achieves \textit{greater stability}, \textit{faster convergence}, and \textit{higher accuracy}. Numerical results across diverse question-answering datasets further show that our approach consistently delivers highest answer correctness and 100\% format correctness.

• The paper introduces a paradigm shift from traditional trajectory-level rewards to turn-level reward design, enabling fine-grained credit assignment in multi-turn reasoning tasks.
• It presents adapted GRPO and PPO variants (MT-GRPO and MT-PPO) that integrate intermediate and outcome rewards to significantly improve training efficiency and agent performance.
• Experimental results show that turn-level rewards boost training stability, answer correctness, and overall task performance in complex, multi-turn interactive environments.

Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Reward Design

The paper "Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Reward Design" investigates reinforcement learning techniques to enhance the reasoning capabilities of LLM agents in scenarios requiring complex, multi-turn interactions. The study focuses on overcoming limitations of sparse outcome rewards by introducing systematic turn-level reward design to facilitate fine-grained credit assignment.

Introduction to Turn-Level Reward Design

Reinforcement Learning (RL) approaches such as Group Relative Policy Optimization (GRPO) and Proximal Policy Optimization (PPO) are traditionally used to train LLM agents but typically operate with trajectory-level rewards. This paper introduces a paradigm shift by systematically studying and implementing turn-level rewards to improve these RL algorithms' effectiveness. Turn-level rewards allow the agent to receive immediate feedback on intermediate steps of decision-making, crucial for refining complex reasoning tasks.

Figure 1: Curves for different training reward components during training with various algorithms. GRPO-OR denotes GRPO with outcome rewards while GPRO-MR indicates GRPO with merged outcome and intermediate rewards.

Implementation of Turn-Level GRPO and PPO Variants

GRPO with Turn-Level Rewards

GRPO traditionally estimates advantage on a trajectory level, ignoring intermediary actions' contributions during long-horizon tasks. The paper modifies GRPO to a new variant—multi-turn GRPO (MT-GRPO)—which accommodates turn-level rewards.

MT-GRPO incorporates iterative advantages for both intermediate and outcome results:

• Intermediate Rewards guide actions during the process.
• Outcome Rewards reflect final task success.

The MT-GRPO algorithm thus computes advantages by leveraging both:

• Current step outcome rewards.
• Intermediate turn-specific feedback.

The complexity arises from dynamically managing rewards per turn, demanding exponential rollout samples, which can be computationally prohibitive in highly interactive tasks.

PPO with Turn-Level Rewards

To address GRPO’s complexity, the paper proposes an adaptation of PPO—multi-turn PPO (MT-PPO)—which employs generalized advantage estimation (GAE) to compute token-level advantages efficiently. This allows assigning specific rewards at each turn without the overhead imposed by GRPO’s sample complexity.

MT-PPO utilizes a critic model for value estimation, thereby achieving scalable and efficient training with turn-specific signals rather than relying solely on sparse trajectory-level rewards.

Figure 2: Overview of the multi-turn reasoning-augmented search agent pipeline, highlighting interleaved reasoning and search iterations.

Case Studies in Multi-Turn Reasoning-Augmented Agents

The approach has been validated on reasoning-augmented search agents that integrate external searches into their decision-making processes. Here, two types of turn-level rewards were carefully crafted:

• Verifiable Rewards: Measure strict correctness, ensuring responses are formatted and content-valid.
• LLM-as-Judge Rewards: Leverage LLMs to evaluate nuanced response quality beyond binary validity.

Experiments demonstrated improved stability in training dynamics and higher accuracy with both types of rewards.

Figure 3: Training reward curves for PPO baselines and MT-PPO across datasets, showcasing stability and convergence improvements in MT-PPO.

Experimental Evaluation and Results

An extensive suite of experiments was conducted across multiple datasets, revealing substantial improvements in answer correctness and format adherence. The MT-PPO variant consistently outperformed conventional PPO algorithms, achieving close to 100% format correctness and demonstrable improvements in answer validity across domain-specific tasks.

Figure 4: Ablation studies showcasing the impact of search count rewards and maximum turn adjustments on performance.

Conclusion

The study establishes the importance of turn-level reward mechanisms in multi-turn agent interactions. By reformulating RL approaches to utilize turn-specific rewards, both training stability and interactive agent accuracy are significantly enhanced. This suggests broader applicability for turn-level reward frameworks in dynamic environments where complex agent decisions guide task completion.

Future work could expand these paradigms to diverse agent-based systems, refining interactions in domains such as complex software navigation, real-time user support interfaces, and adaptive content generation tasks.