E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoning

Abstract: While LLMs have demonstrated significant potential in Tool-Integrated Reasoning (TIR), existing training paradigms face significant limitations: Zero-RL suffers from inefficient exploration and mode degradation due to a lack of prior guidance, while SFT-then-RL is limited by high data costs and capability plateaus caused by low-entropy collapse. To address these challenges, we propose E3-TIR (Enhanced Experience Exploitation), a warm-up paradigm for the early stages of agent training. Specifically, we formulate training as the dynamic integration of three experience types: Expert Prefixes, Expert Guided, and Self-Exploration. By executing diverse branching exploration around expert "anchors" and employing a mix policy optimization mechanism, we effectively mitigate distribution shifts and resolve optimization conflicts arising from shared prefixes. Our method dynamically adapts the model's knowledge boundaries, effectively balancing exploration diversity with training efficiency.Experimental results demonstrate that E3-TIR achieves a 6 performance improvement over traditional paradigms on tool-use tasks, while requiring less than 10 of the synthetic data. Furthermore, in terms of ROI, a comprehensive metric integrating performance, data cost, and training efficiency we achieve a 1.46x gain compared to baselines. Code is available at https://github.com/yuki-younai/E3-TIR.

• The paper introduces E3-TIR, a framework that reformulates early-stage training through expert-guided branching and dynamic experience filtering.
• It demonstrates empirical gains with a 6% average performance improvement and significant data efficiency, reducing reliance on synthetic data by over 90%.
• The method employs mix advantage estimation and gradient detachment to enhance policy robustness, reduce tool-calling failures, and ensure faster convergence in RL tasks.

E3-TIR: Enhanced Experience Exploitation for Efficient Tool-Integrated Reasoning

Motivation and Background

Tool-Integrated Reasoning (TIR) has become foundational in aligning LLMs with complex, compositional reasoning tasks that demand both multi-step decision-making and external tool invocation for factual grounding or code execution. Despite progress, prevailing RL training regimes exhibit key deficiencies: Zero-RL delivers suboptimal exploration with slow convergence, while SFT-then-RL is characterized by high data costs, overfitting, and low-entropy collapse — ultimately resulting in capability rigidity and diminished sample diversity. These effects are underscored by observed performance collapses in SFT+RL and exploration bottlenecks in Zero-RL during early-stage RL training (Figure 1).

Figure 1: (a) Zero-RL: Inefficient exploration and slow convergence. (b) SFT+RL: Vulnerable to overfitting and performance collapse after a high start. (c) E: Enables rapid adaptation via expert guidance with stable, continuous growth.

In response, E3-TIR ("Enhanced Experience Exploitation for Tool-Integrated Reasoning") introduces a principled approach that reformulates early-stage agent training as a dynamic interplay of three experience modalities: expert prefixes, expert-guided exploration, and self-exploration. The framework orchestrates branching exploration anchored on high-entropy expert nodes and utilizes a hybrid policy optimization pipeline to mitigate distribution shift and resolve shared prefix conflicts, directly addressing the noted pathologies of prior paradigms.

Methodological Framework

Expert Experience-Based Rollout and Branch Sampling

E3-TIR’s rollout mechanism initiates from two distinct pools: $\mathcal{D}_{\text{self}}$ (self-exploration) and $\mathcal{D}_{\text{exp}}$ (expert-guided branches). High-entropy prefixes from expert trajectories are algorithmically selected as "anchors." From each anchor, a controlled number of branches are sampled and extended using the agent’s evolving policy. Empirically, this yields substantially greater reward variance and policy entropy versus naive self-exploration, confirming the informativeness and gradient value of guided branching (Figure 2).

Figure 2: Distribution of Standard Deviation and Entropy under Different sampling strategies.

A dynamic experience filtering step prunes redundant or homogeneous expert clusters based on intra-tree reward variance, and adaptively promotes self-exploration once its reward distribution surpasses that of expert branches. This balances exploitation of known-good patterns with the requirement for continual policy diversification and boundary expansion.

Mix Advantage Estimation and Policy Optimization

To address conflicting updates in overlapping prefix trajectories, E3-TIR introduces a dual-level advantage framework: (i) global advantage, measuring each trajectory against the pool mean; and (ii) expert-tree relative advantage, quantifying intra-tree performance differentials. Expert-guided trajectories leverage the sum of both terms, directing credit towards fine-grained reasoning innovations.

Mix policy optimization is then performed via an off-policy, hybrid-shaped objective: self-exploration data is processed on-policy, while expert-anchored data is corrected via importance sampling. Critically, gradient updates to shared prefixes are selectively detached for failed explorations—guaranteeing that only success-driven gradients refine shared structure, thus preventing negative transfer and ambiguous credit assignment. Ablation results highlight the necessity of advantage-aware gradient detaching for stability and learning efficacy (Figure 3).

Figure 3: Ablation analysis of clip ratio and gradient norm on different mix policy optimization.

Experimental Analysis

Performance and Efficiency

E3-TIR consistently surpasses SFT-only, SFT-then-RL, and Zero-RL baselines, evidencing average performance gains of 6% across mathematical and knowledge-intensive reasoning tasks on multiple LLM backbones. Notably, this improvement is achieved with less than 10% of the synthetic data typically required (Table 1, not shown in-essay for space). The composite ROI metric, which integrates accuracy, training step reduction, and data efficiency, demonstrates a 1.46× gain relative to conventional alternatives.

Figure 4: Entropy and reward curves on the warm-up stage, demonstrating accelerated convergence and maintained exploration diversity under expert-guided branching.

Behavior and Generalization

E3-TIR achieves tool-calling efficiency (average calls per query: 1.97 vs. 2.52 for Zero-RL) and sharply reduced failure rates (4.0% vs. 7.4%), confirmed through longitudinal tool invocation audits. Code/solve failure rates are systematically suppressed throughout training (Figure 5), validating the method's ability to extend the agent’s operational boundary.

Figure 5: Comparison of Code Failed Rate and Solve-None curves across different training stages.

In long-horizon, multi-turn agentic scenarios (e.g., ALFWorld, SCIWorld), E3-TIR yields the highest mean scores and demonstrates robust generalization, even outperforming Zero-RL on the more stateful ALFWorld benchmark—confirming that expert-anchored adaptation enables downstream flexibility without the rigidity observed in SFT-initialized agents.

Ablation and Cost-Benefit Analyses

Systematic ablations establish the primacy of expert-anchored branching and advantage-aware detachment; removal of either component precipitates learning collapse or substantial degradation. Mix SFT+RL variants and other offline fusion baselines (Luffy, HPT) underperform E3-TIR significantly (Figure 6).

Figure 6: (a) Cost-Benefit Heatmap during the Warm-up Stage. (b) Comparison with Other Mix Strategies.

ROI analyses show rapid saturation in benefit after modest expert data investment (~4k samples) and a non-monotonic relationship with warm-up budget, indicating that minimal expert input can boot the agent without sacrificing downstream exploration autonomy (Figure 7).

Figure 7: Variation of the sample size and rewards in the preheating stage of the mixed strategy.

Implications and Future Directions

E3-TIR’s methodological innovations—branching around expert anchors, hybrid advantage estimation, and dynamic experience filtering—represent a significant advance in RL-based TIR training, offering a practical route to data- and exploration-efficient optimization. Theoretical analysis confirms that expert-guided branching exponentially increases the probability of successful trajectory discovery as interaction horizon grows, directly overcoming the sparse reward curse in standard RL regimes.

Practically, E3-TIR enables faster and more reliable bootstrapping of TIR agents with minimal annotated expert data, facilitating real-world deployments where data curation is costly or slow. The proposed credit assignment and filtering mechanisms are generic and may be adapted for more sophisticated agent frameworks, multi-tool orchestration, or transfer to domains requiring persistent state across long horizons.

Limitations persist: reliance on high-quality, diverse expert prefixes to seed branching exposes sensitivity to coverage gaps in expert curation, and full generalization to broader interaction settings (e.g., heterogeneous environments demanding compositional tool use) remains an open challenge. Scaling E3-TIR to much larger LLMs and distributed RL architectures may further highlight data-parallel efficiency and broader applicability, especially as model and environment complexity grow.

Conclusion

E3-TIR provides an effective, theoretically justified, and empirically validated framework for early-stage TIR agent training that systematically overcomes the inefficiencies and failure modes of prior paradigms. By shifting from passive experience reuse to dynamically controlled, expert-anchored branching and gradient-aware hybrid optimization, it establishes stronger foundations for data- and training-efficient agentic reasoning and paves the way for further methodological refinement in agentic RL for LLMs.

Reference: "E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoning" (2604.09455)