In-Context Reinforcement Learning for Tool Use in Large Language Models

Abstract: While LLMs exhibit strong reasoning abilities, their performance on complex tasks is often constrained by the limitations of their internal knowledge. A compelling approach to overcome this challenge is to augment these models with external tools -- such as Python interpreters for mathematical computations or search engines for retrieving factual information. However, enabling models to use these tools effectively remains a significant challenge. Existing methods typically rely on cold-start pipelines that begin with supervised fine-tuning (SFT), followed by reinforcement learning (RL). These approaches often require substantial amounts of labeled data for SFT, which is expensive to annotate or synthesize. In this work, we propose In-Context Reinforcement Learning (ICRL), an RL-only framework that eliminates the need for SFT by leveraging few-shot prompting during the rollout stage of RL. Specifically, ICRL introduces in-context examples within the rollout prompts to teach the model how to invoke external tools. Furthermore, as training progresses, the number of in-context examples is gradually reduced, eventually reaching a zero-shot setting where the model learns to call tools independently. We conduct extensive experiments across a range of reasoning and tool-use benchmarks. Results show that ICRL achieves state-of-the-art performance, demonstrating its effectiveness as a scalable, data-efficient alternative to traditional SFT-based pipelines.

• The paper introduces an RL-only training protocol using in-context few-shot prompts to learn tool usage without supervised annotations.
• It demonstrates state-of-the-art performance on web-based QA, multi-hop reasoning, and code-related tasks, significantly outperforming SFT+RL baselines.
• The curriculum strategy reduces prompt support over time, and structured reward shaping ensures proper tool invocation and output formatting.

In-Context Reinforcement Learning for Tool Use in LLMs

Problem Formulation and Motivation

The limitations of LLMs in accessing up-to-date or domain-specific information have become apparent as applications scale to complex tasks that exceed training corpora coverage. The augmentation of LLMs with external tools—such as search engines for information retrieval or Python interpreters for code—and integrating tool usage reasoning into these models is essential for robust performance in compositional and knowledge-intensive scenarios. While previous approaches, notably cold-start SFT followed by RL, have facilitated such tool usage, they remain highly supervision-intensive, demanding large quantities of annotated trajectories. This annotation bottleneck constrains scalability and generalization across new domains.

The ICRL Framework

ICRL introduces an RL-only training protocol for tool-augmented LLMs, eschewing SFT entirely. The core innovation is the usage of in-context examples provided via few-shot prompting within RL rollouts to impart tool invocation schemas, step-wise reasoning decomposition, and output formatting constraints directly during policy learning. The process follows a curriculum: initial rollout prompts include $N$ few-shot prompts (typically $N=3$), which are then systematically reduced as the model acquires the underlying tool-use behaviors. The final phase is zero-shot, with the policy expected to operate without explicit prompt examples.

Formalization

Tool-augmented reasoning is modeled as an MDP. At each step, the LLM can reason, invoke a tool (with explicit XML-based tags for <search>, > , <answer>, etc.), or ingest observations from the tool. The RL loss is computed only on model-generated tokens, not on retrieved tool outputs (via loss masking). The ICRL paradigm uses a GRPO-based RL loss, balancing task accuracy with output format correctness, enforced through structured reward penalties for deviations from the expected tag syntax.

Experimental Results

ICRL is evaluated on web-based QA (TriviaQA, HotpotQA, 2Wiki, Musique, Bamboogle) and math/code reasoning benchmarks (AIME2024/2025). Models based on Qwen2.5-3B/7B/14B and Qwen3-8B are trained solely via ICRL, without SFT or labeled traces.

ICRL demonstrates state-of-the-art performance on all QA benchmarks, surpassing SFT+RL and direct RL baselines. For instance, on Qwen2.5-3B, ICRL achieves an average EM of 40.16, outperforming the strongest alternative (Search-R1) by +8.94. On Qwen2.5-7B, ICRL delivers 49.12 average EM, >7 points above ParallelSearch. The gains on multi-hop/complex reasoning tasks (e.g., 2Wiki, Musique) are particularly significant, with double-digit improvements in EM and robust convergence behavior as prompt supports diminish. Notably, ICRL achieves these results without any SFT or ground-truth tool-use annotation, highlighting its data efficiency and robust generalization.

On code-augmented math tasks, ICRL matches or exceeds ReTool (SFT+RL) on AIME2024/2025, despite ReTool’s dependence on cold-start SFT.

Theoretical and Practical Implications

ICRL's approach eliminates the critical dependency on high-quality labeled tool-use traces, fundamentally lowering the cost and broadening the applicability of tool-augmented LLMs. The method internalizes tool-use protocols through in-context RL, which implies that LLMs can autonomously acquire compositional reasoning policies given only black-box reward signals. Notably, the gradual reduction of few-shot prompt support (curriculum) leads to better multi-turn search behavior and prevents premature termination observed in finer-granularity ablation schedules.

Loss masking ensures RL optimization is not confounded by extrinsic or non-model-generated spans, while composite reward shaping stabilizes learning for long-horizon, sparse-reward tasks typical in multi-hop QA.

ICRL's zero-supervision property offers dramatic gains in data efficiency and deployability. It also provides a unification for reasoning over heterogeneous modalities (retrieval, code execution, etc.), as demonstrated by the transferability from search-based to code-based tasks.

Limitations and Future Directions

While ICRL delivers strong results in established tool-use benchmarks, extension to truly open-ended or lifelong learning settings is still investigational. The reward design, albeit more general and scalable than SFT labeling, relies on task end-point supervision (exact match), which may limit applicability in tasks with ambiguous or latent rewards. Handling real-world non-determinism in tool outputs and further reducing curricular prompt dependence, e.g., via meta-reinforcement learning or curriculum discovery, are promising research vectors.

In addition, scalability beyond current model sizes and modalities, robustness to reward misspecification, and more general tool chaining—all without intermediate natural language scaffolding—remain open research challenges.

Conclusion

ICRL establishes a scalable, supervision-light methodology for training tool-using LLMs. By leveraging in-context demonstrations during RL rollouts, it closes the gap between pure prompt-based generalization and RL-driven adaptation, achieving state-of-the-art results on complex QA and reasoning tasks with zero supervised annotation. The protocol generalizes to both search and code execution domains, setting a new standard for efficient tool-use acquisition in LLMs (2603.08068). These results motivate further exploration of in-context RL paradigms as the foundation for truly autonomous, compositional language agents.