Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents

Abstract: LLMs are increasingly being used for complex problems which are not necessarily resolved in a single response, but require interacting with an environment to acquire information. In these scenarios, LLMs must reason about inherent cost-uncertainty tradeoffs in when to stop exploring and commit to an answer. For instance, on a programming task, an LLM should test a generated code snippet if it is uncertain about the correctness of that code; the cost of writing a test is nonzero, but typically lower than the cost of making a mistake. In this work, we show that we can induce LLMs to explicitly reason about balancing these cost-uncertainty tradeoffs, then perform more optimal environment exploration. We formalize multiple tasks, including information retrieval and coding, as sequential decision-making problems under uncertainty. Each problem has latent environment state that can be reasoned about via a prior which is passed to the LLM agent. We introduce a framework called Calibrate-Then-Act (CTA), where we feed the LLM this additional context to enable it to act more optimally. This improvement is preserved even under RL training of both the baseline and CTA. Our results on information-seeking QA and on a simplified coding task show that making cost-benefit tradeoffs explicit with CTA can help agents discover more optimal decision-making strategies.

• The paper presents a CTA framework that decouples uncertainty calibration from action selection to achieve near-optimal exploration in cost-sensitive settings.
• It models decision-making as a POMDP, enabling agents to adaptively trade off exploration costs and information benefits for improved performance.
• Empirical results demonstrate that CTA outperforms static policies in tasks like QA and code reasoning, achieving superior discounted rewards and robustness.

Cost-Aware Exploration in LLM Agents via Calibrate-Then-Act (CTA)

Problem Formulation and Motivation

The paper "Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents" (2602.16699) identifies a fundamental limitation in current LLM-based agents: they lack explicit mechanisms to reason about and optimize cost-uncertainty tradeoffs in sequential, partially-observed decision-making environments. In an array of practical scenarios—such as information-seeking, coding with ambiguous data formats, scientific diagnosis, and tool-augmented QA—agents must navigate the cost of exploration (e.g., API calls, latency, tool invocation) vs. the potential penalty of acting under uncertainty.

Modern LLM deployments generally rely on static policies: models uniformly query for clarification, always retrieve evidence, or default to extensive verification, regardless of actual uncertainty or environment cost. This static behavior neglects the inherent variation in instance-specific priors and fails to enact policies that are Pareto-optimal for task reward given exploration costs. Absent explicit representation or conditioning on these latent priors, end-to-end RL or basic chain-of-thought prompting does not suffice to reliably induce optimal exploration strategies.

Calibrate-Then-Act (CTA) Framework

The core methodological innovation is the CTA framework, which operationalizes the following principle: agentic policies should be conditioned on explicit estimates of relevant instance-level priors, inducing abstract reasoning about uncertainty-reward-cost tradeoffs. This is realized formally via sequential decision-making modeled as a partially observable Markov decision process (POMDP) with a state space defined over observable information and latent variables, and an action space including both exploration (information-gathering) and commitment actions.

CTA explicitly decouples (a) calibration of uncertainty (i.e., estimating priors from data or the agent’s internal confidence) and (b) the downstream policy learning or prompting that selects actions conditional on these priors and cost parameters. During both inference and RL fine-tuning, the agent is provided with a structured representation of estimated priors as part of its context/prompt, enabling it to (i) rationally trade off further environment exploration/verification vs. early commitment, (ii) modulate the exploration depth/strategy instance-wise rather than globally, and (iii) recover the optimal policy given the cost profile.

Empirical Demonstration and Results

Synthetic Pandora’s Box Benchmark

A variant of Weitzman’s optimal search (the Pandora’s Box problem) is used to establish that LLMs can, when supplied explicit priors, emulate an oracle policy matching theoretical optima. If the agent is not provided explicit priors, or is not prompted to reason through the decision (non-thinking), it cannot recover optimal match rates (11–23%). In contrast, CTA-PROMPTED achieves a 94% optimal match rate, validating that explicit probabilistic context enables correct value-of-information reasoning.

Realistic Cost-Aware Exploration Tasks

Knowledge QA with Optional Retrieval

In this task, answering a question directly (based on parametric knowledge) is fast but potentially inaccurate, while invoking retrieval provides more information at a cost (discounted reward via increased latency or resource use). Experiments on PopQA with the Qwen3-8B LLM show:

• Baselines (never/always retrieve, or prompt-only policies) either unnecessarily incur cost or suffer low accuracy, yielding suboptimal discounted reward.
• CTA-PROMPTED, which conditions on (iso)calibrated model confidence estimates ($\hat{p}_{da}(x)$) and retrieval efficacy, achieves the highest average discounted reward (0.293).
• Retrieval is invoked adaptively only when confidence is insufficient relative to cost, and the policy’s decision surface tightly aligns with the oracle threshold across varying cost regimes.

Code Reasoning with Format Uncertainty

This code exploration benchmark (FileReading) randomly obscures expected CSV parameters. Agents must choose whether to pay the cost of unit testing different file format hypotheses, or attempt code execution directly under uncertainty—potentially failing and paying higher exploration cost. Results show:

• Baseline prompted agents and end-to-end RL collapse to static, verification-first policies regardless of cost profile or uncertainty.
• CTA-PROMPTED and CTA-RL (RL with prior conditioning) exhibit clear cost-aware adaptation, balancing testing and guessing depending on the observed cost ratios.
• CTA-RL outperforms RL in average reward (0.268 vs. 0.259) and achieves a Pareto-optimal frontier of action policies across cost spectra. Action trace analysis shows high-frequency adaptation ("guess-and-go" only when optimal), which never emerges in vanilla RL.
• Importantly, CTA generalizes better on out-of-distribution cost profiles, suggesting robust, context-dependent strategy learning.

Theoretical Implications

CTA formalizes the LLM agent setting as a partially observable MDP with explicit cost and uncertainty components, making clear that without exposure to true priors and cost structures, RL policies cannot reliably internalize optimality criteria. This separation of (i) prior estimation from (ii) optimal action computation represents a shift from traditional RL in language agent design.

By integrating explicit model confidence calibration (e.g., isotonic regression on verbalized confidences, lightweight filename-to-format predictors), the approach highlights the central importance of probability calibration and structural uncertainty estimates for cost-aware agentics. This is consistent with a broader literature emphasizing the limitations of uncalibrated parametric memory and monolithic confidence in LLMs.

Practical Implications and Future Directions

The CTA methodology has broad applicability for cost-sensitive, tool-augmented LLM deployment scenarios—ranging from interactive code agents (e.g., automated debugging, data wrangling) to information-seeking QA, medical/diagnostic reasoning, and scientific discovery, where adaptive exploration policies can dramatically reduce resource usage and latency in exchange for a negligible decrease in average accuracy.

Further lines of inquiry include:

• Developing more powerful or generalizable prior estimators (e.g., learned world models, meta-calibration for intricate downstream tasks)
• Dynamic adaptation of agent strategies under shifting, real-time cost profiles (with non-stationary cost or reward parameters)
• Application to collaborative multi-agent settings, where joint priors can be estimated over distributed, ambiguous tasks
• Integration of CTA with abstention and refusal systems for improved safety/robustness
• Formalization of end-to-end differentiable analogues where prior and policy estimation are mutually optimized.

Conclusion

Calibrate-Then-Act (CTA) demonstrates that LLM agents can realize adaptive, Pareto-optimal cost-aware exploration behavior when explicitly conditioned on calibrated priors over task/environment structure. This approach achieves higher discounted utility and task performance than monolithic RL or static prompt engineering, and serves as a blueprint for designing rational and resource-efficient LLM agents in partially-observed, costly environments. The separation of calibration from action selection is critical for eliciting optimality, with implications for agent architecture, uncertainty quantification, and real-world AI deployment.