LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling

Abstract: Test-time scaling (TTS) has become an effective approach for improving LLM performance by allocating additional computation during inference. However, existing TTS strategies are largely hand-crafted: researchers manually design reasoning patterns and tune heuristics by intuition, leaving much of the computation-allocation space unexplored. We propose an environment-driven framework, AutoTTS, that changes what researchers design: from individual TTS heuristics to environments where TTS strategies can be discovered automatically. The key to AutoTTS lies in environment construction: the discovery environment must make the control space tractable and provide cheap, frequent feedback for TTS search. As a concrete instantiation, we formulate width--depth TTS as controller synthesis over pre-collected reasoning trajectories and probe signals, where controllers decide when to branch, continue, probe, prune, or stop and can be evaluated cheaply without repeated LLM calls. We further introduce beta parameterization to make the search tractable and fine-grained execution trace feedback to improve discovery efficiency by helping the agent diagnose why a TTS program fails. Experiments on mathematical reasoning benchmarks show that the discovered strategies improve the overall accuracy--cost tradeoff over strong manually designed baselines. The discovered strategies generalize to held-out benchmarks and model scales, while the entire discovery costs only $39.9 and 160 minutes. Our data, and code will be open-source at https://github.com/zhengkid/AutoTTS.

• The paper introduces AutoTTS, a novel framework that recasts test-time scaling as an algorithmic search, enabling LLMs to autonomously discover improved inference controllers.
• The study presents the Confidence Momentum Controller, which uses exponential moving average momentum and coupled width-depth control to balance compute allocation with accuracy.
• Experimental results demonstrate that AutoTTS outperforms manual heuristics across benchmarks by reducing token consumption by about 69.5% while maintaining comparable accuracy.

LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling (AutoTTS)

Introduction

Test-time scaling (TTS) offers a compelling mechanism for improving LLM performance by adaptively allocating computational resources during inference. Traditionally, TTS strategies—a crucial determinant of inference effectiveness and cost—have relied on manually-crafted heuristics for branching, deepening, pruning, probing, and aggregation. However, this hand-design paradigm is inflexible, non-adaptive to novel settings, and leaves a substantial portion of the computation-allocation space unexplored. The paper "LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling" (2605.08083) proposes a paradigm shift: recasting TTS as an algorithmic search problem over a structured environment, enabling automatic, agent-driven discovery of optimized TTS controllers.

Environment-Driven TTS Strategy Discovery

The core contribution is the AutoTTS framework, which transfers the human designer’s role: from directly encoding TTS heuristics, to constructing offline, replayable environments where LLM agents can discover TTS strategies autonomously.

Environment Construction: The discovery environment abstracts TTS as controller synthesis over pre-collected reasoning trajectories (branches) and probe signals. The structured action space incorporates BRANCH, CONTINUE, PROBE, PRUNE, and ANSWER, allowing agents to allocate inference budget across width (number of branches) and depth (branch extension). Feedback is supplied in the form of both coarse (accuracy, cost) and fine-grained (execution traces) signals. This deterministically replays candidate controllers over problem sets, eliminating repeated LLM calls and enabling affordable, reproducible evaluation.

Search Space Parameterization: To induce tractability—given the high propensity for agents to introduce over-parameterized strategies—AutoTTS enforces a strict beta parameterization: all controller hyperparameters are deterministic, monotonic functions of a single beta parameter. Beta controls the accuracy-efficiency tradeoff along a continuous axis from frugal to liberal compute allocation, severely mitigating overfitting and search-set specific collapse.

Agentic Discovery Loop: The discoverer LLM (Claude Code) operates over multi-round loops: it proposes controller code, receives full feedback (execution traces plus accuracy/cost), diagnoses failure modes, and edits the candidate controller. Traces are pivotal—aggregated statistics alone are insufficient for targeted algorithmic improvement.

The Confidence Momentum Controller (CMC)

The strongest discovered controller, termed the Confidence Momentum Controller (CMC), introduces four key mechanisms—each markedly distinct from the baseline family (ASC, ESC, Parallel-Probe):

1. Trend-Based Stopping via EMA Momentum: Rather than relying on instantaneous confidence, CMC maintains an exponential moving average (EMA) of answer-pool confidence. Termination only occurs when the EMA is both sufficiently high and not actively declining, robustly avoiding premature halting due to transient confidence spikes.
2. Coupled Width-Depth Control: Widening (branch spawning) and deepening (branch extension) decisions are linked through the EMA delta. Stagnation or regression in confidence triggers further exploration (widening), whereas strong momentum suppresses new branch allocation, establishing a closed feedback loop that adaptively matches compute to problem uncertainty.
3. Alignment-Aware Depth Allocation: At each round, branches aligned with the current consensus ("winner" answer) receive enhanced probe budgets, focusing computation on promising lines of reasoning, but the allocation remains adaptive and non-uniform.
4. Conservative Branch Abandonment: Branches are only abandoned after persistently deviating (multiple rounds) from consensus, and always preserving a core set of active hypotheses, preventing brittle convergence.

These design elements, collectively, yield a highly coordinated controller structure, exhibiting complexity difficult to achieve by manual heuristics.

Experimental Evaluation

AutoTTS is instantiated over Qwen3 models (0.6B–8B) on challenging math reasoning benchmarks (AIME24, AIME25, HMMT25) and non-math tasks (GPQA-Diamond). The replay-based discovery and evaluation regime enables affordable and thorough parameter sweeps: the entire discovery loop costs only \$39.9 and 160 minutes—demonstrating practicality.

Numerical Performance:

• The discovered controller improves the accuracy-token Pareto frontier across all model scales and datasets. For example, with beta=0.5, CMC reduces token consumption by approximately 69.5% compared to SC@64, maintaining comparable accuracy (e.g., 45.3 vs 45.2 averaged across four models).
• At higher budget settings (beta=1.0), CMC further surpasses all baselines on peak accuracy in 5/8 settings.
• Generalization is strong; controllers discovered solely on AIME24 outperform hand-crafted TTS algorithms on held-out AIME25, HMMT25, and even transfer well to DeepSeek-R1-Distill-Llama-8B and GPQA-Diamond.

Ablation studies emphasize that omitting beta parameterization leads to catastrophic generalization failures—even though token usage drops, accuracy collapses, verifying that AutoTTS's search-space formulation is essential for robust discovery.

Analysis and Implications

CMC realizes a controller family whose mechanisms are non-trivially different from all prior hand-crafted and agent-generated baselines; in particular, the use of momentum-aware stopping, feedback-coupled compute allocation, and monotonic, explainable hyperparameter schedules yields strategies with robust generalization properties. Ablations further show that both execution trace feedback and strict single-parameter scheduling are necessary and sufficient for cross-benchmark robustness.

Practically, this framework changes the locus of effort from brittle, case-by-case design of TTS strategies, to the construction of richly instrumented, reusable algorithmic discovery environments. One-time search costs are amortized across model families and tasks, and the environment specification itself becomes the target of continual scientific improvement. This is a marked advancement for automated, unsupervised algorithm design.

Theoretically, casting TTS as controller synthesis, and resolving it via agentic discovery, provides a clear interface between environment abstraction and agent intelligence. The framework readily accommodates further extensions to richer computation spaces (e.g., integration with tree search, verifier-guided reasoning, or interactive verification), supporting the development of controllers with higher degrees of reasoning complexity.

Future Directions

Potential directions include:

• Expanding the action/state space abstraction in the discovery environment, supporting hierarchical and multistage TTS strategies.
• Generalizing the approach to domains with interaction, black-box feedback, or test-time RL adaptation.
• Exploring the performance of open-source (vs. closed-source) coding LLMs as discoverers to promote reproducibility and democratization.

Conclusion

This work introduces a scalable, environment-driven agentic discovery paradigm for TTS, demonstrating that construction of the right environment enables LLM agents to autonomously synthesize controllers that systematically outperform strong hand-designed baselines. By parameterizing the search over a single monotonic beta and tightly integrating behavioral feedback, such strategies generalize across model scale, domain, and task—all at negligible marginal computational cost. This paradigm invites future TTS research to focus on environment design, heralding substantial increases in the efficiency and accessibility of LLM inference (2605.08083).