Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection

Abstract: Scaling laws are used to plan multi-million-dollar training runs, but fitting those laws can itself cost millions. In modern large-scale workflows, assembling a sufficiently informative set of pilot experiments is already a major budget-allocation problem rather than a routine preprocessing step. We formulate scaling-law fitting as budget-aware sequential experimental design: given a finite pool of runnable experiments with heterogeneous costs, choose which runs to execute so as to maximize extrapolation accuracy in a high-cost target region. We then propose an uncertainty-aware method for sequentially allocating experimental budget toward the runs most useful for target-region extrapolation. Across a diverse benchmark of scaling-law tasks, our method consistently outperforms classical design-based baselines, and often approaches the performance of fitting on the full experimental set while using only about 10% of the total training budget. Our code is available at https://github.com/PlanarG/active-sl.

• The paper presents a budget-efficient, active experiment selection method that maximizes extrapolation accuracy for high-cost LLM configurations.
• It employs a mixture-of-Gaussians posterior approximation to capture local basins, decomposing uncertainty into within- and between-basin components for cost-effective predictions.
• Empirical evaluations demonstrate that with only 10% of the pilot budget, the method matches or exceeds full-data performance while remaining robust to misspecification.

Budget-Efficient Scaling Law Fitting via Active Experiment Selection

Motivation and Problem Formulation

LLM training at industrial scales is increasingly governed by empirical scaling laws that predict downstream performance as a function of variables such as model size, data volume, and compute budget. However, the process of fitting scaling laws itself is computationally and financially expensive, often requiring thousands of pilot experiments and incurring multi-million-dollar costs. This cost is further inflated by the requirement to extrapolate into high-cost regimes: accurate prediction for larger, more expensive configurations, where practitioners ultimately make deployment decisions.

The paper frames scaling-law fitting as a budget-aware sequential experiment design problem. Rather than selecting experimental configurations in advance, each candidate experiment (training run) is treated as a costly query. The central objective is to actively select experiments that maximize extrapolation accuracy on a high-cost target region, subject to a finite overall budget. This approach recognizes that fitting scaling laws for modern LLMs is not merely a question of data modeling, but a resource allocation problem with nontrivial financial and scientific stakes.

Active, Uncertainty-Aware Experiment Selection

The core methodological contribution is an uncertainty-aware, sequential experiment selection algorithm for scaling-law fitting. The key components of the approach are:

• Posterior Approximation with Local Basins: Recognizing the multimodal nature of scaling-law loss landscapes, the method maintains a mixture-of-Gaussians approximation over parameter optima ("basins"), each reflecting a local optimum discovered by multi-start fits. Each basin is weighted via a BIC-style information criterion, ensuring efficient utilization of refitting results.
• Target-Aware Uncertainty Objective: To choose the next experiment, the algorithm evaluates the expected reduction in mean squared prediction error (MSPE) over the target region—where accurate prediction is crucial. The uncertainty is decomposed into within-basin (local variance) and between-basin (disagreement across optima) components.
• Cost-Sensitive Sequential Acquisition: Each candidate experiment is scored by its expected reduction of target MSPE, normalized by a power of its cost. This makes the selection explicitly cost-aware and robust to the heterogeneous price of pilot runs in modern LLM workflows.

The decision-theoretic formalism integrates the sequential refitting process, local linearization for tractable posterior updates, and a pairwise form for between-basin variance, enabling efficient evaluation using only a small subset of the candidate experimental configuration pool.

Figure 1: Extrapolation trajectory of the active method in $(\mathrm{lr}, \mathrm{bsz})$ space for a 1B-parameter LLM reaches the low-loss region within $1\%$ of the total fitting budget; right, the selected (sparse, low-cost) design points are visualized in 3D cost space.

Experimental Evaluation and Strong Results

A comprehensive benchmark is introduced, covering 65 scaling-law instances across 8 diverse LLM design tasks (including learning rate/batch size scaling, data and mix allocation, sparsity, and architecture selection). Experiments are compared against several baselines: uniform random, cheapest-first, cost-biased random, and two classical optimal experiment design heuristics (D-optimality and V-optimality adapted to nonlinear, cost-constrained settings).

Key findings include:

• Superior Efficiency: The proposed method consistently achieves the highest target-region $R^2$ across tasks and budgets, especially in the tight-budget regime ($1\%$–$5\%$ of total cost), where experiment selection is most consequential.
• Close-to-Full-Data Performance with Fractional Cost: With only $10\%$ of the original pilot budget, the active method typically matches or exceeds the accuracy of the full-data fit, avoiding unnecessary computation on configurations irrelevant for high-cost extrapolation.

  Figure 2: The proposed method achieves a stronger budget–accuracy tradeoff, approaching the full-data reference performance using a small fraction of the original experimental cost.

• Robustness Under Misspecification: In settings where standard fitting is hampered by model misspecification, the active method outperforms the "All Data" fit, demonstrating that blind inclusion of cheap pilot runs can degrade performance in the high-cost target regime—an important practical insight for real-world applications.

  Figure 3: Parameter t-SNE plot of the learning rate/batch size scaling law: many separated local optima appear, and only the active method reliably selects basins whose low-cost fit transfers to high-cost extrapolation, improving test $R^2$ versus classical heuristics.

Significant ablation analysis reveals that both intra- and inter-basin terms contribute to selection effectiveness; omitting either results in quantifiable drops in performance, emphasizing the necessity of modeling both sources of uncertainty for reliable extrapolative generalization.

Implications and Theoretical Significance

This work reshapes the practical and theoretical landscape of scaling-law application by moving from passive curve-fitting toward decision-theoretic, resource-conscious experimental design. Empirically, reliance on exhaustive pilot sweeps and manual grid design is shown to be inefficient and potentially misleading in extrapolative regimes. The active, cost-sensitive method demonstrates that accurate high-cost prediction can be achieved with a dramatically reduced pilot budget by sequentially targeting experimental configurations that maximally inform the desired region of parameter space.

Theoretically, the approach extends Bayesian optimal experiment design for nonlinear models to the setting of discrete, high-variance, and high-cost runs, integrating mixture-based posterior uncertainty, cost normalization, and sequential selection. The work highlights that, especially under model misspecification and in extrapolative tasks, optimal data selection is not simply a matter of coverage, but requires targeting high-leverage, high-value queries. This insight generalizes to a wide array of scientific and engineering settings where experiments (or simulations) are expensive and high-fidelity extrapolation is critical.

Discussion and Future Directions

The approach is not without limitations: its performance is tied to the fidelity of posterior approximation over basins, and, like most myopic acquisition schemes, does not optimize for multi-step lookahead or account for complex, high-dimensional, or dynamic experiment pools as encountered in live MLOps workflows. Nevertheless, the experimental and theoretical results motivate a shift in LLM scaling practice toward active, uncertainty-aware, and cost-efficient experiment design.

Future work directions include:

• More expressive and robust posterior representations for landscapes with extreme multimodality or severe misspecification.
• Multi-step and globally budgeted design strategies for even more effective allocation in large-scale or longitudinal LLM development.
• Extensions to experiment design with richer constraints (e.g., conditional dependencies, multi-objective scaling, and risk sensitivity).
• Integration into automated, closed-loop LLM engineering systems.

Conclusion

The paper provides a formal, experimentally validated foundation for budget-efficient, active experiment design in scaling-law fitting, with strong practical and theoretical support for supplanting ad hoc or exhaustive pilot studies with principled, uncertainty-aware sequential selection. These developments have direct implications for the economic and scientific feasibility of future LLM scaling, and exemplify a more general paradigm shift toward experiment allocation as a central design problem in large-scale AI development.

---

Reference:

"Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection" (2604.22753)