Can Small Training Runs Reliably Guide Data Curation? Rethinking Proxy-Model Practice

Abstract: Data teams at frontier AI companies routinely train small proxy models to make critical decisions about pretraining data recipes for full-scale training runs. However, the community has a limited understanding of whether and when conclusions drawn from small-scale experiments reliably transfer to full-scale model training. In this work, we uncover a subtle yet critical issue in the standard experimental protocol for data recipe assessment: the use of identical small-scale model training configurations across all data recipes in the name of "fair" comparison. We show that the experiment conclusions about data quality can flip with even minor adjustments to training hyperparameters, as the optimal training configuration is inherently data-dependent. Moreover, this fixed-configuration protocol diverges from full-scale model development pipelines, where hyperparameter optimization is a standard step. Consequently, we posit that the objective of data recipe assessment should be to identify the recipe that yields the best performance under data-specific tuning. To mitigate the high cost of hyperparameter tuning, we introduce a simple patch to the evaluation protocol: using reduced learning rates for proxy model training. We show that this approach yields relative performance that strongly correlates with that of fully tuned large-scale LLM pretraining runs. Theoretically, we prove that for random-feature models, this approach preserves the ordering of datasets according to their optimal achievable loss. Empirically, we validate this approach across 23 data recipes covering four critical dimensions of data curation, demonstrating dramatic improvements in the reliability of small-scale experiments.

• The paper reveals that conventional proxy model training with standard learning rates induces fragile data recipe rankings.
• It demonstrates that tiny learning rates significantly boost cross-scale ranking stability, yielding Spearman correlations above 0.9.
• The study provides a computationally efficient protocol that aligns proxy ablation outcomes with optimal large-scale model performance.

Reliable Data Curation with Proxy Models: Stability, Hyperparameter Sensitivity, and the Principled Use of Tiny Learning Rates

Introduction and Motivation

The paper "Can Small Training Runs Reliably Guide Data Curation? Rethinking Proxy-Model Practice" (2512.24503) rigorously investigates the foundational assumption underlying modern large-scale LLM development pipelines: that small proxy models, trained under fixed "reasonable" hyperparameter configurations, can reliably guide high-stakes data curation decisions for much larger, expensive target model pretraining. Despite widespread adoption in production workflows, the stability and transferability of such proxy-model-based methods remain poorly understood, especially in the context of data/hyperparameter interactions and scaling.

The authors identify a severe protocol-practice mismatch: while data teams typically evaluate data recipes via small model ablations with fixed training configurations for purported "fairness," large model training always entails data-specific hyperparameter tuning. Consequently, proxy-based rankings can be fragile, with orderings that drastically change under small hyperparameter pertubations—particularly in learning rate. The paper both formalizes this fragility and proposes a well-grounded patch: training proxy models with "tiny" learning rates, making ablation results much more robust to both hyperparameter choices and model scaling.

Fragility of Proxy-Based Data Curation

The core analytical observation is that the optimal hyperparameter configuration is non-uniform across data recipes due to nontrivial data/hyperparameter interactions. Empirically, the authors demonstrate that minor changes in learning rate can induce major rank reversals between data recipes— even with the same proxy architecture (Figure 1).

Figure 1: Validation loss rankings of 23 data recipes evaluated on proxy (GPT2-125M) and target (Pythia-1B) models. Standard learning rates produce unaligned rankings; tiny learning rates yield robust cross-scale agreement.

This instability is theoretically motivated via higher-order Taylor expansions: at typical learning rates, second-order loss curvature terms become non-negligible, dominating the first-order "distributional similarity" signals. Thus, fixed proxy configurations do not approximate the under-optimally-tuned downstream reality and may propagate suboptimal choices.

Principled Fix: Tiny Learning Rates

To align proxy-based data curation with the real-world goal—selecting data recipes that maximize optimally tuned target model performance—the authors propose a minimal intervention: train proxy models at learning rates orders of magnitude lower than standard practice. This regime suppresses higher-order interactions, making observed validation losses an accurate proxy for the irreducible gap attributable to distribution mismatch between candidates and holdout validation sets.

The authors justify their approach across three axes:

• Empirical validation: Under tiny learning rates, dataset rankings remain highly stable under both small-to-large model scaling and hyperparameter tuning, and closely track the best-achievable (hyperparameter-optimized) downstream target performance.
• Theoretical analysis: For random feature models (and thus in the NTK/infinite-width regime), they prove that dataset ordering by tiny-learning-rate proxy training converges to the ordering under best achievable target performance (the RKHS norm becomes dominant).
• Practical efficiency: Although extremely small learning rates would be computationally prohibitive for full training, proxy evaluation requires only sufficient discrimination among recipes—early-epoch training at tiny learning rate suffices.

The practical result is an efficient, robust, and easily-adopted fix, requiring no new architecture or non-standard pipeline changes beyond tuning the learning rate during proxy ablations.

Figure 3: Tiny learning rates applied on proxy models produce high dataset ranking agreement (Spearman correlations above 0.9) across a range of evaluation domains and targets.

Experimental Results

The study comprises large-scale experiments spanning 23 candidate data recipes (covering domain mixture, deduplication, quality filters, etc.) and three model families (GPT2, Pythia, OPT), across sizes from 70M to 1B parameters. The experiments systematically sweep learning rates and evaluate:

• Loss and ranking on diverse validation domains (The Pile splits)
• Top-k decision regret (the utility loss if the best proxy-ranked data, or best of top-k, is used rather than the true optimum)
• Downstream benchmark accuracy (HellaSwag, ARC, OpenBookQA, etc.)

Key findings:

• With commonly used learning rates ($10^{-3}$, $3 \times 10^{-4}$), dataset rankings are poorly aligned with target-optimized outcomes (rank correlations $<0.75$), and top-k regret is high (up to 0.25 in loss).
• Reducing the learning rate by 1–2 orders of magnitude (to $10^{-5}$ or $10^{-6}$) dramatically increases ranking stability; correlations above 0.92 are typical for all proxy/target pairs and domains tested.
• This stability persists across ablation on batch size, weight decay, and token-per-parameter ratios—learning rate is the unique fragile axis.
• The patch also has strong resource implications: small models at tiny learning rates require only a fraction of the compute of naïve grid sweeps or multi-epoch proxy experiments yet achieve near-optimal guidance for large-scale data curation.

  Figure 2: Spearman correlation and top-k decision regret as a function of proxy model learning rate. Tiny learning rates minimize decision risk and maximize agreement with fully tuned target performance.

Theoretical Analysis

A formal analysis is given for random feature networks trained by mini-batch SGD. The main theorem establishes that, under standard regularity conditions (e.g., well-posedness, realizability in the RKHS), for sufficient width, batch size, and training duration, there exists a range of sufficiently small learning rates such that the expected validation loss ordering between any pair of data distributions is identical to their ordering under fully optimized, infinite-width training. The proof leverages concentration of measure for kernel matrices, explicit control of optimization and approximation error, and quantification of the rates at which higher-order interactions decay with learning rate.

This result complements prior positive findings regarding hyperparameter transferability in the NTK regime and elucidates the sharply differing effects of learning rate on dataset ranking and attribution as opposed to model scaling or parameterization.

Practical Recommendations

• Sweep only learning rate (and as a secondary check, validate batch size and weight decay insensitivity); learning rates $10^{-5}$ to $10^{-6}$ empirically suffice for all contemporary LLM proxy ablations.
• Training for a fraction of a full epoch suffices for stable ranking under tiny learning rates.
• Apply this tiny-LR protocol for all curation decisions involving deduplication, sample filtering, and domain composition.
• For ablation workflows involving multi-epoch or curriculum pipelines, further research is required to correctly subsample or simulate repetition patterns.

  Figure 4: Visualization of theoretically recommended learning rate regimes. The empirically optimal ranges from bounds are strictly below those required for practical stable learning—confirming the theory.

Implications and Future Directions

This work systematically exposes and patches the reliability gap in standard data-centric AI ablation protocols. The shift in methodology—from "compare at fixed configuration" to "compare at tiny learning rate"—restores alignment between proxy ablation outcomes and the true industrial training pipeline, where datasets are always paired with their optimal hyperparameters. The theoretical and empirical results are robust to scaling, model families, and recipe coverage, and should be adopted as best practice for data teams guiding major LLM training investment.

Theoretically, the findings highlight the importance of first-order loss geometry and distributional alignment for transferability across scales, pointing toward meta-objectives that optimize both data and training configurations jointly. Practically, they enable compute-constrained exploration of high-dimensional data search spaces with strong transfer guarantees.

Key open questions include principled proxy evaluation under multi-epoch and curriculum training regimes, extending theory from random feature approximators to overparameterized transformers, and dynamic joint optimization of data and optimizer configurations through, e.g., differentiable HPO or meta-learning methods.

Conclusion

Standard fixed-configuration proxy-model pipelines for LLM training data selection are fundamentally fragile, especially along the learning rate axis. The simple and theoretically-justified remedy—evaluating data recipes at tiny learning rates—yields trustworthy, hyperparameter-stable guidance that aligns with optimally tuned large-scale model outcomes, as substantiated by both broad empirical evidence and rigorous analysis. This protocol closes the proxy-to-target discrepancy and should serve as an essential component of computationally efficient, reliable data-centric AI practice.