Deriving Hyperparameter Scaling Laws via Modern Optimization Theory

Abstract: Hyperparameter transfer has become an important component of modern large-scale training recipes. Existing methods, such as muP, primarily focus on transfer between model sizes, with transfer across batch sizes and training horizons often relying on empirical scaling rules informed by insights from timescale preservation, quadratic proxies, and continuous-time approximations. We study hyperparameter scaling laws for modern first-order optimizers through the lens of recent convergence bounds for methods based on the Linear Minimization Oracle (LMO), a framework that includes normalized SGD, signSGD (approximating Adam), and Muon. Treating bounds in recent literature as a proxy and minimizing them across different tuning regimes yields closed-form power-law schedules for learning rate, momentum, and batch size as functions of the iteration or token budget. Our analysis, holding model size fixed, recovers most insights and observations from the literature under a unified and principled perspective, with clear directions open for future research. Our results draw particular attention to the interaction between momentum and batch-size scaling, suggesting that optimal performance may be achieved with several scaling strategies.

• The paper presents explicit power-law scaling laws derived from non-asymptotic convergence bounds within the LMO framework.
• It details how optimal learning rate, batch size, and momentum settings scale with token budget to achieve a T^-1/4 convergence rate.
• Empirical validation on transformer pretraining underlines the practical implications for hyperparameter transfer in compute-constrained environments.

Explicit Hyperparameter Scaling Laws from Modern Optimization Theory

Introduction

The paper "Deriving Hyperparameter Scaling Laws via Modern Optimization Theory" (2603.15958) presents a comprehensive theoretical treatment of hyperparameter scaling in large-scale deep learning optimization, specifically focusing on learning rate, momentum, and batch size scheduling as functions of the training horizon and token budget for fixed model architectures. The analysis employs recent non-asymptotic convergence bounds within the Linear Minimization Oracle (LMO) framework, incorporating optimizers such as normalized SGD, signSGD (and thus Adam), and Muon. By minimizing these bounds in relevant regimes, the authors derive explicit power-law scaling schedules and clarify which empirical and previously heuristic scaling rules can be directly justified from optimization theory, thus providing substantial guidance for hyperparameter transfer in compute-constrained and high-cost training environments.

Theoretical Framework

The central contribution is a unified approach for extracting hyperparameter scaling laws by leveraging sharp convergence bounds for stochastic, momentum-normalized optimizers. Building on the LMO framework, the analysis captures a suite of critical methods used in modern LLM pretraining pipelines, notably connecting theory with Adam and Muon empirically.

The core non-convex bound has the form:

$$\min_{1\le k\le K} \mathbb{E}[\|\nabla f(x^k)\|_\star] \leq \frac{\Delta_0}{\eta K} + \frac{2\rho\sigma}{\alpha \sqrt{b} K} + 2\rho\sigma\sqrt{\frac{\alpha}{b} + \frac{7L\eta}{2} + \frac{2L\eta}{\alpha}},$$

where $\eta$ is the learning rate, $b$ the batch size, and $\alpha$ the normalized momentum parameter. This proxy includes deterministic optimization, momentum burn-in, a stochastic noise floor, and smoothness-induced error components. LMO-based optimizers, by virtue of explicit normalization and momentum coupling, go beyond classical SGD in their batch size–optimization horizon interactions, enabling non-trivial optimal batch size phenomena.

Scaling Law Derivations

Fixed Momentum Regime

With momentum fixed, minimizing the proxy bound leads to several concrete predictions:

• For fixed iteration count $K$, the optimal learning rate scales as $\eta^\star \propto K^{-1/2}$, with the minimized bound improving with batch size.
• For a fixed token budget $T = bK$, the optimal learning rate exhibits $\eta^\star \propto b^{1/2} T^{-1/2}$, with the jointly optimal batch size and learning rate obeying $$b^\star \propto T^{1/2}, \, \eta^\star \propto T^{-1/4}$$. Performance, as captured by gradient norm proxy, decays as $T^{-1/4}$. Crucially, the optimal batch size exceeds $1$ only after a momentum-dependent phase, contrasting with SGD where no such non-trivial batch size scaling is present.

  Figure 1: Verification of Theorem~$\ref{thm:fixed-alpha}$, illustrating fixed-momentum scaling laws: optimal performance, batch size, and learning rate behaviors as functions of the token budget $T$.

Fixed Batch Size and Momentum Tuning

By holding batch size fixed and minimizing the proxy with respect to momentum and learning rate, the optimal schedules become $\eta^\star \propto b^{1/2} \alpha^{1/2} T^{-1/2}$ and $\alpha^\star \propto b T^{-1/2}$. Upon further minimization, the achievable rate remains $T^{-1/4}$.

Figure 2: Numerical verification of the fixed-batch-size, momentum-tuned regime—Theorem~$\ref{thm:fixed_batch_mom}$.

Joint Tuning of All Hyperparameters

Minimizing the bound with respect to batch size, learning rate, and normalized momentum yields the asymptotics:

• $b_T^\star \propto T^{1/6}$, $\eta_T^\star \propto T^{-7/12}$, $\alpha_T^\star \propto T^{-1/3}$,
• The convergence rate with respect to token budget remains $T^{-1/4}$ regardless of the chosen scaling regime, up to constant factors.

Figure 4: Numerical verification of Theorem~$\ref{thm:tuned-token}$—asymptotic behavior under joint hyperparameter tuning.

The analysis reveals a flat suboptimality landscape in batch size for broad regimes, contingent on precise retuning of momentum and learning rate; only lower-order terms (burn-in and smoothness) select a specific batch growth law, with many alternative schedules being near-optimal. This is formalized in the multiple batch size scaling options uncovered in the paper.

Figure 6: Performance contours over batch size and token budget $T$, showing near-optimal batch-size schedules as a function of token count.

Figure 8: Iso-performance contours as a function of batch size and iteration count, highlighting the hyperbolic tradeoff between batch and step count.

Empirical and Qualitative Verification

The theoretical predictions are corroborated by both numerical proxy minimization and qualitative experiments on transformer training with actual optimizers (PlainLM implementation), which reveal that optimal learning rate increases with batch size but decreases with longer horizons, consistent with the derived laws.

Figure 9: Experimental results on LLM pretraining, confirming the predicted relationships among batch size, learning rate, and token budget.

Comparison to SGD and Observed Deviations

A significant theoretical divergence is established between normalized-momentum methods and vanilla SGD. For SGD, classical proxy analysis under token-limited budgets yields trivial batch-size dependence after learning rate tuning, with all non-trivial batch effects eliminated. In contrast, the LMO framework for Adam-like optimizers encodes a genuine, horizon-dependent optimal batch size, reconciling with empirical observations in LLM scaling.

The paper also treats a variety of empirical anomalies and reports from recent large-scale studies. Instances where learning rate appears to increase with token budget (with batch size scaling) are shown to arise naturally from protocol path-conditioning and hyperparameter constraints, rather than from deviations in the theoretical schedule. Furthermore, the impact of heavy-tailed noise and the choice of norm in defining variance is discussed, emphasizing circumstances under which the predicted exponents and scaling behaviors might differ from idealized settings.

Budget Transfer and Practical Implications

The framework yields direct "budget transfer" laws: transfer of tuned hyperparameters from a small (pilot) run to a larger training horizon should follow precise power laws as derived from the bound minimizations (e.g., for fixed batch size and momentum, learning rate should scale as $\eta_1 = \eta_0 (T_0/T_1)^{1/2}$ when moving from $T_0$ to $T_1$ tokens). These schedules depend critically on which parameters are constrained (hardware caps or protocol constraints), and the analysis characterizes when tuning momentum or batch size becomes necessary to avoid performance plateaus due to non-vanishing noise floors.

Limitations and Directions for Future Research

Key caveats are explicitly addressed: all results assume constant learning rates, fixed model size, and implicit statistical generalization via the population objective. Theoretical predictions are contingent on finite-variance noise and idealized initialization and may not capture the entirety of empirical scaling observed in large-scale LLM protocols—especially when generalization or protocol idiosyncrasies dominate. Notably, model size scaling, initialization mismatch, weight decay, learning rate annealing, and warmup are not included in the baseline analysis. The question of reconciling remaining discrepancies between empirical and theoretical scaling, particularly as both batch size and token budget increase, is highlighted as an open frontier.

Conclusion

The study presents a systematic and optimization-theoretic derivation of hyperparameter scaling laws for first-order optimizers in deep learning under fixed model architectures. The analysis clarifies when batch size, learning rate, and momentum should be tuned and how these parameters interact with compute budgets, iteration counts, and noise regimes. By recovering and explaining many observed empirical rules via explicit proxy minimization, the work supplies a principled framework for hyperparameter transfer and sheds light on outstanding mismatches between theory and practice. Extensions to account for model scaling, regularization, scheduling, and heavy-tailed noise remain crucial for further advances in scaling law theory and its integration into LLM training protocols.