Rethinking Language Model Scaling under Transferable Hypersphere Optimization

Abstract: Scaling laws for LLMs depend critically on the optimizer and parameterization. Existing hyperparameter transfer laws are mainly developed for first-order optimizers, and they do not structurally prevent training instability at scale. Recent hypersphere optimization methods constrain weight matrices to a fixed-norm hypersphere, offering a promising alternative for more stable scaling. We introduce HyperP (Hypersphere Parameterization), the first framework for transferring optimal learning rates across model width, depth, training tokens, and Mixture-of-Experts (MoE) granularity under the Frobenius-sphere constraint with the Muon optimizer. We prove that weight decay is a first-order no-op on the Frobenius sphere, show that Depth-$μ$P remains necessary, and find that the optimal learning rate follows the same data-scaling power law with the "magic exponent" 0.32 previously observed for AdamW. A single base learning rate tuned at the smallest scale transfers across all compute budgets under HyperP, yielding $1.58\times$ compute efficiency over a strong Muon baseline at $6\times10^{21}$ FLOPs. Moreover, HyperP delivers transferable stability: all monitored instability indicators, including $Z$-values, output RMS, and activation outliers, remain bounded and non-increasing under training FLOPs scaling. We also propose SqrtGate, an MoE gating mechanism derived from the hypersphere constraint that preserves output RMS across MoE granularities for improved granularity scaling, and show that hypersphere optimization enables substantially larger auxiliary load-balancing weights, yielding both strong performance and good expert balance. We release our training codebase at https://github.com/microsoft/ArchScale.

• The paper introduces a hypersphere-constrained optimization method (MuonH) that removes weight decay tuning and enables a single learning rate to transfer across scales.
• It derives scaling laws for model depth, width, and training tokens, revealing universal power-law behaviors such as the optimal LR decreasing as T^-0.32.
• It also presents SqrtGate for Mixture-of-Experts, ensuring stable, granularity-invariant scaling and significantly improved compute efficiency leverage.

Rethinking LLM Scaling under Transferable Hypersphere Optimization

Introduction

The paper "Rethinking LLM Scaling under Transferable Hypersphere Optimization" (2603.28743) develops a new theoretical and empirical framework for hyperparameter transfer in large-scale LLM (LM) pretraining, based on hypersphere-constrained optimization. Unlike prior work that predominantly focuses on adapting first-order optimizers (e.g., SGD, AdamW), the authors analyze MuonH—a second-order method that constrains weight matrices to fixed-norm hyperspheres. They formally derive scaling laws for batch size, width, depth, training tokens, and Mixture-of-Experts (MoE) granularity under this regime, demonstrating that a single learning rate selected at small scale transfers robustly across all compute budgets. They additionally introduce SqrtGate, an MoE gating mechanism suited to the Frobenius-sphere setting. Broadly, this work significantly unifies and sharpens the understanding of structural stability in LM scaling and enables efficient, instability-resistant model development.

Theory: Hypersphere Optimization and Transfer Laws

The central methodological advance is a comprehensive scaling-theoretic analysis of hypersphere-constrained optimization. In this regime, all hidden weights are projected onto a fixed Frobenius-norm sphere after each optimizer step. The authors prove that, for the MuonH optimizer, weight decay vanishes as a first-order term—a property not shared by standard optimizers that require delicate joint tuning of learning rate and decay across scale. Thus, hypersphere projections reduce hyperparameter search to the learning rate alone.

Additionally, they establish width transfer under the matrix Frobenius norm, showing that for nearly isotropic weights, the effective output RMS after fully-connected layers becomes independent of width scaling—realizing μP-style behavior without explicit width-dependent learning rate scaling.

Depth scaling is analyzed in detail. The authors prove that update normalization (as in MuonH) necessitates scaling the learning rate with $L^{-1/2}$ for residual networks with depth $L$ (assuming residual branch scaling $\alpha_L = L^{-1/2}$), and they show that claimed depth-transferability of hypersphere optimizers (in prior work) is incorrect without properly scaled residual multipliers.

Figure 1: Depth-$\mu$P (left) versus unscaled (right) learning rate curves; with proper scaling, the optimal learning rate is stable across depths, validating the theory.

Empirically, they confirm these scaling predictions over wide ranges of model depth and size. For data scaling, they identify that the optimal learning rate decreases with the number of training tokens $T$ by a universal power law $\eta^*\sim T^{-0.32}$, consistent with previous observations for AdamW. This "magic exponent" appears robust to architecture and optimizer class.

Figure 2: Left—Loss as a function of learning rate over token budgets; Right—Optimal learning rate displays a clean log-log power law of exponent 0.32 with respect to number of tokens.

Transferable Stability and Training Dynamics

A core issue in prior scaling work is that hyperparameter transfer methods offer no guarantee of stability: loss spikes, logit explosions, and activation outliers are routine at large scale, often requiring bespoke regularization (z-loss, dynamic weight decay). The authors demonstrate, both theoretically and empirically, that hypersphere constraint naturally bounds logit magnitudes and output activations—effectively controlling instability.

This is quantified via six indicators (logit $Z$-values, output RMS, activation outliers) showing, for both attention and MoE routers, that instability does not increase with scale under the HyperP transfer law; in some cases, controlled metrics even improve at larger scale.

Figure 3: Key stability indicators (e.g., $Z$-values and RMS) remain bounded or improve with scale for MoE models under HyperP, even as depth increases.

Mixture-of-Experts Scaling and SqrtGate

Proper scaling in sparse expert models (MoEs) is further complicated by interactions between expert choice (granularity) and gate scaling, leading to unstable or distorted forward signal propagation. The authors formalize these effects, showing that under classical gating the residual's RMS shrinks with increasing $k$ (number of selected experts). They introduce SqrtGate, an alternative RMS-preserving gating scheme, which ensures granularity-invariant scaling of activations.

Empirically, SqrtGate reduces router logit explosions and improves both MoE validation loss and expert load balance, even allowing the use of substantially higher auxiliary balance weights without quality regression—opposite to prior claims.

Figure 4: SqrtGate enables consistent loss-vs-learning rate curves across a broad range of MoE top-$k$; stability is maintained and optima are shifted lower.

Compute Efficiency Leverage and Empirical Scaling

HyperP's practical benefit is measured in compute efficiency leverage (CEL), comparing the FLOPs required to match a target loss across transfer laws and architectures. Across depth and size, HyperP with MuonH delivers up to $1.58\times$ CEL over strong μP++/Muon baselines at 6e21 FLOPs on dense models. Application to MoE architectures (with SqrtGate) further increases CEL to $3.38\times$.

Figure 5: (Left) Loss vs. total training FLOPs for Muon, MuonH, and HyperP variants. (Right) CEL for HyperP surpasses direct alternatives with scale; MoE gains are dramatic.

The compute advantage increases monotonically with scale, emphasizing the compounding effect of proper hyperparameter transfer. The authors validate that a learning rate tuned at 208M parameters and small compute generalizes up to 13.3B parameter models and >6e21 FLOPs with minimal degradation.

Batch Size Scaling, Sensitivity, and Architectures

Batch size scaling is examined empirically; the optimal learning rate obeys an intermediate scaling law, exponent ≈ 0.56, closely matching SDE-based theoretical predictions. Additionally, the authors analyze sensitivity of LR selection using quadratic fit, and show that with merely 3–5 sweep points, one can estimate optimal LRs to under 5% relative error, sufficient given the quadratic insensitivity of loss minima.

Figure 6: (Left) Loss versus learning rate across batch sizes; (Right) optimal LR scaling with batch size.

Architectural ablations (attention normalization variants, MoE module configurations) show that architectural improvements (e.g., gated attention, QK-norm, SqrtGate, shared experts) yield diminishing performance deltas as compute budgets scale, but deliver strong and growing improvements in training stability.

Practical and Theoretical Implications

This work significantly consolidates and simplifies the hyperparameter search protocol for large-scale LM training by demonstrating that one small-scale search over learning rate suffices for robust, stable scaling of both dense and MoE architectures under hypersphere optimization. This structurally solves the long-standing problem of catastrophic instability during scaling and enables reliable, near-optimal comparisons across model designs.

From a theoretical perspective, eliminating the need for tuned weight decay and batch-size-specific adjustments signals deeper universality in the scaling laws of gradient-based optimizers. The recurring data scaling exponent $0.32$ across distinct optimizer and architecture classes suggests a fundamental property of high-dimensional neural learning dynamics, underscoring the necessity of further theoretical inquiry into its provenance.

Practically, these advances can be immediately leveraged for more reliable exploratory architecture search, efficient resource allocation under compute constraints, and for scaling state-of-the-art models to the frontier regimes of parameter count and training duration without ad hoc regularization.

Conclusion

The paper provides a unified and empirical-theoretical approach to transferable hyperparameter selection under the regime of hypersphere-constrained optimization, establishing that models using MuonH and the HyperP transfer law attain strong stability, scaling efficiency, and minimal manual tuning requirements across the full spectrum of contemporary LLM pretraining. SqrtGate further extends these properties to sparse expert models. The combination of transferable efficiency and structural stability fundamentally reshapes best practices for training large models, with anticipated impact on both future theoretical analyses and practical system design.

Limitations include assumption of the Frobenius norm setting (extension to other matrix norms warrants further work), empirical—but not theoretical—justification for the data scaling "magic exponent," and dependence on Chinchilla FL0Ps-optimality. Extension to hybrid or state-space architectures is an open direction.

The cumulative results indicate that hypersphere optimization under the HyperP framework is a robust and efficient foundation for future LM scaling studies and system development (2603.28743).