Delving into Muon and Beyond: Deep Analysis and Extensions

Abstract: The Muon optimizer has recently attracted considerable attention for its strong empirical performance and use of orthogonalized updates on matrix-shaped parameters, yet its underlying mechanisms and relationship to adaptive optimizers such as Adam remain insufficiently understood. In this work, we aim to address these questions through a unified spectral perspective. Specifically, we view Muon as the p = 0 endpoint of a family of spectral transformations of the form U \boldsymbolΣ^{p} V' , and consider additional variants with p = 1/2 , p = 1/4 , and p = 1 . These transformations are applied to both first-moment updates, as in momentum SGD, and to root-mean-square (RMS) normalized gradient updates as in Adam. To enable efficient computation, we develop a coupled Newton iteration that avoids explicit singular value decomposition. Across controlled experiments, we find that RMS-normalized updates yield more stable optimization than first-moment updates. Moreover, while spectral compression provides strong stabilization benefits under first-moment updates, the Muon update (p = 0) does not consistently outperform Adam. These results suggest that Muon is best understood as an effective form of spectral normalization, but not a universally superior optimization method. Our source code will be released at https://github.com/Ocram7/BeyondMuon.

• The paper presents a unified spectral framework interpolating between standard optimizers and Muon’s orthogonalized updates.
• It employs coupled Newton–Schulz iterations for efficient inverse matrix computations and verifies findings using controlled experiments on nanoGPT.
• Results show that while Muon stabilizes momentum updates, excessive spectral flattening might reduce performance in RMS-normalized regimes.

Spectral Perspectives in Optimizer Design: A Deep Analysis of Muon and Its Extensions

Introduction

This work presents a comprehensive investigation into the Muon optimizer and its spectral generalizations for matrix-shaped parameters in neural networks. The authors introduce a unified framework for analyzing Muon, Adam, and intermediate variants, contextualizing orthogonalization and spectral compression within a rigorous, controlled experimental setting. The analysis aims to clarify Muon's mechanisms, its relationship to adaptive optimizers, and the practical/algorithmic consequences of spectral parametrization.

Unified Spectral Framework for Optimization

The central contribution is a parametric family of spectral transformations applied to update matrices, denoted by $\Psi_p(O_t) = U\Sigma^p V^\top$ for $p \in [0,1]$. This family interpolates between standard optimizers ($p=1$; e.g., Adam, mSGD), fractional spectral compression ($p=\tfrac{1}{2}$, $p=\tfrac{1}{4}$), and the Muon/polar update ($p=0$, i.e., orthogonalization).

The paper delineates two input regimes for these spectral updates:

1. First-moment update ($O_t^{\text{mom}} = M_t$): momentum SGD-like updates.
2. RMS-normalized update ($O_t^{\text{rms}} = M_t \oslash \sqrt{V_t}$): Adam-like updates that incorporate second-order statistics.

These inputs are paired with the four spectral exponents to produce eight configurations, enabling a systematic comparison among Muon, Adam, and novel interpolations.

Efficient Spectral Computation

A core challenge is the computational cost of directly performing singular value decompositions for large-scale matrices. To address this, the authors implement coupled Newton-Schulz iteration algorithms, enabling efficient calculation of inverse fractional matrix roots with only matrix multiplications:

• For $\Psi_{1/2}$ and $\Psi_{1/4}$, inverse square root and inverse fourth root matrices, respectively, are computed on the symmetric Gram matrix $O_t^\top O_t$, yielding scalable spectral compression applicable within deep architectures.

Controlled Experimental Protocol

All methods are evaluated in controlled conditions using nanoGPT (124M parameters), decoupling learning rates for spatially matrix-shaped and vector-shaped parameters, disabling weight decay, and omitting auxiliary stabilization (e.g., QK-Norm, QK-Clip). This design ensures that observed differences are strictly attributable to update rule variations.

Empirical Results: Spectral Compression Dynamics

The experiments yield the following primary findings:

1. Muon stabilizes momentum updates: On first-moment inputs, Muon ($p=0$) robustly defeats mSGD in stability across a broad learning-rate regime but does not universally achieve the highest final performance. Partial spectral compression ($p=\tfrac{1}{4}$) can occasionally yield superior results at optimal learning rates.

Figure 1: mSGDZ/Muon ($U^{0}V^\top$) shows improved stability versus mSGD across learning rates on first-moment updates.

2. RMS-normalized inputs are inherently more robust: Applying spectral compressions to RMS-normalized updates (Adam's paradigm) produces limited gains; full orthogonalization (AdamZ) often underperforms versus Adam and fractional spectrums (AdamS, AdamQ).

Figure 2: AdamZ ($U^{0}V^\top$) demonstrates that Muon-style orthogonalization applied to RMS-normalized updates does not outperform Adam or fractional spectral variants.

3. Spectral exponent regime influences anisotropy: Lowering $p$ compresses the singular spectrum, reducing condition number $\kappa_p = \kappa(O_t)^p$; however, excessive flattening ($p=0$) may amplify noise and suboptimally reweight gradient directions.

   Figure 3: Comparison of four spectral exponents $p\in\{0,\tfrac{1}{4},\tfrac{1}{2},1\}$ elucidates the spectrum flattening and its effect on learning dynamics for various input types.

Theoretical and Practical Implications

Robustness vs. Selectivity

Spectral orthogonalization is effective at mitigating instability generated by highly anisotropic update spectra in momentum methods. However, Muon discards all magnitude information, treating dominant and subdominant directions identically. This can be detrimental in latter-stage training, where low-rank signal directions may bear more information than directions with smaller singular values (typically containing noise).

RMS Normalization and Modern Transformer Training

Adam-style second-moment normalization already controls the per-coordinate gradient scaling, bounding update norms irrespective of raw gradient magnitude. Consequently, further spectral flattening yields only marginal stability improvements and in some cases, degrades performance by diminishing selective directionality.

Computational Overhead

The coupled Newton-Schulz framework enables practical matrix root operations, making the family of spectral optimizers computationally attractive. Nonetheless, for extremely large models, further efficiency innovations may be required.

Future Directions

The paper identifies open areas for exploration:

• Regularization synergy: Integration of matrix-based optimizers with weight decay and other regularization protocols remains unaddressed.
• Higher-order/structured spectral adaptation: There is room for scheme development that selectively compresses spectrum rather than uniformly flattening, to balance robustness and selective anisotropy exploitation.
• Theoretical optimization landscape analysis: Detailed characterization of landscape navigation under varying spectral exponents and normalization regimes can inform both algorithmic and architectural choices in future deep networks.

Conclusion

This work systematizes spectral transformation methodologies in optimizer design, providing a thorough experimental and theoretical basis for understanding Muon's strengths and limitations. The analysis demonstrates that, in the absence of RMS normalization, Muon stabilizes optimization but does not universally surpass adaptive optimizers such as Adam. Fully flattened spectral updates are not inherently superior to partial compression or classical adaptive methods. Future improvements will likely arise from targeted spectrum regularization strategies and integrating matrix-based normalization with established regularization techniques.