PRISM: Structured Optimization via Anisotropic Spectral Shaping

Abstract: We propose PRISM, an optimizer that enhances first-order spectral descent methods like Muon with partial second-order information. It constructs an efficient, low-rank quasi-second-order preconditioner via innovation-augmented polar decomposition. This mechanism enables PRISM to perform anisotropic spectral shaping, which adaptively suppresses updates in high-variance subspaces while preserving update strength in signal-dominated directions. Crucially, this is achieved with minimal computational overhead and zero additional memory compared to first-order baselines. PRISM demonstrates a practical strategy for integrating curvature-adaptive properties into the spectral optimization paradigm.

• The paper presents PRISM, an optimizer that integrates low-rank covariance approximations to enable adaptive anisotropic damping in complex loss landscapes.
• It leverages innovation-augmented polar decomposition, modulating updates based on signal-to-noise ratios for accelerated convergence and reduced training loss.
• Experimental results confirm that PRISM is robust to hyperparameter variations and remains stable under aggressive learning rates compared to baselines.

PRISM: Structured Optimization via Anisotropic Spectral Shaping

Introduction

The paper introduces PRISM, an optimizer that enhances the spectral optimization paradigm by incorporating partial second-order information via low-rank covariance approximation. This addresses the critical deficiency of first-order optimizers lacking adaptive mechanisms for anisotropically dampening updates in directions with high gradient variance, which frequently leads to instability in non-convex, ill-conditioned loss landscapes. PRISM leverages innovation-augmented polar decomposition, where the innovation term efficiently acts as a low-rank proxy for the covariance of gradients, enabling curvature-adaptive preconditioning without increasing memory or computational cost significantly.

Spectral and Structured Optimization Foundations

Contemporary deep learning optimizers predominantly use adaptive statistics (e.g., AdamW) or structured approaches (e.g., Shampoo, Muon) for curvature-aware optimization. While adaptive optimizers individually scale parameters via running second moments, they disregard the structural interdependencies within weight matrices. Structured optimization methods treat parameters as linear operators, employing approaches such as spectral descent (SSD) and Kronecker-factored preconditioning (Shampoo).

Muon scales spectral optimization to large-scale models by employing Newton-Schulz iterations to efficiently orthogonalize the momentum matrix without incurring the overhead of SVD. However, its approach is isotropic and built from first moments only, leading to statistically blind updates that ignore underlying covariance and can result in overly aggressive steps in noise-dominated subspaces.

PRISM: Innovation-Augmented Polar Decomposition

PRISM introduces an innovation term $D_t = G_t - M_t$ alongside the momentum, forming an augmented matrix $\tilde{M}_t = [M_t;\ \gamma D_t]$. The subsequent polar decomposition yields an implicit preconditioner:

$$P_t = (M_t^\top M_t + \gamma^2 D_t^\top D_t)^{-1/2}$$

This term introduces covariance-awareness by adaptively shaping updates: strong damping is imposed along directions with high instantaneous variance, while preserving update magnitude in signal-dominated eigenspaces. The mechanism is inherently low-rank due to the rank-1 nature of $D_t$, ensuring stateless memory and minimal overhead.

Anisotropic Spectral Filtering and SNR-driven Adaptivity

PRISM's efficacy is rooted in its adaptive spectral gain, which modulates update magnitude based on the empirical SNR along each principal direction:

$$\rho_k = \frac{\| M_t v_k \|}{\sqrt{\| M_t v_k \|^2 + \gamma^2 \| D_t v_k \|^2}} = \frac{1}{\sqrt{1 + 1/\text{SNR}_k^2}}$$

In high-SNR regimes, PRISM approaches the behavior of isotropic spectral optimizers, retaining aggressive learning. In low-SNR directions, it enforces substantial damping, thus regularizing and stabilizing the training trajectory.

Experiments

PRISM was benchmarked on causal LLM pretraining using the Qwen2 architecture and the FineWeb-Edu dataset. Parameters were partitioned between structured (spectral) and unstructured (AdamW) groups. Baselines included AdamW and Muon (equivalent to PRISM with $\gamma = 0$). Cosine annealing schedules with high learning rate clipping thresholds were deployed to rigorously assess robustness.

Training Loss Comparison

Initial results demonstrate that PRISM achieves both accelerated convergence and improved final loss compared to Muon and AdamW. Notably, PRISM achieves a lower final loss (3.269) versus Muon's (3.285), establishing the impact of anisotropic spectral shaping.

Figure 1: Training loss curves for AdamW, Muon, and PRISM, showcasing PRISM's accelerated convergence and lower final loss.

Hyperparameter Sensitivity

PRISM consistently outperforms Muon across a wide range of damping coefficients $(\gamma \in [0.5, 10.0])$, affirming robustness to hyperparameter selection and demonstrating that the mechanism is not reliant on precise tuning for effectiveness.

Comparison to Tikhonov Damping

Ablation studies with uniform Tikhonov damping confirmed that merely regularizing updates fails to outperform Muon; only PRISM's selective, SNR-aware approach yields substantial gains.

Figure 2: Training loss comparison between PRISM and Muon with Tikhonov damping under identical conditions.

Stability under Aggressive Learning Rates

When exposed to increased learning rates (up to 0.1), Muon diverges rapidly, while PRISM exhibits superior stability and self-recovery dynamics. When high curvature regions are encountered, PRISM adaptively increases damping in high-variance dimensions via the innovation term.

Internal Mechanisms: Norm and Spectral Gain Analysis

PRISM's implicit regularization is reflected in the reduced growth rate of the Frobenius norm $\|O_t\|_F$ relative to Muon. Furthermore, empirical analysis of spectral gain against SNR confirms PRISM's theoretical adaptive behavior, as its gain closely follows the predicted curve and diverges from Muon's fixed isotropic gain.

Figure 3: Frobenius norm $\|O_t\|_F$ across training steps, illustrating PRISM's implicit regularization via controlled update magnitudes.

Practical and Theoretical Implications

PRISM extends the spectral optimization paradigm to encompass quasi-second-order adaptivity without significant additional cost. By efficiently leveraging instantaneous innovations as covariance proxies, it bridges the gap between computationally efficient first-order techniques and more expressive, but expensive, second-order methods. Its stateless architecture and SNR-driven adaptivity facilitate stable, aggressive training at scale, broadening practical optimizer selection for LLMs and complex architectures. The framework lays groundwork for algorithmic extensions involving more sophisticated low-rank covariance estimators and multi-sided spectral shaping strategies, with the potential to further minimize variance-induced instability in future AI systems.

Conclusion

PRISM provides an efficient, memory-preserving optimizer integrating structured curvature adaptivity directly into the spectral descent methodology. By leveraging innovation-augmented polar decomposition, it selectively suppresses updates in noisy subspaces while maintaining aggressive optimization in signal-rich directions. Empirical results confirm comprehensive improvements in convergence, stability, and robustness over established baselines. PRISM's design positions it as a practical quasi-second-order optimizer, bridging the spectrum between highly scalable first-order algorithms and sophisticated second-order preconditioners.