Manifold constrained steepest descent

Abstract: Norm-constrained linear minimization oracle (LMO)-based optimizers such as spectral gradient descent and Muon are attractive in large-scale learning, but extending them to manifold-constrained problems is nontrivial and often leads to nested-loop schemes that solve tangent-space subproblems iteratively. We propose \emph{Manifold Constrained Steepest Descent} (MCSD), a single-loop framework for optimization over manifolds that selects a norm-induced steepest-descent direction via an LMO applied to the Riemannian gradient, and then returns to the manifold via projection. Under standard smoothness assumptions, we establish convergence guarantees for MCSD and a stochastic momentum variant. We further introduce \emph{SPEL}, the spectral-norm specialization of MCSD on the Stiefel manifold, which admits scalable implementations via fast matrix sign computations. Experiments on PCA, orthogonality-constrained CNNs, and manifold-constrained LLM adapter tuning demonstrate improved stability and competitive performance relative to standard Riemannian baselines and existing manifold-aware LMO methods.

• The paper presents the MCSD method that bypasses nested tangent-space iterations by using a linear minimization oracle and direct projection onto manifolds.
• It introduces SPEL, a spectral-norm specialization for the Stiefel manifold, which achieves rapid convergence in PCA and orthogonality-constrained deep networks.
• The work provides convergence guarantees and empirical evidence, demonstrating practical scalability in large-scale neural network and LLM adapter tuning.

Manifold Constrained Steepest Descent: A Unified Single-Loop Framework for Manifold Optimization

Overview of MCSD Framework

The paper “Manifold constrained steepest descent” (2601.21487) introduces the Manifold Constrained Steepest Descent (MCSD) methodology, providing a general and computationally tractable single-loop approach for optimization over smooth manifolds with norm-constrained steepest descent steps. In contrast to prior approaches which typically require solving nested iterative subproblems in the tangent space (e.g., manifold Muon, Riemannian Frank-Wolfe), MCSD selects descent directions using a linear minimization oracle (LMO) applied to the Riemannian gradient and then projects directly onto the manifold, bypassing the need for inner tangent-space solvers. This structure leads to a significant reduction in per-iteration computational overhead, especially in large-scale applications.

Figure 1: One iteration of Riemannian Gradient Descent, Manifold Muon, and SPEL on the unit sphere, highlighting the direct projection in MCSD versus nested-loop tangent-space solutions.

Algorithmic Contributions

1. General Two-Step MCSD Algorithm

At each iteration, MCSD performs a step in the ambient space in the LMO-chosen direction (typically dual to the chosen norm, such as the spectral norm for matrices) with a step size $\alpha_t$, followed by Euclidean projection onto the manifold $\mathcal{M}$. The update rule is

$$x_{t+1} = P_{\mathcal{M}}\left(x_t + \alpha_t d_t\right), \qquad d_t \in \mathrm{LMO}_{\|\cdot\|}(\nabla_{\mathcal{M}}f(x_t))$$

where $P_{\mathcal{M}}$ is the (efficient) projection and $\nabla_{\mathcal{M}}f(x_t)$ is the Riemannian gradient.

2. Spectral-Norm Specialization (SPEL) for Stiefel Manifold

The paper further proposes SPEL, a specialization for the Stiefel manifold with the spectral norm. SPEL's update involves fast matrix sign computations (e.g., via Newton–Schulz or Polar Express iterations), yielding computational costs closely matching Euclidean spectral gradient methods.

3. Stochastic and Momentum Variants

A stochastic MCSD is derived using heavy-ball momentum, incorporating stochastic gradient estimates, enabling applicability to large-batch or mini-batch regimes in neural network training.

Theoretical Results

The authors establish convergence guarantees for both the deterministic and stochastic MCSD under standard manifold smoothness and regularity assumptions. Importantly, MCSD’s complexity matches that of classical nonconvex first-order methods, and the framework admits step sizes up to a constant for well-conditioned manifolds such as the Stiefel manifold.

Key technical results include explicit bounds on the local smoothness constants for the composition $f \circ P_{\mathcal{M}}$ and concrete operator norm bounds for the second derivative of the Stiefel projection, ensuring the validity of the descent lemma globally on the feasible region.

Numerical Experiments and Empirical Evidence

Principal Component Analysis (PCA)

The paper evaluates MCSD and SPEL on Stiefel-constrained PCA, comparing against Riemannian gradient descent (RGD) and manifold Muon variants that employ nested iterative tangent-space solvers. On moderate- to large-scale synthetic PCA problems, SPEL consistently surpasses RGD in convergence rate and is competitive in accuracy with all tangent-space Muon variants, but requires significantly less wall-clock time due to the absence of inner loops.

Figure 2: Convergence behavior of manifold optimizers for PCA with $(n,p)=(200,5)$, demonstrating SPEL’s rapid subspace recovery versus RGD and nested manifold Muon approaches.

Orthogonality-Constrained Deep Neural Networks

The SPEL optimizer is applied to orthogonality-constrained CNNs, specifically Wide ResNet-28 on CIFAR-100. Compared with RGD/RAdam, Muon, and standard optimizers (SGD/AdamW), SPEL achieves the highest test accuracy and favorable training loss profiles. The method demonstrates low sensitivity to learning-rate choices within a broad range, simplifying hyperparameter tuning across architectures.

Figure 3: Left: Test accuracy vs epoch. Middle: Training loss vs epoch. Right: Learning-rate sensitivity for SPEL and Muon. SPEL maintains stability and superior peak accuracy.

Large-Scale LLM Adapter Tuning

For adapter-based low-rank adaptation (LoRA/StelLA) on LLaMA-3-8B and LLaMA-3.1-8B, SPEL matches the accuracy and convergence of state-of-the-art manifold-aware optimizers, serving as a direct drop-in replacement with lighter optimizer state (no second-moment estimates) and no increase in tuning burden.

Figure 4: Comparison of SPEL and StelLA optimizers for LLaMA-3-8B adapter-tuning tasks, indicating parity in training dynamics and downstream accuracy.

Contrasts with Existing Manifold Optimization Paradigms

MCSD departs from nested-loop structures such as Riemannian Frank-Wolfe, manifold Muon (dual/ADMM-based), and conditional gradient methods that are prohibitively expensive at scale due to costly tangent-space subproblem solutions. Instead, MCSD achieves both projection-free (when the manifold’s projection operator is efficient) and norm-aware updates, unifying previous steepest-descent approaches within a two-step iteration. The update direction in MCSD/SPEL does not require membership in the tangent space, in contrast to most other geometric optimizers, resulting in further computational simplification.

Practical and Theoretical Implications

• Scalability: MCSD (and in particular, SPEL) enables efficient application of manifold constraints (notably orthogonality) in deep learning at significant scale, including for large CNNs and LLM adapters, without incurring the typical overhead from nested tangent-space routines.
• Modularity: The algorithm can leverage highly optimized approximate matrix projection and sign routines amenable to GPU acceleration, making it suitable for modern ML pipelines.
• Convergence Guarantees: The convergence analysis, applicable to both deterministic and stochastic variants, provides nonconvex stationarity guarantees analogous to state-of-the-art unconstrained or Euclidean-constrained methods, with explicit complexity bounds.

Outlook and Future Directions

The MCSD framework opens several trajectories for future research. Extending MCSD to other manifold geometries where efficient projection and LMO computation are available is natural, as is integration with adaptive step-size or layerwise scaling strategies for further acceleration in deep networks. The demonstrated stability and parameter insensitivity in large-scale settings suggest MCSD-based methods could become default choices for enforcing geometric constraints in foundational models and for tasks requiring robust spectral or orthogonality regularization. Furthermore, the single-loop architecture is promising for hardware-efficient implementation in distributed or resource-constrained settings.

Conclusion

“Manifold constrained steepest descent” (2601.21487) establishes a comprehensive, theoretically sound, and empirically validated approach for norm-constrained optimization over manifolds, with a focus on scalability and simplicity. By leveraging direct LMO-based direction selection and efficient projection, MCSD (and its spectral Stiefel instance, SPEL) facilitates the routine application of geometric constraints in modern machine learning, providing both practical speedups and rigorous convergence properties, and demonstrating state-of-the-art accuracy in orthogonality-constrained models and LLM adaptation tasks.