Stabilizing Native Low-Rank LLM Pretraining

Abstract: Foundation models have achieved remarkable success, yet their growing parameter counts pose significant computational and memory challenges. Low-rank factorization offers a promising route to reduce training and inference costs, but the community lacks a stable recipe for training models from scratch using exclusively low-rank weights while matching the performance of the dense model. We demonstrate that LLMs can be trained from scratch using exclusively low-rank factorized weights for all non-embedding matrices without auxiliary "full-rank" guidance required by prior methods. While native low-rank training often suffers from instability and loss spikes, we identify uncontrolled growth in the spectral norm (largest singular value) of the weight matrix update as the dominant factor. To address this, we introduce Spectron: Spectral renormalization with orthogonalization, which dynamically bounds the resultant weight updates based on the current spectral norms of the factors. Our method enables stable, end-to-end factorized training with negligible overhead. Finally, we establish compute-optimal scaling laws for natively low-rank transformers, demonstrating predictable power-law behavior and improved inference efficiency relative to dense models.

• The paper introduces Spectron, a framework using spectral renormalization and gradient orthogonalization to stabilize native low-rank LLM pretraining from scratch.
• It demonstrates that fully factorized low-rank models can match or surpass dense model performance while reducing parameter counts and inference costs.
• Empirical results confirm that Spectron ensures stable training dynamics and optimal scaling, outperforming previous low-rank and hybrid approaches.

Stabilizing Native Low-Rank LLM Pretraining with Spectron

Introduction

The substantial growth in LLM parameter counts has escalated both training and inference costs. Low-rank factorization is widely recognized as a promising method for reducing compute and memory requirements. However, previous attempts to pretrain LLMs natively with only low-rank weights have encountered severe instability and required auxiliary dense guidance or hybrid architectures. This work introduces a scalable framework for stable, fully low-rank pretraining—Spectron—which leverages spectral renormalization and gradient orthogonalization to overcome the pathological optimization dynamics inherent in naive factorized architectures (2602.12429).

Figure 1: Validation loss for a 780M dense Transformer versus a 454M factorized Transformer, demonstrating that Spectron achieves dense-level performance with substantially fewer parameters.

Spectral Instability in Native Low-Rank Training

Low-rank parameterization expresses weight matrices as $W = AB^\top$ with $A \in \mathbb{R}^{m \times r}$, $B \in \mathbb{R}^{n \times r}$, and $r \ll \min(m, n)$. While effective for low-rank fine-tuning (e.g., LoRA), end-to-end factorized training from initialization is highly unstable due to scaling invariance: any scaling $(\lambda A, (1/\lambda) B)$ leaves $W$ unchanged but allows arbitrarily large singular values in the factors, resulting in divergent spectral norm growth of weight updates.

Figure 2: Low-rank parameterization induces 10-30$\times$ larger spectral norm in weight updates compared to dense training, revealing inherent instability.

Naive per-factor SGD or AdamW updates, unconstrained, induce explosions in $\|\Delta W\|_2$, in turn destabilizing activations. This pathology is absent in dense architectures, where standard optimizers maintain bounded spectral norm dynamics.

Spectron: Spectral Renormalization and Gradient Orthogonalization

Spectron directly addresses the instability mechanism by bounding the composite spectral norm of weight updates. The approach is motivated by the matrix update equation:

$$\Delta W = \Delta A B^\top + A \Delta B^\top + \Delta A \Delta B^\top$$

Through gradient orthogonalization and dynamic renormalization, Spectron ensures that $\|\Delta W\|_2 \leq \eta$, where $\eta$ follows the prescribed learning rate. Explicitly, given estimates of $\|A\|_2$ and $\|B\|_2$ via power iteration, the per-iteration update constraint is:

$\rho = \frac{\eta}{\|A\|_2 + \|B\|_2 + 1}$

where both $\Delta A$ and $\Delta B$ are orthogonalized and scaled to have spectral norm less than $\rho$. This provably limits the aggregate update norm of $W$.

Figure 3: Spectral norm constraints maintain bounded $\|\Delta W\|_2$, stabilization of activation RMS change, and controlled weight spectral norm throughout factorized training.

Compared to self-guided "dense guidance" methods which rely on auxiliary dense weights for stability (Section 3.1 in (2602.12429)), Spectron requires negligible extra compute (sub-1% overhead) and does not constrain model structure with dense components, making the approach suitable for large-scale, memory-bound deployments.

Empirical Results

Performance and Efficiency

Across LLaMA-style models at multiple scales, full-factorized Spectron-trained LLMs:

• Achieve validation losses on par with or better than dense baselines under equal compute (FLOP-matched) budgets.
• Outperform self-guided and naive AdamW low-rank training in terms of perplexity and downstream task accuracies.
• Remain stable even at aggressive learning rates, an unattainable regime for naive factorized optimizers.

  Figure 4: Validation loss comparison showing Spectron achieves both faster convergence and better final performance than self-guided training and naive AdamW.

  

  Figure 5: A 454M-parameter fully factorized model matches the performance of a 780M-parameter dense Transformer under equal FLOP budgets, reducing inference cost by 42\% while maintaining accuracy.

Scaling and Inference Efficiency

Spectron-trained low-rank models enjoy drastically improved scaling efficiency:

• Lower parameter counts are needed for a given perplexity, translating into smaller and faster inference models.
• Given any fixed compute budget, the optimal low-rank model (N_opt) tracks as $C^{0.479}$, whereas the dense-model Chinchilla law yields $C^{0.49}$ [hoffmann2022an]. Training tokens for optimality scale as $C^{0.521}$ versus dense's $C^{0.51}$.

Figure 6: Low-rank models require fewer parameters than dense models for equivalent perplexity, implying higher inference efficiency at all scales.

Figure 7: Optimal model size scaling for low-rank architectures is lower than for dense, yielding substantial inference cost reductions across compute budgets.

Figure 8: IsoFLOP validation loss curves exhibit clear minima, confirming well-defined compute-optimal model sizes for factorized transformers analogous to dense Chinchilla scaling.

Complementary Mechanisms

Ablations demonstrate that gradient orthogonalization (i.e., Muon updates) and spectral renormalization act synergistically. Either component alone improves convergence and stability, but their combination is necessary for optimal, loss-equivalent training in the factorized regime. The approach generalizes to different rank ratios, but extreme compression (e.g., $r/n < 0.25$) fundamentally degrades performance.

Implications and Future Directions

The work establishes that constrained, fully low-rank architectures are not only trainable from scratch but also competitive in performance with dense models under equal compute, provided updates are spectrally regularized and orthogonalized. This finding contradicts long-standing assumptions that full-rank auxiliary weights or dense pretraining are inherently required for optimality in large-scale transformers.

Practically, Spectron paves the way for scalable LLMs in hardware-restricted environments. Future directions include developing communication-efficient distributed strategies for factorized layers (see also [nabli2025acco]), extending to multimodal architectures, and investigating the impact of aggressive data regimes where model compactness and deployability become limiting factors.

From a theoretical perspective, the near-equivalence of scaling exponents to dense models—albeit shifted toward smaller size/larger dataset optima—suggests that low-rank structural inductive bias does not burden the expressivity of transformers, but instead constrains optimization toward more resource-efficient solutions.

Conclusion

This work presents a practically effective and theoretically grounded framework for native low-rank LLM pretraining, culminating in Spectron—a method that leverages dynamic spectral renormalization and orthogonalized updates to ensure stability and parity with dense architectures. The approach eliminates reliance on auxiliary dense weights, enables FLOP-optimal, inference-efficient deployment, and invites further exploration in compute/bandwidth-constrained and data-rich settings.

Reference: "Stabilizing Native Low-Rank LLM Pretraining" (2602.12429)