Low-Rank MLP Parameterization Methods

• Low-Rank MLP parameterization is a technique that decomposes weight matrices into lower-dimensional forms, significantly reducing parameters while preserving performance.
• Factorized, regularized, and hypernetwork-based approaches enable efficient fine-tuning and adaptation, with methods like LoRA and PoLAR demonstrating competitive results.
• Empirical benchmarks show that these low-rank adaptations match or outperform dense models in tasks such as language modeling, computer vision, and commonsense reasoning.

A low-rank MLP parameterization denotes any approach for representing the weight matrices of multilayer perceptrons (MLPs) in a form that constrains or induces the matrix rank to be much smaller than the ambient dimension. This paradigm dramatically reduces parameter and compute complexity, facilitates parameter-efficient adaptation, and can exploit emergent properties of network training dynamics. The field includes factorized, regularized, reparameterized, and group-structured methods, and recent advances demonstrate that low-rank parameterizations match or surpass dense (full-rank) baselines in a wide range of adaptation, fine-tuning, and even pretraining contexts.

1. Factorized Low-Rank Parameterizations

The classical approach factorizes a weight matrix $W \in \mathbb{R}^{m \times n}$ into a product of two lower-dimensional matrices: $W = U V^\top$ with $$U \in \mathbb{R}^{m \times r},\ V \in \mathbb{R}^{n \times r}$$ and $r \ll \min(m, n)$. This reduces parameter count from $mn$ to $r(m+n)$, significantly compressing large MLPs and reducing their inference footprint (Barone, 2016). The basic forward mapping for one layer is

$y = \phi(U V^\top x + b).$

Adding a diagonal term $D$ (low-rank-plus-diagonal) improves expressivity in aggressively compressed regimes: $W = U V^\top + D$, with $D=\mathrm{diag}(d)$. Skip-connection (“passthrough”) architectures further decouple expressivity from rank by ensuring the network state can propagate information even through bottlenecked weights.

Rank selection is task-dependent: for example, $r \approx n/4$ or $n/8$ suffices for language modeling or synthetic sequence modeling; extreme settings ($r \approx n/50$) require the diagonal enhancement. Universal approximation is preserved if $r \geq \min(m, n)$, and, via stacking and passthrough, even tighter bottlenecks can approximate arbitrary mappings to precision $\epsilon$ with polynomial depth (Barone, 2016).

2. Low-Rank Parameterization in Efficient Fine-Tuning

Parameter-efficient fine-tuning (PEFT) for large pre-trained models often uses low-rank adaptation, with LoRA (“Low-Rank Adaptation”) serving as paradigm. Instead of training the full $W_0$, LoRA learns an update in a low-rank subspace: $$\Delta W = B A, \quad B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times d}.$$ The adapted matrix is $W = W_0 + \Delta W$, updating only $O(r d)$ parameters per adapted layer. This approach preserves memory and computation efficiency and is widely adopted in transformer architectures for both attention and MLP sublayers (Bihany et al., 9 Jun 2025).

Several modern extensions of this basic framework address expressivity, optimization, and statistical efficiency:

• Multiplicative Low-Rank (LoRMA): Moves beyond the additive $\Delta W$ to multiplicative forms $W' = M W_0$, where $M = I + B A$ (with $B,A$ low-rank), exploring a strictly richer set of transforms via full-rank parameter inflation (identity or permutation-based), leading to improved empirical rank and faster convergence at equivalent parameter budgets (Bihany et al., 9 Jun 2025).
• Polar Decomposition (PoLAR): Employs a polar-style factorization $\Delta W = U S V^\top$ with $U,V$ constrained to be column-orthogonal (on the Stiefel manifold) and $S$ unconstrained, which provably enforces high “stable rank” and ameliorates collapse to a single dominant direction, thus improving utilization of the nominal rank $r$ and accelerating convergence via Riemannian optimization (Lion et al., 3 Jun 2025).
• Bayesian and Monte Carlo Methods: Techniques such as MonteCLoRA endow the low-rank factors $A, B$ with mixture-of-Gaussian hierarchies and/or Bayesian priors, allowing sampling and marginalization, which stabilizes fine-tuning, reduces estimator variance, and enhances robustness to hyperparameter settings (Sengupta et al., 2024).
• Hypernetwork & Overparameterized Variants: Approaches like RepLoRA and OP-LoRA use small MLPs (“hypernetworks”) to generate $A$ and $B$ from codes or embeddings, exploiting overparameterization for implicit adaptation of learning rates and momentum, and achieving both statistical efficiency and improved optimization dynamics (Truong et al., 5 Feb 2025, Teterwak et al., 2024).

3. Theoretical Foundations for Low-Rank MLPs

Recent theoretical work has established that standard MLPs trained by gradient descent on smooth activations undergo weight updates concentrated in invariant low-dimensional subspaces (Xu et al., 5 Feb 2026). For a two-layer MLP $f_{W_1}(X) = W_2 \,\phi(W_1 X)$ with output dimension $K$, one finds:

• The Jacobian of the loss $\nabla_{W_1}L$ has rank at most $K$ at initialization (for smooth $\phi$ and small $\|W_1(0)\|$).
• Throughout training, the weight dynamics for $W_1$ remain within a $2K$-dimensional subspace, with the remaining directions updated at only $O(\epsilon)$ scale.
• This leads to the parameterization

$W_1(t) = U\,\widetilde{W}_1(t)\,V^\top,$

where $\widetilde{W}_1$ is $2K \times 2K$ and captures almost all effective learning. Initializing $U, V$ according to the dominant singular vectors of the initial gradient ensures that training with this low-rank $W_1$ matches full-rank MLP performance on classification tasks, provided output dimension $K$ is small.

Empirical validations on datasets like Fashion-MNIST and CIFAR-10 confirm that low-rank parameterizations with $r \approx 2K$ match the accuracy of dense models, if properly initialized in the correct subspace (Xu et al., 5 Feb 2026).

4. Advanced Regularization and Rank Control

Explicit low-rank regularization presents a route for continuous rank-induction, with methods exploiting smooth surrogates for the rank function. The Quadratic Reweighted Rank Regularizer (Q3R) (Ghosh et al., 6 Nov 2025) replaces non-differentiable rank penalties with a smoothed log-determinant,

$F_\epsilon(W) = \sum_{i} f_\epsilon(\sigma_i(W)),$

where $f_\epsilon$ switches between quadratic and (logarithmic) penalties depending on the singular value magnitude. Training alternates truncated SVD with IRLS-inspired quadratic majorization, ensuring that the solution remains within an explicit rank budget. After training, matrices are truncated for highly efficient inference. Q3R achieves reductions in parameter count >50% with minimal accuracy drop on ViT and transformer models.

Second-order optimization with differentiable bilinear parameterization, as formalized in VarPro/LM methods, provides a smooth surrogate to classical nuclear norm or hard-rank penalties. The bilinear form $W = U V^\top$ admits explicit quadratic regularizers on $U, V$, is highly amenable to second-order optimization, and yields rapid convergence even for ill-conditioned models (Örnhag et al., 2018).

5. Structured, Joint, and Analytical Parameterizations

Expressivity and efficiency can be further enhanced by structured factorizations and analytical post-training low-rank reductions:

• Joint Tensor-Train (TT) Parameterization: By jointly generating multiple low-rank matrices (e.g., for up- and down-projections) using a shared TT-core network, one can enforce correlated adaptation, yielding both improved parameter efficiency and optimization dynamics. The TensorGuide framework shows that such joint TT parameterizations outperform both classical and per-matrix TT decompositions, as measured by faster convergence and improved accuracy (Qi et al., 19 Jun 2025).
• Analytical CUR-based Selection (A³): Instead of factorization, the A³ approach analytically selects the best $r$ neuron dimensions in the MLP, forming CUR-type masks and reducing the hidden width directly. This post-training procedure replaces bottleneck sublayers with smaller ones selected via a data-informed heuristic. This yields reduced memory and compute without inference overhead and outperforms typical SVD-like layerwise compression (Wong et al., 19 May 2025).

6. Optimization, Sample Efficiency, and Empirical Benchmarks

Low-rank parameterizations can raise optimization challenges, such as ill-conditioning, sensitivity to initialization, and slow convergence. Overparameterization (hypernetwork-based) and structured reparameterizations mitigate these issues by providing adaptive effective learning rates and improved search spaces (Teterwak et al., 2024, Truong et al., 5 Feb 2025). Bayesian and Monte Carlo methods stabilize the optimization trajectory and reduce variance in model outputs (Sengupta et al., 2024).

Empirically, low-rank MLP parameterizations have demonstrated strong or state-of-the-art results in:

• Natural language understanding (GLUE): LoRMA and PoLAR match or marginally surpass dense or LoRA-adapted baselines with order-of-magnitude fewer parameters (Bihany et al., 9 Jun 2025, Lion et al., 3 Jun 2025).
• Commonsense reasoning, vision-language multi-task benchmarks, and mathematical reasoning: Polarization, multiplicative updates, and joint-structured parameterizations consistently yield empirical gains (Lion et al., 3 Jun 2025, Qi et al., 19 Jun 2025).
• Compression and acceleration: Analytical and Q3R-based methods achieve and maintain superior performance at parameter reduction rates >60%, with negligible loss in accuracy (Wong et al., 19 May 2025, Ghosh et al., 6 Nov 2025).
• Downstream utility: Overparameterized and Bayesian variants enhance robustness to optimizer and batch-size choices, stabilize training, and accelerate convergence.

7. Comparison, Trade-offs, and Open Directions

A summary of recently proposed parameterizations:

Method	Param. Count	Expressivity	Optimization
Classical LoRA	$r(d + k)$	Additive low-rank	Fast, easy, but limited stable rank
LoRMA	$O(r(d + k))$	Multiplicative, full-rank via inflation	Matched or better than LoRA
PoLAR	$r(d + k)+r^2$	Enforced stable-rank, polar decomposition	Riemannian optimization
Q3R	$d k$	Explicit rank via regularization	IRLS, moderate overhead
RepLoRA/OP-LoRA	$O(r(d + k))$ + small MLP	Overparam., adaptive	Hypernetwork acceleration
TensorGuide	$O(\sum_k r_{k-1} n_k r_k)$	Joint/structured, TT	NTK-theoretically faster
A³	--	Analytical selection	Post-training, no inference overhead

Recent work shows that naive low-rank parameterization risks under-utilizing the subspace (low stable rank), slow convergence, or collapse to a single direction. Structured approaches (PoLAR, TensorGuide), multiplicative parameterizations (LoRMA), and hypernetwork-based reparametrization (RepLoRA, OP-LoRA) avoid these pitfalls, with empirical gains in accuracy, convergence speed, robustness, and parameter efficiency. For tasks with small output dimension $K$, emergent low-rank training dynamics of MLPs mean that parameterizing and training only the “large-movement” 2$K$-dimensional subspace is near-optimal (Xu et al., 5 Feb 2026).

A plausible implication is that task-specific low-rank parameterizations, augmented with geometric, probabilistic, or hypernetwork structure, will remain central in the scalable adaptation and compression of deep models. Future work may explore further integration of low-rank priors at pretraining, learnable rank-scheduling strategies, and mixed-method hybridizations.

---

References:

• (Barone, 2016) Low-rank passthrough neural networks
• (Örnhag et al., 2018) Bilinear Parameterization For Differentiable Rank-Regularization
• (Sengupta et al., 2024) Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation
• (Teterwak et al., 2024) OP-LoRA: The Blessing of Dimensionality
• (Truong et al., 5 Feb 2025) RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Experts
• (Wong et al., 19 May 2025) A3: an Analytical Low-Rank Approximation Framework for Attention
• (Lion et al., 3 Jun 2025) PoLAR: Polar-Decomposed Low-Rank Adapter Representation
• (Bihany et al., 9 Jun 2025) LoRMA: Low-Rank Multiplicative Adaptation for LLMs
• (Qi et al., 19 Jun 2025) Joint Tensor-Train Parameterization for Efficient and Expressive Low-Rank Adaptation
• (Ghosh et al., 6 Nov 2025) Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
• (Xu et al., 5 Feb 2026) Emergent Low-Rank Training Dynamics in MLPs with Smooth Activations