Reparameterized PEFT Methods

• Reparameterized PEFT is a fine-tuning framework that updates pretrained models using lightweight, structured low-rank parameterizations to drastically reduce trainable parameters.
• Techniques such as LoRA, DyLoRA, AdaLoRA, and LieRA leverage mathematical tools including low-rank factorization and Lie group theory for efficient update strategies.
• Empirical studies show these methods achieve near full fine-tuning performance while significantly lowering computational costs and memory requirements.

Reparameterized Parameter-Efficient Fine-Tuning (PEFT) refers to a family of approaches in which parameter updates for large pretrained models are not made by direct adjustment of the whole parameter set, but rather by learning lightweight, structured parameterizations—often low-rank factorized forms—that efficiently adapt models to downstream tasks while substantially reducing the number of trainable parameters and associated computational costs. This framework encompasses foundational techniques such as LoRA, DyLoRA, AdaLoRA, as well as mathematical generalizations based on Lie group theory, enabling both linear and higher-dimensional parameter space adaptation for diverse model architectures in NLP and vision. Reparameterized PEFT is now a central paradigm for scalable adaptation of LLMs, vision transformers, and multimodal models (Si et al., 1 Apr 2025, Prottasha et al., 19 Apr 2025).

1. Mathematical Foundations and Formalism

Let $$W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$$ denote a pretrained parameter matrix of a layer (commonly in a Transformer block). Standard fine-tuning updates $W$ directly: $W \leftarrow W + \Delta W$, with $\Delta W$ being a dense, full-size parameter matrix. Reparameterized PEFT methods instead freeze $W$ and express $\Delta W$ via a structured low-dimensional parameterization: $W' = W + \Delta W, \qquad \Delta W = B A$ where $A \in \mathbb{R}^{r \times d_\text{in}}$, $B \in \mathbb{R}^{d_\text{out} \times r}$, and $r \ll \min(d_\text{in}, d_\text{out})$. During forward propagation, an input $x$ produces: $y' = (W + B A)x = Wx + B(Ax)$ The only trainable parameters are $A$ and $B$, reducing parameter count from $d_\text{in} d_\text{out}$ to $r (d_\text{in} + d_\text{out})$.

Generalizations to higher-dimensional tensors—such as convolutional kernels—have been developed to preserve the structure of the parameter manifold. In Lie group-based frameworks (e.g., LieRA), parameter tensors (such as convolutional weights $$\mathcal{W} \in \mathbb{R}^{C_\text{in} \times C_\text{out} \times k \times k}$$) are modeled as elements of an Abelian Lie group $(G, \odot)$ under elementwise (Hadamard) multiplication. Updates are then performed via perturbations $\Delta \mathcal{W} \in \mathfrak{g}$ in the associated Lie algebra, mapped smoothly back using the exponential map: $$\mathcal{W}_{\text{new}} = \mathcal{W}_\text{base} \odot \exp(\Delta \mathcal{W})$$ The standard implementation uses a first-order Taylor approximation: $$\exp(\Delta \mathcal{W}) \approx 1 + \Delta \mathcal{W}$$, leading to the efficient update $$\mathcal{W}_\text{new} \approx \mathcal{W}_\text{base} + \mathcal{W}_\text{base} \odot \Delta \mathcal{W}$$ (Si et al., 1 Apr 2025).

2. Core Algorithms and Reparameterization Variants

The most widely adopted reparameterized PEFT technique is LoRA (Low-Rank Adaptation). The core variants include:

• LoRA: Updates are parameterized as $\Delta W = BA$; only $A$ and $B$ are trained, with typical ranks $r \sim 1$–8 for LLMs. Initialization uses $A \sim \mathcal{N}(0, \sigma^2 I)$, $B = 0$ (Prottasha et al., 19 Apr 2025).
• DyLoRA: Block-wise dynamic low-rank adaptation, selectively updating sub-blocks at each step, focusing on regions with the largest gradient magnitudes.
• AdaLoRA: Adapts rank dynamically during training by introducing a diagonal scaling $\Lambda = \mathrm{diag}(\lambda_i)$ between $A$ and $B$ ($\Delta W = B\Lambda A$); small $\lambda_i$ are pruned over time.
• LieRA: Generalizes LoRA to higher-dimensional or structured weight spaces, employing Lie group theory for updates that preserve spatial and topological relationships, particularly useful for adapting convolutional kernels in computer vision models (Si et al., 1 Apr 2025).
• Further extensions: LoRA-Dropout (structured dropout on $A$, $B$), Laplace-LoRA (Bayesian priors), QLoRA (quantized low-rank adaptation for memory efficiency), and RoCoFT (row- and column-wise factorization for structure-aware adaptation) (Prottasha et al., 19 Apr 2025).

All these methods share the property that adaptation parameters are much fewer than full-model fine-tuning, often achieving sub-1% parameter footprints.

3. Theoretical Properties and Design Considerations

Reparameterized PEFT methods are grounded in empirical and theoretical observations:

• Low intrinsic dimension: Empirically, effective fine-tuning often resides in a subspace of much lower dimension than $W$. The low-rank factorization acts as a bi-linear bottleneck, restricting adaptation to a manifold compatible with many downstream tasks (Prottasha et al., 19 Apr 2025).
• Parameter efficiency: For LoRA-style adaptation, the parameter overhead is $r (d_\text{in} + d_\text{out})$, compared with $d_\text{in} d_\text{out}$ for full fine-tuning; for LieRA, parameter overhead is mathematically identical because only the low-rank factors for $\Delta W$ are learned (Si et al., 1 Apr 2025).
• Regularization and stability: The low-rank constraint implicitly regularizes by limiting overfitting to small or imbalanced datasets. In LieRA, the Lie group structure guarantees that updates remain invertible and weights never collapse to zero, enhancing numerical stability in deep architectures (Si et al., 1 Apr 2025).
• Gradient flow: With the exponential map formulation, gradient propagation is controlled: the Jacobian is the exponential itself; with the first-order approximation, the Jacobian is simply the identity, simplifying optimization (Si et al., 1 Apr 2025).

4. Optimization, Implementation, and Resource Trade-Offs

Optimization involves training only the introduced factors $A$ and $B$ (and possible auxiliary scalings), keeping the backbone $W$ frozen. Gradient flow is direct due to the linear (or, with LieRA, efficiently approximated exponential) structure.

Parameter and memory complexity:

• Full fine-tuning: $O(d_\text{in} d_\text{out})$ (e.g., 102M parameters for ConvNeXt-V2-B).
• LoRA and LieRA: $O(r (d_\text{in} + d_\text{out}))$ (e.g., 14.5M for $r=16$).
• Compute overhead is minor: forward pass introduces one (LoRA/LieRA) or two (in general) additional small matrix multiplications.

Practical implementation uses simple overrides for affected modules. For convolutional layers, LieRA preserves spatial structure by operating directly in the tensor's native algebraic space, avoiding distortions from matriсization (Si et al., 1 Apr 2025).

5. Empirical Performance and Benchmarks

Extensive experimental studies demonstrate that reparameterized PEFT methods match or outperform additive or direct PEFT baselines, often approaching full fine-tuning performance. Representative results:

Task/Model	Method	Params or %	Accuracy / Score
VTAB-1k (ConvNeXt-V2-B)	Full FT	102M	78.2
	LoRA	14.5M	74.1
	LieRA	14.5M	75.5
COCO det.+seg. (ConvNeXt-V2-B + Mask R-CNN)	LoRA	17.3M	38.4 (mAP)
	LieRA	17.3M	42.3 (mAP)
LLaMA-7B commonsense reasoning	LoRA	0.42%	70.9
	LieRA	0.42%	75.2
DeBERTaV3-base GLUE	LoRA (r=2)	0.18%	88.13
	LieRA (r=2)	0.18%	88.97
GLUE/RoBERTa (LoRA)	Full FT	124.6M	SST-2: 92.89
	LoRA	0.89M	SST-2: 93.31

Ablation studies confirm that:

• First-order approximations in LieRA yield almost the same accuracy as the exact exponential ($<0.2$\% gap), with half the training cost.
• Gains from LieRA over LoRA are consistent across ranks ($0.5$–$1.5$\% per task).
• For LoRA, dynamic and adaptive variants (DyLoRA, AdaLoRA) provide further gains, especially in resource-constrained scenarios (Si et al., 1 Apr 2025, Prottasha et al., 19 Apr 2025).

6. Extensions, Limitations, and Future Directions

Reparameterized PEFT serves as a foundation for further parameter and computation reduction by selective fine-tuning strategies (e.g., FISH-Tuning), hybridization with adapters and prefix-tuning, and quantization-aware training (Xue et al., 5 Apr 2025, Prottasha et al., 19 Apr 2025).

• FISH-Tuning applies Fisher information masking to restrict adaptation within the LoRA/Adapter low-rank subspace to only the most important components, achieving further parameter and memory savings (Xue et al., 5 Apr 2025).
• X-PEFT extends this notion to adapter banks, learning binary or soft masks over pre-existing adapters, realizing $10^3$–$10^4\times$ reduction in per-profile memory with comparable performance (Kwak et al., 2024).

Current limitations and research challenges include:

• Understanding why low-rank reparameterization suffices for transfer and what task-specific factors modulate its effectiveness.
• Automating layerwise or task-aware reparameterization schedules.
• Extending group-theoretic generalizations to non-commutative structures (e.g., for rotational equivariance or specialized attention structures) (Si et al., 1 Apr 2025).
• Federated and continual learning contexts, adaptive and meta-learned factorization, and interpretability of learned adaptation spaces.
• Efficient adaptation in extremely parameter- and memory-constrained settings (Prottasha et al., 19 Apr 2025, Kwak et al., 2024).

7. Broader Impact and Implications

Reparameterized PEFT fundamentally transforms the scalability and accessibility landscape for adapting large pretrained models. By reducing both the number of trainable parameters and peak memory requirements by orders of magnitude, these methods make downstream deployment feasible for smaller organizations and resource-constrained environments. In vision, language, and multimodal tasks, their empirical effectiveness rivals or surpasses full fine-tuning and traditional adapter-based methods.

A plausible implication is the democratization of large model adaptation: reparameterized PEFT provides a unified, theoretically principled framework that can be specialized to or extended for task-, domain-, or hardware-specific constraints, enabling efficient transfer learning at scale (Prottasha et al., 19 Apr 2025).