Reparameterized PEFT: Efficient Fine-Tuning

• The technique introduces a minimal, trainable update to frozen pre-trained weights, reducing parameters and memory requirements.
• It employs methods such as low-rank factorization, adapters, and prefix-tuning to maintain performance, often matching full fine-tuning accuracy.
• Applications span NLP, vision, and federated learning, with efficient inference achieved by folding the updates into the main model.

Reparameterized Parameter-Efficient Fine-Tuning (PEFT) is a class of techniques for adapting large pre-trained models to new tasks by freezing the vast majority of parameters and updating only a small, structured subset. These methods achieve substantial reductions in trainable parameters, memory footprint, and computational cost while maintaining—or in some cases even exceeding—the performance of full-model fine-tuning. The central principle is to introduce or learn a low-dimensional or structured update to pre-trained weights, rather than modifying the entire parameter set. Reparameterized PEFT describes approaches where this update is itself parameterized in a compact or factorizable mathematical form.

1. Fundamental Principles and General Framework

In the standard reparameterized PEFT setup, the goal is to adapt a heavyweight model (parameters Θ₀) to a new task by introducing minimal extra parameters Δ, without incurring significant additional inference cost. Typically, the approach is to freeze most of the model weights and introduce a set of trainable parameters that reparameterize the original network in a restricted, often low-dimensional space. Let $W_0 \in \mathbb{R}^{d \times k}$ be a frozen pre-trained weight matrix. The effective, fine-tuned weight is given by:

$W' = W_0 + \Delta W,$

where $\Delta W$ is a reparameterized update, parameterized in a way that introduces few new parameters with strong inductive biases (e.g., low-rank structure). During training, only the parameters underlying $\Delta W$ are updated; at inference, $\Delta W$ is typically merged or "folded" into $W_0$, preserving computational efficiency (Wang et al., 2024).

The motivation for this reparameterization is threefold:

• To reduce memory requirements during training, as only a small subset of parameters requires gradients,
• To minimize inference overhead by absorbing the trained update into the existing model structure,
• To improve generalization in low-data regimes by restricting learned updates to constrained subspaces.

2. Core Reparameterized PEFT Methodologies

The dominant instantiations employ a variety of factorizations and parameterizations for $\Delta W$, summarized in the following table:

Method	Update Formulation	Trainable Params per Layer
LoRA	$B A^\top$	$r(d + k)$
Adapter	$W_D \phi(W_E h)$	$m d$ per layer
Prefix-Tuning	$P_k$, $P_v$	$\ell d$ per layer
Hypernetworks	$H(e_\mathrm{task})$	$\#$(hypernet params)

Low-Rank Adaptation (LoRA)

LoRA parametrize the update as $\Delta W = B A^\top$, where $A \in \mathbb{R}^{k \times r}$, $B \in \mathbb{R}^{d \times r}$, and $r \ll \min(d, k)$. Only $A$ and $B$ are trained, leading to a substantial reduction in trainable parameters (typically 0.1–1% per layer compared to full fine-tuning) (Wang et al., 2024, Han et al., 2024, Prottasha et al., 19 Apr 2025). At inference, $W' = W_0 + B A^\top$ can be precomputed with negligible runtime overhead.

Variants include:

• Dynamic-Rank LoRA ("DyLoRA"): Allows the effective rank $r$ to vary adaptively during optimization (Han et al., 2024).
• Adaptive-LoRA ("AdaLoRA"): Introduces an SVD-like $\Delta W= P \Lambda Q$, learning which singular values to retain (Han et al., 2024).
• Sparse-LoRA, KronA, Compacter: Use sparsity or Kronecker decompositions for further parameter compression.

Adapter Modules

Adapters insert small bottleneck feedforward modules into the Transformer, parameterized as:

$$h_{i+1} = h_i + W_D^{(i)} \phi(W_E^{(i)} \mathrm{LN}(h_i)),$$

where $W_E^{(i)} \in \mathbb{R}^{m \times d}$, $W_D^{(i)} \in \mathbb{R}^{d \times m}$, and only the bottleneck modules are updated (Wang et al., 2024). Adapters enable robust transfer at a typical parameter cost of $\sim$1% per layer.

Prefix and Prompt-Tuning

Prefix-tuning appends trainable "virtual" token embeddings to the input of each self-attention block:

$K' = [P_k; K],\quad V' = [P_v; V],$

with $P_k, P_v \in \mathbb{R}^{\ell \times d}$. Only these prefix tokens are optimized, with total overhead typically in the 0.01–0.1% range (Wang et al., 2024).

Hypernetwork-Based Methods

Hypernetworks generate adapter weights as a function of a task embedding, so the adaptation is itself a learned function (Wang et al., 2024). Only the hypernetwork parameters and task embedding are trained; the approach is effective for multi-task and dynamic adaptation scenarios.

3. Empirical Performance and Overhead

Quantitative studies consistently show that reparameterized PEFT methods achieve retention of $>$95% of full fine-tuning accuracy on NLP and vision benchmarks while reducing trainable parameters by 1–2 orders of magnitude (Wang et al., 2024). Specifically:

• LoRA (GLUE/QA): Matches or slightly exceeds (by 0.1–0.3%) full fine-tuning at $<$0.5% parameter cost.
• AdaLoRA: Provides further improvements (1–2%) over fixed-rank LoRA in low-data regimes (Han et al., 2024).
• Adapters: Slight accuracy gap to full fine-tuning but highly robust across domains.
• Prefix/prompt-tuning: Effective on generation tasks, less so on deep classification tasks.

Reparameterized PEFT also scales favorably in system implementations:

• Memory savings from only storing gradients for adapters,
• Inference cost is negligible since the update is folded into base weights,
• Highly scalable in multi-task and federated learning scenarios, as task-specific adapters remain small (Bian et al., 29 Apr 2025).

4. Theoretical Underpinnings and Inductive Bias

Reparameterized PEFT leverages the empirical observation that gradient updates of large pre-trained models typically lie in a low-dimensional subspace. This is supported by evidence that the Jacobian of model outputs w.r.t. weights spans a subspace whose dimension grows slowly with network size (Prottasha et al., 19 Apr 2025). Matrix decomposition theory (Eckart–Young-Mirsky theorem) states the best rank-$r$ approximation to any weight update captures the majority of its variance. Introducing structured low-rank or Kronecker adapters imposes strong inductive biases, which:

• Regularize learning in data-poor regimes,
• Minimize the risk of overfitting,
• Facilitate rapid adaptation and modularity (via simple swapping/merging of adapter modules).

Recent advances formalize these intuitions by analyzing the impact of different subspace constraints (e.g., LoRA vs. AdaLoRA vs. SVD-based extensions) and demonstrate tighter couplings (orthogonality, diagonal dominance) can hamper expressiveness, whereas flexible, unconstrained factorizations (e.g., FLoRA: $\Delta W = A G B$ with non-diagonal $G$) yield optimal downstream performance (Si et al., 2024).

5. Applications and Domain-Specific Extensions

Reparameterized PEFT has demonstrated effectiveness in:

• LLM adaptation for text classification, QA, and generative tasks (Prottasha et al., 19 Apr 2025, Wang et al., 2024).
• Vision-Language and vision-only models (e.g., ViT, SAM-COBOT), exploiting Kronecker adapters, hypercomplex layers, spectral reparameterization, and tensor decompositions (Peng et al., 2023, Liang et al., 2024).
• Federated learning of foundation models: LoRA, FedEx-LoRA, and LoRA-FAIR achieve communication and computation costs $>$50\times lower than full-model FL, with negligible loss in aggregate or personalized performance (Bian et al., 29 Apr 2025).
• 3D point cloud learning: Spectral reparameterization enables highly efficient fine-tuning with less than 1% of baseline parameters, outperforming full model adaptation on several ScanObjNN tasks (Liang et al., 2024).

6. Implementation Guidelines and Research Directions

Best practices for deploying reparameterized PEFT include:

• Choose LoRA with $r=4$–16 for NLP tasks; dynamic/adaptive rank variants where automatic budget tuning is required (Han et al., 2024).
• Use adapters or hypernetwork methods for robustness or multi-task settings.
• Prefix-tuning is effective for large context-length, generative applications.
• Enforce orthogonality or apply sparsity-promoting regularization where appropriate (e.g., AdaLoRA, SoRA).
• In multi-tenant or federated systems, coordinate batching of large MVMs and handle adapters with side-channel compute for maximal throughput (Wang et al., 2024, Bian et al., 29 Apr 2025).

Open research directions highlighted include:

• Multi-objective and hybrid PEFT (combining syntax, semantics, multimodal connectors),
• Automatic neural architecture search for module placement and bottleneck sizing,
• Continual/lifelong adaptation while preserving previous tasks,
• Calibration and uncertainty estimation for low-data fine-tuning,
• Differential privacy integration into PEFT,
• Theoretical analysis of update subspaces, expressivity limits, and convergence in distributed/federated contexts (Wang et al., 2024, Bian et al., 29 Apr 2025).

7. Summary Table of Main Reparameterized PEFT Techniques

Technique	Update Structure	Parameter Savings	Use Cases	Notable Variants/Extensions
LoRA	$B A^\top$	$0.1\%$–$1\%$	NLP, vision, FL	AdaLoRA, DyLoRA, SoRA
Adapter	$W_D \phi(W_E h)$	$\sim 1\%$	Robust transfer, multi-domain	Hypernetwork-adapters
Prefix/Prompt Tuning	$P_k$, $P_v$ in attention	$0.01$–$0.1\%$	Generation, prompting	Soft prompt, virtual tokens
Kronecker/Matrix Decomp	KronA, Compacter	$<0.5\%$	ViT, dense prediction, low-memory environments	Compacter, KronA
Spectral Reparameterization	GFT basis + PCSA	$<1\%$	Point cloud, 3D data	PCSA, PointGST

Performance reported for these methods is typically within 1–3% of full-model fine-tuning, often at 10–100$\times$ lower parameter and memory footprint, with negligible inference cost once updates are folded (Wang et al., 2024, Liang et al., 2024, Han et al., 2024, Prottasha et al., 19 Apr 2025).

---

Reparameterized PEFT represents a critical advance in scalable model adaptation, enabling efficient, generalizable, and resource-friendly fine-tuning of foundation models across modalities and deployment scenarios (Wang et al., 2024, Han et al., 2024, Bian et al., 29 Apr 2025, Prottasha et al., 19 Apr 2025).