High-Rank Multiplicative PEFT Adaptation

• The technique introduces novel multiplicative parameterizations, such as LoRDS and KRAdapter, that elevate effective rank to approximate full matrix adaptation with reduced trainable parameters.
• Methods leverage advanced strategies like tensor decompositions, low-rank scaling, and structured matrix updates to break the limitations of classical low-rank PEFT approaches.
• Empirical benchmarks demonstrate improved downstream performance and robust generalization in LLMs and multimodal architectures, with efficiency matching full fine-tuning.

High-rank multiplicative parameter-efficient fine-tuning (PEFT) adaptation refers to a family of methods designed to overcome the intrinsic expressivity limitations of classical low-rank adapters in transformer-based models. Unlike traditional additive, strictly low-rank methods, modern high-rank multiplicative techniques leverage either novel parameterizations—such as low-rank scaling, tensor-network structures, distributed orthogonalization, or structured unrestricted-rank updates—or different forms of multiplication to generate weight updates whose effective rank approaches or matches full matrix adaptation, all with dramatically reduced trainable parameter count and, in some cases, without additional inference overhead. Recent research demonstrates these approaches can unlock performance and generalization gains across LLMs, vision-language architectures, and multimodal transformers in both in-distribution and out-of-distribution tasks (Tang et al., 30 Jan 2026, Sehanobish et al., 2024, Gurung et al., 23 Sep 2025, Albert et al., 1 Aug 2025, Gu et al., 3 Sep 2025, Wang et al., 24 May 2025).

1. Motivation: Limitations of Conventional Low-Rank PEFT

Conventional PEFT methods such as LoRA parameterize updates $\Delta W$ to a frozen pre-trained weight $W_0$ by factorizing $\Delta W = BA$ with $$A \in \mathbb{R}^{r \times d_\text{in}}, B \in \mathbb{R}^{d_\text{out} \times r}, r \ll \min(d_\text{in}, d_\text{out})$$ (Kalajdzievski, 2023). This imposes a hard upper-bound on the update rank ($\operatorname{rank}(\Delta W) \leq r$), restricting the adaptation space to a low-dimensional linear submanifold determined by $r$. Empirical spectral analyses show downstream adaptation "deltas" induced by full fine-tuning often exhibit high effective rank, especially on complex reasoning, program synthesis, or robustness tasks (Wang et al., 24 May 2025, Albert et al., 1 Aug 2025). As a result, strict low-rank approaches cap attainable accuracy and generalization. While increasing $r$ appears a direct remedy, it can cause unstable optimization and a linear growth in parameter count, prompting research into alternative PEFT parameterizations that guarantee high (or unrestricted) update rank under strict efficiency constraints.

2. High-Rank Multiplicative Parameterizations

a. Low-Rank Decomposed Scaling (LoRDS)

LoRDS reframes block-wise or element-wise quantization by modeling the elementwise scaling matrix $S$ as a low-rank product $S = BA$ and effecting adaptation through element-wise multiplication: $W' = Q \odot S$ with $Q = \operatorname{Round}(W / S)$ (Tang et al., 30 Jan 2026). During PEFT, $Q$ is frozen, and only $B,A$ are trained. The update $\Delta W_\text{mul} = Q \odot (B'A' - BA)$ empirically exhibits a high-rank, long-tailed spectrum since the element-wise product "unfolds" low-rank factors across all singular directions of $Q$. At export, $S$ is merged into the quantization routine, incurring zero inference cost.

b. Random Tensor Networks and Tucker Decompositions (TeRA)

TeRA parameterizes weight updates using a random Tucker network: for a weight matrix $W_0 \in \mathbb{R}^{J_1 \times J_2}$, $\Delta W$ is tensorized into $N$ modes (folded to $\Delta \mathcal{W}$), with only small diagonal scaling vectors $\{d^{(i)}\}$ trained, while the random core $\mathcal{G}$ and factors $\{A^{(i)}\}$ are frozen and shared (Gu et al., 3 Sep 2025). This allows the effective rank of the update to scale with mode dimensions, decoupling expressivity from parameter count. In specific parameterizations ($R_i=I_i$), the rank of the unfolded $\Delta W$ can be made full.

c. Khatri–Rao Product Adapters (KRAdapter)

KRAdapter leverages the Khatri–Rao product to obtain column-wise Kronecker factorization: $\Delta W = \alpha (U \odot V)$ with $$U \in \mathbb{R}^{k_1 \times d_\text{in}}, V \in \mathbb{R}^{k_2 \times d_\text{in}}$$, $k_1 k_2 \geq d_\text{out}$ (Albert et al., 1 Aug 2025). This structure, unlike the matrix product in LoRA, guarantees—under mild conditions—full column rank almost surely for random $U, V$, achieving a high effective rank while maintaining parameter efficiency akin to LoRA.

d. Row and Column Multiplicative Scaling (HyperAdapt)

HyperAdapt introduces trainable diagonal row and column scalings $A = \operatorname{diag}(a_1, \dots, a_n)$, $B = \operatorname{diag}(b_1, \dots, b_m)$ such that $W' = A W_0 B$ (Gurung et al., 23 Sep 2025). This $n + m$ parameterization leaves the rank of the update $\operatorname{rank}(\Delta W)$ upper-bounded by $2r$ (where $r = \operatorname{rank}(W_0)$), so nearly full-rank when $W_0$ is full-rank.

e. Structured Unrestricted-Rank Matrices (SURM)

SURMs use low displacement rank matrices (LDRM) to encode updates in forms such as sums of circulant or Toeplitz structures: $\Delta W = \sum_{i=1}^{r} Z_1(g_i) Z_{-1}(h_i)$ (Sehanobish et al., 2024). For $r=1$ (circulant case), the update requires only $2n$ parameters but can be full rank, depending on the structure and optimization of generator vectors. Kronecker-based variants ($\Delta W = A \otimes B$) also enable high-rank updates for minimal parameter cost.

f. Distributed Orthogonal Adapters (HD-PiSSA)

HD-PiSSA distributes initialization of low-rank adapters across $K$ devices such that each GPU gets an orthogonal set of rank-$r$ SVD modes of $W_0$. Gradients for the adapters are computed independently, then aggregated and applied to the shared $W$, yielding updates with effective rank up to $2Kr$ (up to dimension limit), vastly exceeding rank limits of standard PEFT (Wang et al., 24 May 2025).

3. Mathematical Characterization of Expressivity

All high-rank multiplicative PEFT methods fundamentally break the explicit rank constraint of classic low-rank adaptation through either multiplicative integration, mode-wise scaling, or structure-enabled expansion. In LoRDS, the Hadamard product $Q \odot (B'A' - BA)$ elevates the effective rank of the update—empirically exhibiting long-tailed singular spectra—where a strictly low-rank additive $\Delta W$ would have spectral decay truncated at $r$ (Tang et al., 30 Jan 2026). For KRAdapter and certain SURMs, the mathematical guarantees (e.g., Khatri–Rao product theory, properties of LDRM) assert that updates almost surely attain full column/row rank under broad random initialization assumptions (Albert et al., 1 Aug 2025, Sehanobish et al., 2024).

TeRA analytically decouples achievable update rank from parameter count by fixing random projectors and only training small diagonal scaling vectors. Theoretical bounds show that, for suitable tensor order and factorization parameters, the update approaches arbitrary full-rank corrections, with residual error converging as mode dimensions grow (Gu et al., 3 Sep 2025).

HyperAdapt's row and column scaling achieves actual update rank $2r$ (when $W_0$ is full rank), yet with only $n+m$ parameters (Gurung et al., 23 Sep 2025).

4. Empirical Performance and Comparative Studies

Empirical benchmarks consistently show high-rank multiplicative PEFT solutions surpassing classical low-rank additive baselines and, in many cases, approaching full fine-tuning quality. On Llama3-8B, LoRDS achieves a 9.6% gain in downstream PEFT performance (Commonsense-170k, 4-bit quantization) over QLoRA while matching or exceeding SOTA quantitative metrics (Tang et al., 30 Jan 2026). Table 1 summarizes representative empirical results:

Method	Params (%)	Acc./Metric (Range)	Notes
LoRA (r=32/128)	0.17–1.05	78–87 (CSR)/86–87	Shows saturating returns
LoRDS	1–2 (84M/84M)	87.68 (Commonsense)	Zero inference overhead
HyperAdapt	0.03 / $n+m$	86–87 (GLUE/CSR)	Nearly full-rank updates
KRAdapter	$2\sqrt{d_\text{in} d_\text{out}}$	Up to +2% over LoRA	Full-rank, robust OOD
SURM (Circulant)	$2n$	+1–2% vs LoRA	FFT-inference, O(n) params
HD-PiSSA (K=8)	$8r$ (per device)	+10 points (multi-t)	16x effective rank vs LoRA
TeRA	0.0033%	Parity with HiRA	Full-rank guarantee

Rank ablations confirm that only high-rank methods exhibit monotonic performance gains as parameter count increases; low-rank additive PEFT (e.g., LoRA) saturates rapidly for $r \gtrsim 16–32$ (Kalajdzievski, 2023, Albert et al., 1 Aug 2025). Singular spectrum plots universally display long-tailed distributions for multiplicative/high-rank PEFT, in contrast to the strictly truncated spectra of low-rank additive methods (Tang et al., 30 Jan 2026, Gurung et al., 23 Sep 2025).

5. Efficiency, Practical Integration, and Limitations

Multiplicative high-rank PEFT methods generally achieve parameter and computational efficiencies on par with—or even exceeding—classical approaches. In LoRDS and HyperAdapt, no additional operators are required at inference: all learned scalings absorb into quantized kernels or are materialized once, preserving throughput and memory footprint (Tang et al., 30 Jan 2026, Gurung et al., 23 Sep 2025). SURM variants with circulant or Toeplitz structure support $O(n)$ parameter counts and use FFTs for matvecs, ensuring sub-quadratic inference overhead (Sehanobish et al., 2024). For KRAdapter and TeRA, although parameter counts may marginally exceed lowest-rank LoRA for high expressivity, the methods retain practical tractability for billion-parameter models due to their favorable scaling.

Optimally, block sizes, learning rates, and rank/tensorization choices should be tuned per-architecture; e.g., LoRDS recommends $B=64$, $r=\left\lfloor\frac{nm}{B(n+m)}\right\rfloor$ for LLM layers (Tang et al., 30 Jan 2026), TeRA achieves best tradeoff by tensorizing only one matrix dimension, and SURM recommends circulant (rank-1) as default.

Limitations of high-rank multiplicative PEFT include potential overfitting with extreme parameterization or small datasets (Kalajdzievski, 2023, Gu et al., 3 Sep 2025), as well as the need for careful numerical initialization (e.g., the rsLoRA scaling fix for large $r$ (Kalajdzievski, 2023)) or device coordination (as in HD-PiSSA (Wang et al., 24 May 2025)). Some methods introduce additional complexity in kernel or distributed management for massive models.

6. Theoretical and Algorithmic Innovations

Recent high-rank multiplicative PEFT frameworks are theoretically justified by advances in matrix/tensor decomposition, random projection theory, and block-circulant structure properties. Notably, the switching from additive to multiplicative parameterization (e.g., Hadamard product in LoRDS or HyperAdapt) alters the effective update spectrum without sacrificing parameter efficiency, and analytical results guarantee full or near-full rank achievable with parameter budgets orders of magnitude smaller than full fine-tuning (Gu et al., 3 Sep 2025, Gurung et al., 23 Sep 2025).

Algorithmically, these approaches integrate smoothly with quantization-aware training (LoRDS), data-parallel distributed training (HD-PiSSA), or standard transformer pipelines (HyperAdapt, SURM). Pseudocode routines are materially identical or require only minor adjustments (e.g., insertion of per-layer scalings or device-wise orthogonal basis initialization).

7. Comparative Analysis and Prospects

There is now a diverse ecosystem of high-rank, multiplicative PEFT techniques. All share the goal of unifying the parameter efficiency of classic PEFT with the expressivity and empirical coverage of full fine-tuning. As reported in cross-domain studies, modern high-rank multiplicative PEFTs can achieve parity with full fine-tuning or state-of-the-art PEFT, with significant reductions in trainable parameters and resource load, often alongside quantization or multi-modal extensions (Tang et al., 30 Jan 2026, Sehanobish et al., 2024, Gu et al., 3 Sep 2025). It is plausible that future research will further integrate these parameterizations with dynamic adaptation, pruning, or meta-learning for maximally flexible yet budgeted adaptation strategies.

---

References

• "Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation" (Tang et al., 30 Jan 2026)
• "A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA" (Kalajdzievski, 2023)
• "TeRA: Vector-based Random Tensor Network for High-Rank Adaptation of LLMs" (Gu et al., 3 Sep 2025)
• "Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product" (Albert et al., 1 Aug 2025)
• "HD-PiSSA: High-Rank Distributed Orthogonal Adaptation" (Wang et al., 24 May 2025)
• "HyperAdapt: Simple High-Rank Adaptation" (Gurung et al., 23 Sep 2025)
• "Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuning" (Sehanobish et al., 2024)