OrthoGeoLoRA: Geometric Adaptation Framework

• OrthoGeoLoRA is a geometric, parameter-efficient adaptation framework that addresses LoRA limitations using an SVD-inspired, orthogonal update structure.
• It enforces Stiefel manifold constraints to eliminate gauge freedom, scale ambiguity, and rank collapse, thereby optimizing convergence and representational capacity.
• Empirical results demonstrate that OrthoGeoLoRA significantly boosts metrics like Recall@3 and NDCG@3, enhancing performance in large language model fine-tuning.

OrthoGeoLoRA is a geometric parameter-efficient adaptation framework designed to address representational and optimization limitations in standard Low-Rank Adaptation (LoRA) methods for foundation models and text encoders. It replaces LoRA’s unconstrained low-rank update with an SVD-inspired structure, enforcing orthogonal constraints (Stiefel manifold) on the update factors. This design eliminates gauge freedom, scale ambiguity, and rank collapse, enabling full utilization of the assigned parameter budget and improving convergence properties, representational capacity, and downstream performance in realistic concept retrieval tasks and LLM fine-tuning (Wang et al., 14 Jan 2026, Park et al., 25 Aug 2025).

1. Geometric Pathologies in Standard LoRA

Standard LoRA adapts a frozen weight matrix $W_0 \in \mathbb{R}^{d_{out} \times d_{in}}$ by introducing a low-rank update $\Delta W = BA^{\top}$, where $A \in \mathbb{R}^{d_{in} \times r}$, $B \in \mathbb{R}^{d_{out} \times r}$, and $r \ll d$. While this decreases trainable parameters, three major geometric pathologies arise:

• Gauge Freedom: For any invertible $M \in \mathbb{R}^{r \times r}$, the transform $B' = BM$, $A' = A(M^{-T})$ yields an identical update ($B'A'^{\top} = BA^{\top}$), resulting in flat optimization valleys and redundant parameterizations.
• Scale Ambiguity: Multiplying $B$ by $c$ and $A$ by $1/c$ leaves $BA^{\top}$ unchanged. This confounds feature direction and magnitude, complicating learning rate and regularization schedules.
• Rank Collapse: Unconstrained gradient descent frequently produces collinear columns and a lower effective rank in $BA^{\top}$ than nominal $r$, wasting adaptation capacity.

These flaws do not occur in the ideal SVD-based approximation $U_r\Sigma_rV_r^{\top}$, in which $U_r$ and $V_r$ are orthonormal and $\Sigma_r$ is diagonal (Wang et al., 14 Jan 2026).

2. OrthoGeoLoRA’s SVD-Inspired Formulation

OrthoGeoLoRA parameterizes the low-rank update as:

$\Delta W = B \Sigma A^{\top}$

where

• $A \in \operatorname{St}(d_{in}, r)$,
• $B \in \operatorname{St}(d_{out}, r)$,
• $$\Sigma = \operatorname{diag}(\sigma_1, \ldots, \sigma_r) \succeq 0$$,

with

$$\operatorname{St}(d, r) = \{X \in \mathbb{R}^{d \times r}: X^{\top}X = I_r\}$$

the Stiefel manifold of matrices with orthonormal columns. This structure:

• Removes gauge and scale freedom; $\Delta W$ is unique up to discrete permutation and sign-flip ambiguities.
• Decouples directional information from scaling; direction is encoded by $B$, $A$, and magnitude in $\Sigma$.
• Prevents rank collapse; $A$, $B$ have full rank $r$ unless $\sigma_i \to 0$ (Wang et al., 14 Jan 2026, Park et al., 25 Aug 2025).

3. Optimization: Stiefel Constraints and Geometric Reparameterization

OrthoGeoLoRA enforces $A$, $B$ orthogonality either by explicit Riemannian optimization or geometric reparameterization compatible with standard optimizers:

• Riemannian Optimization (Stiefel-QR Retraction): The Euclidean gradient $\nabla_B L$ is projected onto the tangent space $T_B\operatorname{St}(d, r)$ via

$$\xi = \nabla_B L - B\,\text{Sym}(B^{\top} \nabla_B L)$$

where $\text{Sym}(M) = (M + M^{\top})/2$. Retraction is performed by QR decomposition: after each step, set $B_{next} = Q$ from $Y = B - \eta \xi$, $Y = QR$ (Park et al., 25 Aug 2025).

• Geometric Reparameterization (QR/Householder Orthogonalization): Maintain unconstrained Euclidean parameters $\hat{A}$, $\hat{B}$, $s$. On each forward pass,
  1. $$A = \text{Orth}(\hat{A}),\quad B = \text{Orth}(\hat{B})$$
  2. $$\Sigma = \mathrm{diag}(\text{softplus}(s) + \epsilon)$$ Backpropagation respects manifold geometry; no projection or custom step is required for $A$, $B$ (Wang et al., 14 Jan 2026).

Approach	A/B Update Method	B Orthogonality Enforced?
AdamW on Euclidean space	Gradient step	No
Stiefel-QR Retraction	Riemannian gradient+QR	Yes
Geometric Reparameterization	Orth(·), AdamW	Yes (by construction)

4. Integration with Orthogonal Layers and Manifolds

OrthoGeoLoRA leverages geometric constructions for strict orthogonality in deep adapters:

• Each orthogonal layer is represented by a base $Q$ and a tangent perturbation $A \in \mathfrak{so}_n$ (skew-symmetric matrices for $O_n(\mathbb{R})$).
• Updates are performed as $Q \leftarrow Q\,\exp(-\eta \Delta)$, where $\Delta \in \mathfrak{so}_n$ and $\|\Delta\|_F^2$ controls geodesic step size.
• Interpolation between weights $Q_0, Q_1$ uses the geodesic $$Q(t) = Q_0 \exp\bigl(t\,\text{Log}_O(Q_0^{\top}Q_1)\bigr)$$, with the principal skew-symmetric logarithm construct as in (Dolcetti et al., 2016).
• Matrix exponentials use either the Rodrigues formula for skew-SVD or Pade-approximation; principal log recovery relies on Schur decompositions (Dolcetti et al., 2016).

These techniques ensure strict adherence to orthogonality, controlled manifold drift, and computational tractability ($O(n^3)$ per layer).

5. Empirical Results and Benchmarking

OrthoGeoLoRA was benchmarked on a hierarchical concept retrieval task with the European Language Social Science Thesaurus (ELSST):

• Dataset: Multilingual policy abstracts and blurb-generation using DeepSeek-V3; 24 synthetic descriptions per concept; annotated by experts (accuracy 3.95/5, fluency 2.71/3, inter-annotator $\alpha=0.85, 0.72$).
• Metrics: Mean Reciprocal Rank (MRR), Recall@1, Recall@3, NDCG@1, NDCG@3.
• Adapter rank: $r=8$; base encoder multilingual-e5-small; optimizer AdamW; batch $128$ (Wang et al., 14 Jan 2026).

Method	MRR	Recall@1	Recall@3	NDCG@1	NDCG@3
Zero-shot	0.954	0.304	0.831	0.939	0.889
LoRA	0.973	0.315	0.898	0.963	0.936
AdaLoRA	0.948	0.299	0.833	0.923	0.882
DoRA	0.974	0.316	0.894	0.965	0.935
LoHa	0.961	0.308	0.846	0.947	0.897
LoKr	0.955	0.304	0.850	0.937	0.898
OrthoGeoLoRA	0.983	0.321	0.939	0.978	0.964

OrthoGeoLoRA achieves +4.1 Recall@3 and +2.8 NDCG@3 gain over LoRA for $r=8$. It shows faster convergence, full singular-value utilization (flat spectrum), and negligible inference overhead (Wang et al., 14 Jan 2026).

For LLM fine-tuning across commonsense, reading comprehension, and mathematics (r=16), Stiefel-LoRA outperforms AdamW-LoRA by up to 12.1 points on BoolQ and 3–4 points elsewhere; effective rank analysis shows Stiefel optimization maintains rank $r$ while AdamW typically wastes capacity ($R_{eff}\approx12$ for $r=16$) (Park et al., 25 Aug 2025).

6. Practical Implementation and Algorithmic Outline

Core adapter implementation proceeds as:

• Parameters: $\hat{A} \in \mathbb{R}^{d_{in}\times r}$, $\hat{B} \in \mathbb{R}^{d_{out}\times r}$, $s \in \mathbb{R}^{r}$.
• Forward:
  1. $$A = \text{Orth}(\hat{A}),\quad B = \text{Orth}(\hat{B})$$
  2. $$\Sigma = \operatorname{diag}(\mathrm{softplus}(s) + \epsilon)$$
  3. $u = A^{\top}x$, $v = \Sigma u$, $\Delta y = B v$.
  4. Output $y = W_0 x + (\alpha/r)\Delta y$.

• Backpropagation via AdamW on $(\hat{A}, \hat{B}, s)$; orthogonalization is differentiable.

• At inference, $B\Sigma A^{\top}$ can be absorbed into $W_0$; training overhead (QR or Householder) is negligible for typical $r$ (Wang et al., 14 Jan 2026).

For LLMs, Stiefel step size ($\eta_B \approx 0.3$), rank ($r=16,32,64$), scaling ($\alpha=32,64$), and dropout (0.05) are used with LoRA applied to Q, K, V, and other layers; QR or Cayley retraction is standard for $B$ updates (Park et al., 25 Aug 2025).

7. Theoretical Foundations from Manifold Geometry

The differential-geometric underpinnings, especially for strict orthogonal layers and geodesic interpolation, are derived from the Frobenius-induced Riemannian metric on $O_n(\mathbb{R})$:

• The tangent space at $Q$ is $T_QO_n = \{V: Q^{\top}V$ skew-symmetric$\}$.
• Geodesics: $\gamma(t) = Q \exp(tA)$, $A \in \mathfrak{so}_n$.
• Distance: $d_F(Q_0,Q_1) = \|\Log_O(Q_0^{\top} Q_1)\|_F$.
• Principal logarithms and SVD-like block structure support orthogonal updates in adapters (Dolcetti et al., 2016).
• OrthoGeoLoRA exploits these constructs for parameterization, gradient flow, and controlled drift within $O(n^3)$ cost per layer.

---

OrthoGeoLoRA enforces SVD-like structure on adapter updates by leveraging Stiefel manifold constraints, geometric parameterization, and Riemannian optimization. This approach resolves critical architectural and optimizer inefficiencies in LoRA-based PEFT, achieves superior performance with full representational utilization, and integrates into deep encoder architectures and orthogonal layers according to established manifold geometry, with empirical effectiveness validated on realistic social science and LLM benchmarks (Wang et al., 14 Jan 2026, Park et al., 25 Aug 2025, Dolcetti et al., 2016).