RLola Specifications: A Riemannian Approach

• RLola is a geometry-aware optimization framework for low-rank adaptation that resolves factorization ambiguity by mapping adapters to a fixed-rank manifold.
• It employs intrinsic Riemannian optimization methods, using efficient SVD-based retraction and tangent space projections to accelerate convergence.
• The framework features principled initialization via BackPropRSVD, ensuring robust and stable updates across diverse architectures and tasks.

RiemannLoRA (RLola) specifications define a parameter-free, geometry-aware optimization framework for low-rank adaptation (LoRA) in the fine-tuning of large pre-trained neural network models. By casting LoRA adapters as points on a fixed-rank matrix manifold and conducting intrinsic Riemannian optimization, RLola eliminates the rank-factorization ambiguity and enables principled, task-driven initialization. The mathematical, numerical, and algorithmic design of RLola improves both convergence speed and final accuracy over traditional LoRA variants across diverse architectures and tasks (Bogachev et al., 16 Jul 2025).

1. Mathematical Foundations and Manifold Geometry

Let $W\in\mathbb R^{m\times n}$ be a pretrained weight to be fine-tuned with a rank-$r$ LoRA adapter: $\Theta = W + AB^\top$, with $A\in\mathbb R^{m\times r}$ and $B\in\mathbb R^{n\times r}$, $r<\min(m,n)$. Standard LoRA minimization operates in the Euclidean product space $(A,B)$: $\min_{A,B}\; \mathcal L(\Theta,W+AB^\top)$ This factorized formulation introduces an inherent ambiguity, as $AB^\top = (AS)(BS^{-\top})^\top$ for any invertible $S\in\mathrm{GL}_r$. RLola, by contrast, prescribes that the adapter $\Delta W$ be regarded intrinsically as a point $$X\in\mathcal M_r = \{X\in\mathbb R^{m\times n} : \mathrm{rank}(X)=r\}$$, an embedded smooth manifold of dimension $(m+n)r-r^2$.

For stability, RLola maintains orthonormal representations: $X=A_L B^\top$ with $A_L^\top A_L=I_r$ or $X=AB_R^\top$ with $B_R^\top B_R=I_r$. This approach allows all updates to respect the geometry of the fixed-rank constraint, avoiding overparameterization issues such as drift in the $r\times r$ Gram matrices.

2. Optimization on the Fixed-Rank Manifold

Given a point $X=A_LB^\top \in\mathcal M_r$, the tangent space $T_X\mathcal M_r$ is defined as: $$T_X\mathcal M_r = \left\{ \xi = \dot{A}B^\top + A_L\dot{B}^\top \mid \dot{A}^\top A_L = 0 \right\}$$ The Riemannian gradient is computed as the projection of the ambient (Euclidean) gradient: $$\mathrm{grad}\,F(X) = P_{T_X\mathcal M_r}(\nabla F(X))$$ where the projection is given efficiently by: $$P_{T_X}(Z) = (I - A_LA_L^\top)Z B_RB_R^\top + A_LA_L^\top Z$$

A "double-backprop" trick avoids materialization of the full $\nabla F(X)\in\mathbb R^{m\times n}$: gradients are computed with respect to auxiliary variables of shape $m\times r$ and $n\times r$, which reduces both memory and computation demands.

To move along the manifold, RLola computes an ambient update direction $\xi\in T_X\mathcal M_r$, forms $X+\xi$, and retracts this back to the closest rank-$r$ point via truncated SVD: $$R_X(\xi) = \arg\min_{Y\in\mathcal M_r} \|Y - (X+\xi)\|_F = U_r\Sigma_r V_r^\top$$ Since the rank of $(X+\xi)$ is at most $2r$, the SVD operates in $2r\times 2r$ subspaces, yielding a per-step complexity of $\mathcal O((m+n)r^2 + r^3)$.

3. Initialization and Algorithmic Procedure

Instead of standard LoRA's random or zero initialization, RLola achieves a principled initialization by seeking the adapter $\Delta W^{(0)}_*\in\mathcal M_r$ whose tangent space aligns maximally with the loss gradient: $$\Delta W^{(0)}_* = \arg\max_{X\in\mathcal M_r}\|P_{T_X}\nabla_W\mathcal L(W)\|_F^2$$ The optimal solution is given by the rank-$r$ part of the top $2r$-truncated SVD of $\nabla_W\mathcal L(W)$, generalizing existing LoRA-GA initializations. Approximate computation is performed by randomized power iteration ("BackPropRSVD") with $2(q+1)$ backpropagations and overall cost $\mathcal O((m+n)r^2)$.

RLola's main loop (per adapter, layer, or module):

1. Initialize $A_L, B$ from BackPropRSVD.
2. For each step:
   • Orthonormalize right factor ($B_R$)
   • Compute double-backprop partials ($\dot A, \dot B$)
   • Compute Riemannian momentum and transport to current tangent space
   • Optionally simulate Adam by normalizing step magnitudes
   • Apply update, retract to rank-$r$ via SVD
   • Update momentum buffers

Recommended hyperparameters (Commonsense and MetaMath experiments): $r=16$; dropout $0.05$; LR with linear warmup/decay (warmup at $0.1$ of total steps); batch size $64$; epochs $2$ (Commonsense), $1$ (MetaMathQA) (Bogachev et al., 16 Jul 2025).

4. Numerical Stability and Complexity

RLola enforces orthonormality for $A_L$ and $B_R$ using frequent QR decompositions ($\mathcal O((m+n)r^2)$), preventing rank collapse and accumulation of numerical errors. All SVDs are of size at most $2r\times 2r$, performed by LAPACK or randomized power methods.

For each training step, RLola's computational costs above a standard backpropagation are:

• SVD retraction: $\mathcal O((m+n)r^2 + r^3)$
• Tangent projection and momentum update: $\mathcal O((m+n)r^2)$
• Additional backpropagation calls: $2$ per step (or $2(q + 1)$ for initialization with power iterations)

RLola requires storage of two additional $\mathcal O((m+n)r)$ matrices for momentum and second-moment accumulators, compared to standard LoRA's $\mathcal O(2mr)$. The overhead is modest for typical values of $r$.

5. Empirical Performance

Empirical results on LLM fine-tuning demonstrate that RLola matches or surpasses standard LoRA and other recent approaches in both convergence speed and final metric attainment.

For commonsense reasoning tasks (Llama 3.2 1b, $r=16$), RLola-LOI (with locally optimal initialization) consistently outperforms or matches the highest accuracy across datasets (e.g., BoolQ, PIQA, SIQA, HellaSwag, WinoG, ARC-E/C, OBQA), with averages such as $73.4\pm0.3$ (SGD, RLola-LOI) and $74.3\pm0.2$ (Adam, RLola-LOI). RLola also achieves faster convergence rates—Figure 1 (in the original manuscript) shows steeper declines in training loss relative to other LoRA-style optimizers (Bogachev et al., 16 Jul 2025).

On transfer scenarios (MetaMathQA $\rightarrow$ GSM8K), RLola achieves comparable or superior generalization: for example, RLola obtains $51.3\%$ GSM8K accuracy and $74.7\%$ Commonsense accuracy, compared to LoRA-GA at $50.0\%$ and $70.1\%$, respectively.

6. Significance and Implications

By formulating LoRA adapters as points on the fixed-rank manifold and leveraging intrinsic Riemannian optimization, RLola resolves critical issues of factorization ambiguity and suboptimal initialization. RLola's methodology can be applied to any LoRA-compatible architecture (including LLMs and diffusion models), and its numerically stable, geometry-respecting implementation is compatible with standard deep learning practice. This suggests that geometry-aware updates represent a robust avenue for future research in parameter-efficient model adaptation, particularly in regimes sensitive to initialization or prone to rank degeneracy (Bogachev et al., 16 Jul 2025).