Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning

Abstract: Low-Rank Adaptation (LoRA) is the prevailing approach for efficient LLM fine-tuning. Building on this paradigm, recent studies have proposed alternative initialization strategies and architectural modifications, reporting substantial improvements over vanilla LoRA. However, these gains are often demonstrated under fixed or narrowly tuned hyperparameter settings, despite the known sensitivity of neural networks to training configurations. In this work, we systematically re-evaluate four representative LoRA variants alongside vanilla LoRA through extensive hyperparameter searches. Across mathematical and code generation tasks on diverse model scales, we find that different LoRA methods favor distinct learning rate ranges. Crucially, once learning rates are properly tuned, all methods achieve similar peak performance (within 1-2%), with only subtle rank-dependent behaviors. These results suggest that vanilla LoRA remains a competitive baseline and that improvements reported under single training configuration may not reflect consistent methodological advantages. Finally, a second-order analysis attributes the differing optimal learning rate ranges to variations in the largest Hessian eigenvalue, aligning with classical learning theories.

• The paper shows that optimal learning rate tuning is the key factor in achieving near-identical performance among LoRA variants.
• Comprehensive hyperparameter sweeps and Hessian analysis reveal that performance differences typically remain within 1–2%.
• The study challenges previous methodologies, emphasizing the need for systematic tuning in LLM parameter-efficient fine-tuning.

Re-assessing LoRA Variants: The Centrality of Learning Rate in LLM PEFT

Introduction

This work scrutinizes a central assumption in the burgeoning literature on Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning (PEFT) of LLMs: that proposed LoRA variants yield robust performance improvements over vanilla LoRA. The authors demonstrate, via comprehensive hyperparameter sweeps and Hessian-based analyses, that performance enhancements attributed to recent LoRA modifications are, in practice, largely contingent on the proper tuning of the learning rate (LR). When LoRA and its variants are each tuned to their optimal learning rates, their peak performances on standard mathematical reasoning and code generation tasks are nearly indistinguishable, typically differing by only 1–2%. The empirical and theoretical findings challenge the methodological rigor of prior comparative studies and have significant implications for both scientific practices and the PEFT research agenda.

Comparative Evaluation of LoRA and Its Variants

The study contrasts vanilla LoRA with four representative improvements: PiSSA, MiLoRA, Init[AB], and DoRA. These variants span SVD-based initialization schemes (PiSSA, MiLoRA), alternative residual constructions (Init[AB]), and minor architectural modifications (DoRA). The core methodology involves unified experimental protocols, grid search over learning rates (spanning three orders of magnitude), batch size, and adapter rank, evaluated on multiple LLMs (Qwen3-0.6B, Gemma-3-1B, Llama-2-7B) and canonical tasks (MetaMathQA, GSM8K, MATH for math reasoning; CodeFeedback, HumanEval, MBPP for code generation).

Key empirical results:

• With model- and method-specific LR tuning, all approaches achieve nearly identical peak test accuracies (typical differences 0.4–1.7% depending on task and model size), nullifying previously reported advantages of advanced LoRA variants.
• The optimal LR is highly method-dependent and can vary by an order of magnitude across different initialization or adapter designs.
• Subtle rank-dependent effects exist (e.g., PiSSA overtakes LoRA only at higher ranks; MiLoRA occasionally outperforms at low ranks), but no variant displays consistent superiority across settings.
• The dominant factor for fine-tuning success is not batch size or adapter configuration, but precise LR selection. Fixing batch size and tuning LR achieves markedly superior results compared to the converse.

Hessian-Based Analysis of Learning Rate Dynamics

To explain these findings, the authors relate optimal LR to the curvature of the loss landscape, quantified by the maximum eigenvalue $\lambda_{\max}$ of the Hessian with respect to trainable LoRA parameters. Following classical theory, the optimal LR should scale inversely with $\lambda_{\max}$: $n^* \propto 1/\lambda_{\max}$. Systematic computation of Hessian spectra for different methods demonstrates:

• SVD-informed initializations (e.g., PiSSA) induce substantially larger $\lambda_{\max}$ at initialization, necessitating smaller LRs for stable convergence.
• Other variants (MiLoRA, Init[AB]) can yield slightly larger Hessian curvature (by factors of 1.8–3.2), reflected in their lower optimal LRs, but the relationship with vanilla LoRA remains monotonic and predictable.
• Despite these shifts in loss landscape geometry, the ultimate optimization endpoint and generalization performance remain nearly invariant given proper LR calibration.

Implications for Methodology and Advancement in PEFT

The paper identifies pervasive shortcomings in the empirical protocol of prior LoRA-based PEFT research. A meta-analysis of 52 recent works shows that less than 30% report non-default LR tuning for the vanilla LoRA baseline, and almost none systematically tune LR, batch size, and rank jointly. This practice renders claims of consistent improvement by advanced variants questionable, as their performance frequently degrades or aligns with vanilla LoRA under rigorous tuning.

In terms of practical implications and future directions:

• Vanilla LoRA, when combined with systematic LR tuning, remains a robust and competitive baseline for LLM PEFT.
• Minor architectural or initialization modifications, absent improvements in the intrinsic efficacy given fair tuning, yield at best marginal and context-specific gains.
• Future developments should emphasize new adaptation paradigms beyond weight-based low-rank factorization, including representation-level adaptation or nonlinear transformation of activation subspaces.
• Community benchmarks should enforce method-specific hyperparameter searches and utilize Hessian-based or loss curvature diagnostics to guide optimization protocols.

Conclusion

The work provides compelling evidence that previously reported gains from advanced LoRA variants are predominantly attributable to differences in learning rate regime rather than substantive methodological superiority. It establishes that, under systematic tuning, vanilla LoRA "suffices" for effective LLM adaptation, with advanced variants only conferring rare, marginal, and non-universal advantages. The study sets a new standard for methodological rigor in benchmarking PEFT approaches and implies that near-term progress in PEFT may require fundamentally new adaptation principles rather than further incremental modifications of the LoRA paradigm.

Reference:

"Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning" (2602.04998)