Evaluating Parameter Efficient Methods for RLVR

Abstract: We systematically evaluate Parameter-Efficient Fine-Tuning (PEFT) methods under the paradigm of Reinforcement Learning with Verifiable Rewards (RLVR). RLVR incentivizes LLMs to enhance their reasoning capabilities through verifiable feedback; however, while methods like LoRA are commonly used, the optimal PEFT architecture for RLVR remains unidentified. In this work, we conduct the first comprehensive evaluation of over 12 PEFT methodologies across the DeepSeek-R1-Distill families on mathematical reasoning benchmarks. Our empirical results challenge the default adoption of standard LoRA with three main findings. First, we demonstrate that structural variants, such as DoRA, AdaLoRA, and MiSS, consistently outperform LoRA. Second, we uncover a spectral collapse phenomenon in SVD-informed initialization strategies (\textit{e.g.,} PiSSA, MiLoRA), attributing their failure to a fundamental misalignment between principal-component updates and RL optimization. Furthermore, our ablations reveal that extreme parameter reduction (\textit{e.g.,} VeRA, Rank-1) severely bottlenecks reasoning capacity. We further conduct ablation studies and scaling experiments to validate our findings. This work provides a definitive guide for advocating for more exploration for parameter-efficient RL methods.

• The paper demonstrates that structured PEFT variants, particularly DoRA and MiSS, consistently outperform standard LoRA with improvements up to 46.6% accuracy.
• The paper finds that extreme parameter reduction via methods like VeRA and IA3 leads to severe performance drops, establishing an expressivity floor in RLVR.
• The paper reveals that SVD-based initialization causes spectral collapse in RLVR, underscoring the need for alternative strategies in adapter tuning.

Parameter-Efficient Fine-Tuning Methods in RL with Verifiable Rewards: A Systematic Evaluation

Introduction

This work offers a comprehensive empirical evaluation of Parameter-Efficient Fine-Tuning (PEFT) methods within the Reinforcement Learning with Verifiable Rewards (RLVR) framework for LLMs. RLVR leverages deterministic verifiers providing binary feedback to accelerate the evolution of reasoning capabilities in generative models, especially on mathematical reasoning tasks. While the field has generally defaulted to the standard Low-Rank Adaptation (LoRA) architecture for RL fine-tuning, this paper interrogates whether architectural or optimization variants offer improved efficiency-versus-accuracy tradeoffs in the RLVR setting.

Methodology

A benchmark comprising 12+ PEFT variants was deployed across DeepSeek-R1-Distill model families. The evaluation focused on high-stakes, process-verified mathematical reasoning benchmarks (e.g., MATH500, AIME24/25, AMC, HMMT, Minerva) under the RLVR paradigm using DAPO and GRPO family RL algorithms. The methods evaluated span:

• Structural variants modifying adaptation parameterization: DoRA (decouples direction and magnitude), AdaLoRA (adaptive budget), MiSS (sharded structure).
• Initialization strategies: PiSSA and MiLoRA (singular value decomposition-based), LoRA+ (learning rate re-weighting), rsLoRA.
• Efficiency-focused methods: LoRA-FA (partially frozen adapters), VeRA (vector-based adaptation).
• Non-standard paradigms: IA3 (vector scale), LayerNorm Tuning.

Models were evaluated for both average accuracy and pass@$k$ metrics, ensuring the reliability and diversity of reasoning traces in extensive ablation and scaling studies (1.5B/7B parameters).

Main Empirical Findings

Superiority of Structured PEFT Methods

Structural variants, specifically DoRA, AdaLoRA, and MiSS, consistently surpassed the performance of standard LoRA. Notably, DoRA exceeded full-model RL fine-tuning (e.g., 46.6% average accuracy vs. 44.9% for full and 42.5% for baseline LoRA). The underlying success factors include a loosened constraint that mitigates the rigidity of vanilla low-rank updates, and, in DoRA’s case, magnitude-direction decoupling facilitating larger, more adaptive policy shifts.

Expressivity Floor in Extreme Parameter Reduction

There exists a strict lower bound on the trainable parameter capacity necessary for RLVR—beyond which reasoning abilities collapse. While moderate adapter freezing (e.g., LoRA-FA) maintains competitive results, extreme parameter compression (VeRA, Rank-1, LayerNorm tuning, IA3) leads to substantial performance drops (e.g., VeRA: 40.7%, IA3: 22.3%), supporting the assertion that RLVR, unlike SFT, cannot operate effectively in severely bottlenecked adaptation regimes.

SVD-Informed Initialization: Spectral Collapse

SVD-based initialization strategies—often successful in SFT—yield catastrophic performance in RLVR contexts. PiSSA and MiLoRA, which initialize adapters via principal or minor singular components of pretrained weights, suffer spectral collapse and optimization misalignment. PiSSA’s focus on principal subspace updates directly contradicts RLVR’s off-principal weight update dynamics (as shown in [Zhu et al., 2025] and mirrored by this paper’s spectral analyses), while MiLoRA lacks sufficient initialization magnitude to constrain learning away from principal gradients, both culminating in near-random accuracy.

Algorithm and Hyperparameter Robustness

The dominance of structured PEFT variants (particularly DoRA) is largely algorithm-invariant, holding across RLVR optimizer choices (GRPO, DAPO, Dr. GRPO) and robust under extensive ablations in batch sizes, learning rates, and adapter ranks. Notably, higher LoRA ranks (16/32) consistently yield improvements over extreme adapter minimization (rank 1).

Consistency at Scale

All key results generalize robustly when moving from 1.5B to 7B parameter models. The performance hierarchy (DoRA > LoRA ≈ LoRA+) remains stable, with accurate outperformance maintained even as base model scaling increases.

Implications

Practical Recommendations

• Standard LoRA should no longer be the de facto approach for RLVR. The evidence strongly supports rapid migration to structurally enhanced adapters (e.g., DoRA or MiSS) for RL-based adaptation of LLMs.
• Extreme parameter bottlenecking is inadvisable in RLVR settings, as it curtails the adaptive plasticity required for reward-driven reasoning shift.
• SVD-based initializations are contraindicated for RLVR. Adapter initialization and spectral control methods must acknowledge and avoid conflicts with the optimization landscape of RL with sparse, verifiable reward signals.
• Algorithm engineering (e.g., RLVR variant selection) is a secondary concern to selecting optimal PEFT architectures.

Theoretical Implications and Open Questions

The findings suggest that the inductive biases appropriate for PEFT adapters in SFT do not trivially transfer to RL, particularly in the presence of sparse gradients and policy-induced optimization geometry. A mechanistic theory explaining why magnitude-direction decoupling aligns with off-principal update requirements remains undeveloped and constitutes a promising avenue for future research.

Future Directions

Expansion to multimodal/multi-turn RLVR, further scaling studies, and improved training infrastructure (e.g., high-performance RL frameworks beyond TRL) are highlighted as requisite to cement PEFT applicability in RL. Critically, establishing mathematical rigor for the observed empirical phenomena—especially the expressivity floor and spectral collapse dynamics—will bridge theory and practice.

Conclusion

This work establishes that standard LoRA is suboptimal for RLVR fine-tuning tasks. Structural adapter variants, particularly DoRA and MiSS, achieve higher reasoning accuracy—including surpassing full-parameter baselines—by better aligning with RL-driven optimization demands. Severe parameter compression or spectral principal-component initialization is strongly contraindicated. These insights provide a concrete roadmap for efficient yet high-performance RL adaptation of LLMs, closing the gap between engineering practice and the inherent optimization dynamics of RLVR.

Citation: Yin et al., "Evaluating Parameter Efficient Methods for RLVR" (2512.23165)