LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation

Abstract: Low-Rank Adaptation (LoRA) has emerged as a popular parameter-efficient fine-tuning (PEFT) method for LLMs, yet it still incurs notable overhead and suffers from parameter interference in multi-task scenarios. We propose LoRA with Reduced Interference (LoRI), a simple yet effective approach that freezes the projection matrices $A$ as random projections and sparsifies the matrices $B$ using task-specific masks. This design substantially reduces the number of trainable parameters while maintaining strong task performance. Moreover, LoRI minimizes cross-task interference in adapter merging by leveraging the orthogonality between adapter subspaces, and supports continual learning by using sparsity to mitigate catastrophic forgetting. Extensive experiments across natural language understanding, mathematical reasoning, code generation, and safety alignment tasks demonstrate that LoRI outperforms full fine-tuning and existing PEFT methods, while using up to 95% fewer trainable parameters than LoRA. In multi-task experiments, LoRI enables effective adapter merging and continual learning with reduced cross-task interference. Code is available at: https://github.com/juzhengz/LoRI

• The paper introduces LoRI, a parameter-efficient fine-tuning method for multi-task LLMs that drastically reduces trainable parameters and cross-task interference compared to traditional LoRA.
• LoRI achieves parameter efficiency by freezing random projection matrices and using sparse, task-specific masks on counterpart matrices, leveraging orthogonality to minimize interference.
• Experiments show LoRI matches or surpasses standard PEFTs and full fine-tuning on NLU, code generation, and continual learning tasks while reducing parameters by up to 95%.

LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation

The research paper introduces LoRI (LoRA with Reduced Interference), a parameter-efficient fine-tuning (PEFT) methodology designed to optimize the performance of LLMs in multi-task settings. LoRI addresses the inefficiencies and high memory requirements inherent in traditional Low-Rank Adaptation (LoRA) methods by introducing specific adaptations that maintain model efficacy while significantly reducing the number of trainable parameters.

Core Concepts and Methodology

LoRI fundamentally alters the adaptation paradigm by freezing the low-rank projection matrices, $A$, as random projections, while modifying the counterpart matrices, $B$, using task-specific sparse masks. This approach drastically reduces the overhead associated with trainable parameters, with sparseness in $B$ being validated via magnitude-based selection across different layers and projections.

The sparsification mechanism in LoRI not only curtails parameters but also aids in preserving pre-trained knowledge during the adaptation process. By applying this new configuration, LoRI effectively exploits the orthogonality between adapter subspaces to minimize cross-task interference. This characteristic is particularly strategic in adapter merging scenarios, where LoRI leverages orthogonal subspaces to integrate multiple adapters without substantial performance degradation.

Experimental Evaluation and Results

Through extensive experimentation across varied tasks such as natural language understanding (NLU), mathematical reasoning, code generation, and safety alignment, LoRI demonstrated superior performance against conventional fine-tuning methods and advanced PEFT approaches like DoRA. Notably, LoRI reduced trainable parameters by as much as 95% compared to LoRA, without compromising overall model accuracy.

Key findings from the experiments include:

• NLU Tasks: LoRI achieved heightened accuracy across multiple datasets, consistently surpassing or equaling the performance of both full fine-tuning (FFT) and other PEFT approaches.
• Code Generation: On the challenging HumanEval benchmark, LoRI exceeded standard LoRA adaptations, highlighting its capacity to maintain high performance with fewer parameters.
• Continual Learning: LoRI significantly mitigated catastrophic forgetting in sequential learning scenarios, demonstrating robust maintainability of safety alignments during task-specific model updates.

Theoretical Implications and Future Directions

LoRI's design underscores the potential of sparsity as a strong regularizer, necessary for refining the adaptation process without extensively altering foundational model weights. This strategy aligns well with academic discourse emphasizing the latent task-specific knowledge housed within pre-trained LLMs, suggesting that fine-tuning acts as a catalyst to unlock this pre-existing potential rather than inventing new capabilities.

Looking forward, LoRI's applicability can extend beyond LLMs to other modalities such as vision and audio, possibly integrating structured sparsity techniques to enhance hardware compatibility and model pruning efficiency. The prospects of employing LoRI in multi-modal fusion tasks further reinforce its versatility and underline opportunities for future exploration in creating more flexible and resource-efficient AI systems.

Overall, this paper presents a compelling advancement in the field of parameter-efficient adaptation, providing a rigorous methodological and experimental foundation for future investigations into multi-task model optimization and consolidation.