Dynamic Low-Rank Sparse Adaptation for Large Language Models

Abstract: Despite the efficacy of network sparsity in alleviating the deployment strain of LLMs, it endures significant performance degradation. Applying Low-Rank Adaptation (LoRA) to fine-tune the sparse LLMs offers an intuitive approach to counter this predicament, while it holds shortcomings include: 1) The inability to integrate LoRA weights into sparse LLMs post-training, and 2) Insufficient performance recovery at high sparsity ratios. In this paper, we introduce dynamic Low-rank Sparse Adaptation (LoSA), a novel method that seamlessly integrates low-rank adaptation into LLM sparsity within a unified framework, thereby enhancing the performance of sparse LLMs without increasing the inference latency. In particular, LoSA dynamically sparsifies the LoRA outcomes based on the corresponding sparse weights during fine-tuning, thus guaranteeing that the LoRA module can be integrated into the sparse LLMs post-training. Besides, LoSA leverages Representation Mutual Information (RMI) as an indicator to determine the importance of layers, thereby efficiently determining the layer-wise sparsity rates during fine-tuning. Predicated on this, LoSA adjusts the rank of the LoRA module based on the variability in layer-wise reconstruction errors, allocating an appropriate fine-tuning for each layer to reduce the output discrepancies between dense and sparse LLMs. Extensive experiments tell that LoSA can efficiently boost the efficacy of sparse LLMs within a few hours, without introducing any additional inferential burden. For example, LoSA reduced the perplexity of sparse LLaMA-2-7B by 68.73 and increased zero-shot accuracy by 16.32$\%$, achieving a 2.60$\times$ speedup on CPU and 2.23$\times$ speedup on GPU, requiring only 45 minutes of fine-tuning on a single NVIDIA A100 80GB GPU. Code is available at https://github.com/wzhuang-xmu/LoSA.

• The paper introduces a framework that integrates low-rank adaptation and dynamic sparsity, significantly reducing inference latency.
• It employs Representation Mutual Information to set layer-wise sparsity rates and dynamically allocate low-rank tuning based on reconstruction errors.
• Experimental results demonstrate improved perplexity and zero-shot accuracy on models such as LLaMA-2-7B and OPT with minimal overhead.

Dynamic Low-Rank Sparse Adaptation for LLMs

This paper presents Dynamic Low-Rank Sparse Adaptation (LoSA), an innovative framework designed to enhance the performance of sparse LLMs by integrating low-rank adaptation within a unified structure. This approach aims to address the challenges of deploying sparse models without increasing inference latency.

Introduction

LLMs, though effective in diverse natural language processing tasks, are computationally demanding during deployment due to their large size. Existing solutions include model compression techniques such as sparsity, quantization, and knowledge distillation that help reduce model size while preserving efficacy. However, sparsity methods face performance degradation at high sparsity levels. Parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaptation (LoRA) offer a pathway to fine-tuning sparse models efficiently, albeit with integration issues.

LoSA proposes a framework that incorporates both sparsity and low-rank adaptation, ensuring dynamic layer sparsity rates based on Representation Mutual Information (RMI) and adjusting the rank of LoRA modules in response to changes in layer-wise reconstruction errors. This not only maintains performance at high sparsity ratios but also achieves significant reductions in inference latency.

Figure 1: Comparing traditional sparse LLM with LoRA and the proposed LoSA method.

Methodology

Problem Formulation

LoSA views the integration of sparsity and low-rank adaptation as a layer-wise reconstruction problem. The objective is to minimize discrepancies in the output between dense and sparse layers while dynamically adjusting low-rank adaptations. This is achieved through a unified optimization problem that determines sparsity masks, layer-wise sparsity rates, and rank allocations for low-rank adaptations.

Layer-wise Sparsity Rate Determination

The authors propose using RMI to determine the importance of each layer dynamically. This approach hinges on the Information Bottleneck principle, which balances mutual information between input/output representations. By measuring the mutual information across layers, LoSA effectively sets layer-wise sparsity rates, replacing computational bottlenecks with efficient importance metrics.

Sparsity-Aware Rank Allocation

Traditionally, LoRA uses uniform rank assignments, ignoring variability in reconstruction errors arising from sparsity. LoSA, however, utilizes these errors to allocate tuning budgets dynamically, providing more resources to layers with higher discrepancies.

Dynamic Sparsity and Adaptation

LoSA applies a progressive sparsity schedule that increases the sparsity rate over iterations while concurrently adjusting low-rank adaptation ranks. This method leverages the cubic sparsity schedule to ensure efficient integration of sparsity with adaptation, ultimately merging low-rank modules with sparse weights post-training.

Experiments

Extensive experiments demonstrated LoSA's capabilities across various models, including LLaMA and OPT, showcasing improvements in perplexity and zero-shot accuracy with minimal computational overhead. For instance, LoSA reduced the perplexity of a sparse LLaMA-2-7B by 68.73 and improved zero-shot accuracy by 16.32%.

Conclusion

LoSA represents a significant advancement in the joint optimization of sparse LLMs with low-rank adaptations. The method's dynamic nature ensures that inference latency is minimized while maintaining robust performance metrics. This framework sets a foundation for future developments in efficient model compression and adaptation strategies in AI research.