SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation

Abstract: In recent years, the development of diffusion models has led to significant progress in image and video generation tasks, with pre-trained models like the Stable Diffusion series playing a crucial role. Inspired by model pruning which lightens large pre-trained models by removing unimportant parameters, we propose a novel model fine-tuning method to make full use of these ineffective parameters and enable the pre-trained model with new task-specified capabilities. In this work, we first investigate the importance of parameters in pre-trained diffusion models, and discover that the smallest 10% to 20% of parameters by absolute values do not contribute to the generation process. Based on this observation, we propose a method termed SaRA that re-utilizes these temporarily ineffective parameters, equating to optimizing a sparse weight matrix to learn the task-specific knowledge. To mitigate overfitting, we propose a nuclear-norm-based low-rank sparse training scheme for efficient fine-tuning. Furthermore, we design a new progressive parameter adjustment strategy to make full use of the re-trained/finetuned parameters. Finally, we propose a novel unstructural backpropagation strategy, which significantly reduces memory costs during fine-tuning. Our method enhances the generative capabilities of pre-trained models in downstream applications and outperforms traditional fine-tuning methods like LoRA in maintaining model's generalization ability. We validate our approach through fine-tuning experiments on SD models, demonstrating significant improvements. SaRA also offers a practical advantage that requires only a single line of code modification for efficient implementation and is seamlessly compatible with existing methods.

• The paper introduces SaRA, which efficiently fine-tunes pre-trained diffusion models by leveraging the less impactful 10-20% of parameters.
• It employs sparse weight matrix optimization and nuclear-norm-based low-rank training to enhance efficiency while mitigating overfitting.
• The method requires minimal code changes, reduces memory needs, and outperforms traditional techniques like LoRA in task-specific adaptation.

The paper "SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation" presents a novel approach to enhancing the efficiency of fine-tuning pre-trained diffusion models, which are widely used in generative tasks like image and video generation. Despite the increasing capabilities of such models, their size and complexity often pose challenges for downstream task adaptation.

Key Insights and Contributions:

1. Parameter Ineffectiveness Exploration:
   • The authors investigate the contribution of parameters in pre-trained diffusion models, finding that the smallest 10% to 20% by absolute value do not significantly impact the generation process. This insight drives the development of their fine-tuning strategy.
2. Sparse Weight Matrix Optimization (SaRA):
   • The core of their method, SaRA, involves re-utilizing these less important parameters by optimizing a sparse weight matrix. This allows the model to adapt to new tasks without the need for large-scale parameter adjustments.
3. Nuclear-Norm-Based Low-Rank Sparse Training:
   • To prevent overfitting during fine-tuning, the authors introduce a nuclear-norm-based low-rank training scheme. This approach helps maintain the model's efficiency by focusing on important parameter adjustments.
4. Progressive Parameter Adjustment Strategy:
   • A novel progressive strategy is employed to maximize the utilization of retrained or finetuned parameters, enhancing the model's adaptation to task-specific requirements.
5. Unstructural Backpropagation Strategy:
   • This strategy substantially reduces memory requirements during fine-tuning, making the process more feasible on resource-constrained systems.
6. Comparison with Traditional Methods:
   • SaRA is shown to outperform conventional methods such as LoRA in maintaining the generalized capability of the model while enhancing task-specific performance.
7. Practical Implementation:
   • A key practical advantage of SaRA is its compatibility with existing systems, requiring minimal modification—a single line of code—to implement this efficient fine-tuning strategy.

Validation and Results:

The authors validate their approach through experiments on Stable Diffusion (SD) models. The results demonstrate significant improvements in the generative capabilities of the models when applied to specific downstream tasks, highlighting SaRA's effectiveness over traditional fine-tuning methods.

In summary, SaRA offers a resource-efficient, practically implementable method for fine-tuning diffusion models, enhancing their applicability across various specialized generative tasks while maintaining generalization abilities.