Pre-train and Fine-tune: Recommenders as Large Models

Abstract: In reality, users have different interests in different periods, regions, scenes, etc. Such changes in interest are so drastic that they are difficult to be captured by recommenders. Existing multi-domain learning can alleviate this problem. However, the structure of the industrial recommendation system is complex, the amount of data is huge, and the training cost is extremely high, so it is difficult to modify the structure of the industrial recommender and re-train it. To fill this gap, we consider recommenders as large pre-trained models and fine-tune them. We first propose the theory of the information bottleneck for fine-tuning and present an explanation for the fine-tuning technique in recommenders. To tailor for recommendation, we design an information-aware adaptive kernel (IAK) technique to fine-tune the pre-trained recommender. Specifically, we define fine-tuning as two phases: knowledge compression and knowledge matching and let the training stage of IAK explicitly approximate these two phases. Our proposed approach designed from the essence of fine-tuning is well interpretable. Extensive online and offline experiments show the superiority of our proposed method. Besides, we also share unique and important lessons we learned when deploying the method in a large-scale online platform. We also present the potential issues of fine-tuning techniques in recommendation systems and the corresponding solutions. The recommender with IAK technique has been deployed on the homepage of a billion-scale online food platform for several months and has yielded considerable profits in our business.

• The paper introduces an Information-Aware Adaptive Kernel (IAK) to efficiently fine-tune recommender systems using information bottleneck theory.
• The paper demonstrates that compressing general knowledge and matching it with domain-specific data significantly improves CTR-AUC and CTCVR-AUC metrics.
• The paper provides practical deployment insights for scalable, adaptable recommenders in dynamic, multi-domain environments while addressing cold start challenges.

Overview of "Pre-train and Fine-tune: Recommenders as Large Models"

The paper by Jiang et al., explores the application of pre-training and fine-tuning methodologies, commonly used in NLP, to enhance recommender systems. The authors address the challenge of capturing dynamic user interests across various conditions such as periods, regions, and scenes, which are common in real-world recommendation systems but inadequately handled by existing multi-domain learning models.

The proposed solution involves treating recommendation systems as large pre-trained models that can be fine-tuned for specific downstream tasks. The paper introduces the Information-Aware Adaptive Kernel (IAK) technique, designed to fine-tune recommenders efficiently. The authors conceptualize fine-tuning in two phases: knowledge compression and knowledge matching. They leverage the information bottleneck theory to provide an interpretative framework for these phases, aiming to retain the essential information while discarding irrelevant data that does not contribute to the specific domain tasks.

Key Contributions and Techniques

1. Information Bottleneck for Fine-tuning: This theoretical construct is pivotal for fine-tuning in recommendation systems. It involves compressing the general knowledge from the pre-trained model and matching it with domain-specific knowledge, effectively adapting the model to diverse task-specific needs without extensive retraining.
2. Information-Aware Adaptive Kernel (IAK): IAK is a modular, encoder-decoder structure that adapts large pre-trained models to specific tasks with minimal adjustments. This feature is particularly advantageous for industrial-scale systems where retraining entire models is computationally expensive.
3. Deployment and Practical Insights: The paper shares practical insights from deploying this system on a large-scale online platform. It highlights strategies like parallel inference and dynamic batch-aware training to optimize the deployment of fine-tuned recommenders across multiple domains.
4. Addressing Potential Issues: The paper discusses potential issues, such as the pseudo cold start problem and user/item overlapping, which arise from the traditional fine-tuning processes.

Experimental Validation

The authors validate the superiority of their approach through extensive experiments using 11 real-world datasets. The results indicate that the enhanced models outperform state-of-the-art baselines in terms of CTR-AUC and CTCVR-AUC metrics. The effectiveness of IAK in various environments, including multi-scene, multi-region, and multi-period tasks, is demonstrated vigorously.

Practical and Theoretical Implications

The paper’s methodology shows potential to significantly reduce the computational cost associated with recommendations by allowing models to be fine-tuned for specific conditions without full-scale retraining. On a theoretical front, the application of information bottleneck theory offers a new lens through which the fine-tuning process can be understood and improved, contributing to the interpretability of recommendation systems.

Future Directions

Future research could explore optimizing this approach by integrating more advanced adaptive mechanisms into the IAK technique or refining the process of selecting which parts of general knowledge should be retained or discarded during fine-tuning. Additionally, addressing the hidden issues like pseudo cold start and overlapping could yield more robust models for dynamic environments.

Overall, this research presents a promising approach to improving the flexibility and efficiency of recommendation systems while maintaining high performance across various domains.