Efficient and Effective Adaptation of Multimodal Foundation Models in Sequential Recommendation

Abstract: Multimodal foundation models (MFMs) have revolutionized sequential recommender systems through advanced representation learning. While Parameter-efficient Fine-tuning (PEFT) is commonly used to adapt these models, studies often prioritize parameter efficiency, neglecting GPU memory and training speed. To address this, we introduced the IISAN framework, significantly enhancing efficiency. However, IISAN was limited to symmetrical MFMs and identical text and image encoders, preventing the use of state-of-the-art LLMs. To overcome this, we developed IISAN-Versa, a versatile plug-and-play architecture compatible with both symmetrical and asymmetrical MFMs. IISAN-Versa employs a Decoupled PEFT structure and utilizes both intra- and inter-modal adaptation. It effectively handles asymmetry through a simple yet effective combination of group layer-dropping and dimension transformation alignment. Our research demonstrates that IISAN-Versa effectively adapts large text encoders, and we further identify a scaling effect where larger encoders generally perform better. IISAN-Versa also demonstrates strong versatility in our defined multimodal scenarios, which include raw titles and captions generated from images and videos. Additionally, IISAN-Versa achieved state-of-the-art performance on the Microlens public benchmark. We will release our code and datasets to support future research.

• The paper introduces the IISAN-Versa framework that reduces GPU memory consumption up to 15 times and cuts training time by up to 20 times compared to traditional methods.
• The paper presents a decoupled PEFT structure, DPEFT, which offers granular control for adapting both symmetrical and asymmetrical multimodal models in sequential recommendation tasks.
• The study demonstrates that scaling larger text encoders within IISAN-Versa significantly boosts recommendation performance, achieving state-of-the-art results on the Microlens benchmark.

Efficient and Effective Adaptation of Multimodal Foundation Models in Sequential Recommendation

This paper addresses two significant challenges in the domain of recommendation systems: the immense computational demands for fine-tuning large-scale multimodal foundation models (MFMs) and the inherent limitations of existing parameter-efficient fine-tuning (PEFT) techniques. The authors propose a novel framework, IISAN-Versa, which aims to efficiently and effectively adapt MFMs in sequential recommendation tasks by leveraging advanced architectural strategies.

Recent breakthroughs in recommendation algorithms have demonstrated the potential of LLMs and advanced vision encoders, like GPT-4, DALL-E, and ViT, in achieving superior performance. In this context, the IISAN-Versa framework introduces a decoupled PEFT structure known as DPEFT. This structure enables IISAN-Versa to accommodate both symmetrical and asymmetrical MFMs. By adopting both intra- and inter-modal adaptation strategies, the research effectively reduces computational resource requirements, such as GPU memory usage and training time, while enhancing model performance.

Key Findings:

• Decoupled PEFT Framework: The authors demonstrate that their decoupled approach enables substantial efficiency improvements compared to traditional full fine-tuning and embedded PEFT methods like Adapter and LoRA. The decoupling allows for more granular control over the adaptation process, resulting in reduced GPU memory consumption—up to 15 times compared to FFT—and accelerated training time by a factor of up to 20.
• Scaling Laws: A pivotal discovery of the research is the scaling effect observed when larger text encoders are integrated within the IISAN-Versa framework. Larger models generally lead to improved performance, highlighting the advantageous potential of current large pre-trained models for multimodal recommendation tasks.
• Versatility: The IISAN-Versa framework demonstrates robust adaptability across diverse multimodal scenarios. It achieved state-of-the-art results on the Microlens benchmark, underscoring its capacity to effectively manage and encode various data types, including text, image, and video.

The paper also makes several noteworthy contributions to ongoing research in multimodal learning and sequential recommendation. The introduction of IISAN-Versa serves as a catalyst for further exploration of efficient model adaptation techniques that strive to balance performance and resource consumption. Specifically, it encourages future research to investigate more fine-grained inter-modal merging techniques and explore the asymmetrical adaptation potential of other large-scale pre-trained transformers, potentially extending the framework’s applicability beyond current benchmarks. Speculatively, IISAN-Versa could be adapted for various tasks like video retrieval and question answering, leveraging its inherent efficiency and effectiveness in recommendation settings.

Although IISAN-Versa addresses many practical challenges associated with the adaptation of MFMs in recommendation systems, the research also acknowledges certain limitations, particularly in making optimal use of vision transformers. As larger and more robust vision encoders become available, their integration could unlock further gains in recommendation performance.

In conclusion, the work presents a compelling advancement in the field of sequential recommendation by harmonizing sophisticated adaptation techniques with practical computational efficiency. Such progress not only broadens the application horizon of sophisticated MFMs but also reflects a strategic shift toward more resource-efficient AI solutions, positioning IISAN-Versa as a noteworthy contribution to contemporary recommendation systems research.