RecAI: Leveraging Large Language Models for Next-Generation Recommender Systems

Abstract: This paper introduces RecAI, a practical toolkit designed to augment or even revolutionize recommender systems with the advanced capabilities of LLMs. RecAI provides a suite of tools, including Recommender AI Agent, Recommendation-oriented LLMs, Knowledge Plugin, RecExplainer, and Evaluator, to facilitate the integration of LLMs into recommender systems from multifaceted perspectives. The new generation of recommender systems, empowered by LLMs, are expected to be more versatile, explainable, conversational, and controllable, paving the way for more intelligent and user-centric recommendation experiences. We hope the open-source of RecAI can help accelerate evolution of new advanced recommender systems. The source code of RecAI is available at \url{https://github.com/microsoft/RecAI}.

• The paper introduces RecAI, a comprehensive LLM-powered toolkit that enhances recommender systems through five specialized pillars.
• It employs dynamic task planning, fine-tuning of language models, and a knowledge plugin to improve recommendation accuracy and explainability.
• Evaluations reveal improvements in metrics like recall, NDCG, and conversational engagement, underscoring its potential for next-generation recommender systems.

The paper "RecAI: Leveraging LLMs for Next-Generation Recommender Systems" (2403.06465) introduces RecAI, a toolkit designed to enhance recommender systems (RSs) with the capabilities of LLMs. The toolkit aims to create more versatile, explainable, conversational, and controllable recommendation experiences. RecAI comprises five foundational pillars, each addressing a specific application scenario.

These pillars are:

• Recommender AI Agent: An LLM-driven agent, where the LLM serves as the "brain" for user interaction, reasoning, planning, and task execution. Traditional recommender models act as "tools" to enhance the LLM's capabilities. The framework, named InteRecAgent, incorporates core tool types such as information query, item retrieval (distinguishing between hard and soft conditions), and item ranking. Key components include memory (Candidate Bus and User Profiles), task planning (a "plan-first" approach using dynamic demonstrations for in-context learning), and tool-learning (using smaller LLMs (SLMs) like a fine-tuned 7B-parameter Llama (RecLlama) to emulate GPT-4's instruction-following capabilities). Evaluations of InteRecAgent are detailed in \cite{huang2023recommender}.
• Recommendation-oriented LM: Focuses on fine-tuning LLMs for recommendation tasks, enabling them to process diverse textual inputs and return relevant items. Two types of models are introduced:
  • RecLM-emb: An embedding-based Recommendation LLM designed to retrieve items based on textual input. Detailed information can be found in \cite{lei2024aligning}.
  • RecLM-gen: A generative recommendation LM that decodes responses directly into natural language. A fine-tuned 7B Llama-2-chat model can surpass GPT-4 in item ranking tasks. Detailed information can be found in \cite{lu2024aligning}. RecLM-gen offers advantages such as improved accuracy through domain-specific fine-tuning, reduced system costs, and seamless, real-time user interactions.
• Knowledge Plugin: Supplements LLMs by dynamically incorporating domain-specific knowledge into prompts without altering the LLMs themselves, which is useful when fine-tuning is not feasible. This pillar employs a Domain-specific Knowledge Enhancement (DOKE) paradigm that involves extracting domain-relevant knowledge, selecting knowledge pertinent to the current sample, and formulating this knowledge into natural language. The plugin can boost LLMs' performance on item ranking by gathering item attributes and collaborative filtering signals. Additional details can be found in \cite{yao2023knowledge}.
• RecExplainer: Aims to elucidate the workings of embedding-based recommender models by interpreting the underlying hidden representations using LLMs. Approaches include behavior alignment (fine-tuning the LLM to predict items), intention alignment (LLM learns to process the recommender model's embeddings), and hybrid alignment. Tasks to fine-tune an LLM include predicting the next item, ranking items, classifying interests, detailing item characteristics, maintaining general intelligence through ShareGPT training, and reconstructing user history for intention alignment. More technical details and evaluations can be found in \cite{lei2023recexplainer}.
• Evaluator: Provides a tool for automatic evaluation across five key dimensions of LLM-augmented recommender systems, which include generative recommendation, embedding-based recommendation, conversation, explanation, and chit-chat. Generative recommendation evaluates the accuracy of item names generated by LLMs using fuzzy matching. Embedding-based recommendation supports evaluation of embedding-based matching models. Conversation assesses conversational recommendation efficacy through a GPT-4-powered user simulator. Explanation evaluates the informativeness, persuasiveness, and helpfulness of explanations using an LLM (e.g., GPT-4) as a judge. Chit-chat critiques the system's responses to non-recommendation dialogues. Evaluation metrics include NDCG and Recall for the first three dimensions, and pairwise comparisons (wins, losses, and ties) for Explanation and Chit-Chat, using an LLM as a judge.