LEADRE: Multi-Faceted Knowledge Enhanced LLM Empowered Display Advertisement Recommender System

Abstract: Display advertising provides significant value to advertisers, publishers, and users. Traditional display advertising systems utilize a multi-stage architecture consisting of retrieval, coarse ranking, and final ranking. However, conventional retrieval methods rely on ID-based learning to rank mechanisms and fail to adequately utilize the content information of ads, which hampers their ability to provide diverse recommendation lists. To address this limitation, we propose leveraging the extensive world knowledge of LLMs. However, three key challenges arise when attempting to maximize the effectiveness of LLMs: "How to capture user interests", "How to bridge the knowledge gap between LLMs and advertising system", and "How to efficiently deploy LLMs". To overcome these challenges, we introduce a novel LLM-based framework called LLM Empowered Display ADvertisement REcommender system (LEADRE). LEADRE consists of three core modules: (1) The Intent-Aware Prompt Engineering introduces multi-faceted knowledge and designs intent-aware <Prompt, Response> pairs that fine-tune LLMs to generate ads tailored to users' personal interests. (2) The Advertising-Specific Knowledge Alignment incorporates auxiliary fine-tuning tasks and Direct Preference Optimization (DPO) to align LLMs with ad semantic and business value. (3) The Efficient System Deployment deploys LEADRE in an online environment by integrating both latency-tolerant and latency-sensitive service. Extensive offline experiments demonstrate the effectiveness of LEADRE and validate the contributions of individual modules. Online A/B test shows that LEADRE leads to a 1.57% and 1.17% GMV lift for serviced users on WeChat Channels and Moments separately. LEADRE has been deployed on both platforms, serving tens of billions of requests each day.

• The paper presents a novel framework (LEADRE) that integrates intent-aware prompt engineering, advertising-specific knowledge alignment, and efficient LLM deployment to enhance ad recommendations.
• It reports significant performance improvements with enhanced retrieval accuracy and notable GMV uplift in both offline experiments and online A/B tests on WeChat Channels and Moments.
• The study emphasizes practical LLM integration in industrial-scale advertising systems, showcasing fine-tuning strategies and latency-aware deployment for scalable, personalized ad delivery.

LEADRE: Multi-Faceted Knowledge Enhanced LLM Empowered Display Advertisement Recommender System

Introduction

LEADRE introduces a framework to integrate LLMs within display advertising systems to overcome limitations of traditional ID-based retrieval mechanisms, leveraging LLMs' knowledge capabilities. This framework addresses three pivotal challenges: capturing implicit user interests, bridging LLMs and advertising system knowledge gaps, and efficient LLM deployment in high-demand environments. LEADRE consists of three modules: Intent-Aware Prompt Engineering, Advertising-Specific Knowledge Alignment, and Efficient System Deployment.

Framework Overview

LEADRE utilizes LLMs in the display advertising recommendation process through a structured approach:

1. Intent-Aware Prompt Engineering: This module designs intent-aware <Prompt, Response> pairs containing user behavior sequences and ad descriptions, serving as fine-tuning corpus to adjust LLMs for generating personalized ads. User interests are modeled by combining long-term interests (derived from profiles and historical data) and short-term interests (from recent interactions).

   Figure 1: Overall Framework of Ads indexing in LEADRE.

2. Advertising-Specific Knowledge Alignment: Semantic and business value alignments are conducted through auxiliary tasks and Direct Preference Optimization (DPO) to ensure LLMs generate ads aligned with semantic aptness and high business value.

   Figure 2: Overall Framework of Ads Constrained Generation Module and LLM Fine-tuning Module.

3. Efficient System Deployment: Integrates latency-tolerant and latency-sensitive services to facilitate responsive performance. TensorRT-LLM acceleration further enhances scalability and computational efficiency.

   Figure 3: Framework of Latency-Aware Model Deployment.

Experiments and Results

Offline Experiments

Extensive offline experiments validated LEADRE's modules. The evaluation metrics were Hit Ratio (HR@K) and Normalized Discounted Cumulative Gain (NDCG@K).

• Component Effectiveness: Validation of each prompt component showed improvements with content domain behaviors and user interest summary in the retrieval process.
• Tuning Strategies: Sequential tuning involving explicit and implicit alignment tasks showed significant performance improvements in generating accurate retrievals.
• Semantic ID Effectiveness: Comparison of ads' S-IDs generated by varying embedding models showed Hunyuan embedding models contributing effectively to retrieval system rankings.

Online A/B Tests

20% traffic A/B tests on Tencent's WeChat Channels and Moments confirmed LEADRE's impact with a recorded 1.57% improvement in GMV for WeChat Channels and 1.17% for Moments. The integration into the ranking phase yielded additional GMV improvement, showcasing LEADRE's capability to deliver tailored advertising content at a large scale.

Conclusion

LEADRE represents a structured approach to leverage LLMs in industrial-scale display advertising systems, effectively translating user and ad data into high-value, personalized ad recommendations. Future developments include enhancing next-N generation capabilities and designing improved quantization techniques for more effective semantic IDs.

In summary, LEADRE demonstrates how specific prompt engineering, semantic alignment, and efficient deployment strategies can effectively integrate LLMs into practical advertising systems, paving the way for smarter content delivery strategies in the advertising domain.