Scaling Laws for Online Advertisement Retrieval

Abstract: The scaling law is a notable property of neural network models and has significantly propelled the development of LLMs. Scaling laws hold great promise in guiding model design and resource allocation. Recent research increasingly shows that scaling laws are not limited to NLP tasks or Transformer architectures; they also apply to domains such as recommendation. However, there is still a lack of literature on scaling law research in online advertisement retrieval systems. This may be because 1) identifying the scaling law for resource cost and online revenue is often expensive in both time and training resources for large-scale industrial applications, and 2) varying settings for different systems prevent the scaling law from being applied across various scenarios. To address these issues, we propose a lightweight paradigm to identify the scaling law of online revenue and machine cost for a certain online advertisement retrieval scenario with a low experimental cost. Specifically, we focus on a sole factor (FLOPs) and propose an offline metric named R/R* that exhibits a high linear correlation with online revenue for retrieval models. We estimate the machine cost offline via a simulation algorithm. Thus, we can transform most online experiments into low-cost offline experiments. We conduct comprehensive experiments to verify the effectiveness of our proposed metric R/R* and to identify the scaling law in the online advertisement retrieval system of Kuaishou. With the scaling law, we demonstrate practical applications for ROI-constrained model designing and multi-scenario resource allocation in Kuaishou advertising system. To the best of our knowledge, this is the first work to study the scaling laws for online advertisement retrieval of real-world systems, showing great potential for scaling law in advertising system optimization.

• The paper introduces the novel R/R* offline metric that aligns traditional evaluations with online revenue outcomes.
• It empirically validates the Broken Neural Scaling Law, revealing a predictable non-linear relation between FLOPs and model performance.
• It outlines efficient cost estimation paradigms, achieving up to 2.8% revenue improvements in real-world advertising systems.

An Examination of Scaling Laws in Online Advertisement Retrieval

The paper "Scaling Laws for Online Advertisement Retrieval" by Yunli Wang et al. offers a comprehensive study on the applicability of neural scaling laws within the field of online advertisement retrieval. The authors present a novel approach to identifying scaling behaviors in large-scale advertising systems, leveraging a primarily offline methodology to minimize resource consumption. This work specifically focuses on machine learning models used in the retrieval stages of advertisement selection, aiming to draw parallels between computational resource allocation and consequent improvements in business revenue.

Key Contributions

The paper contributes in multiple facets to the field of online advertising systems:

1. Introduction of a New Offline Metric: The authors propose a novel offline metric, $R/R^*$, which correlates strongly with online revenue. The metric, designed to address the gap between traditional offline evaluation metrics and actual revenue goals, considers the commercial value—specifically the eCPM—of each ad, thus reflecting the primary revenue maximization target.
2. Establishment of a Scaling Law: A central finding of the paper is the validation of the Broken Neural Scaling Law (BNSL) in the context of retrieval models within advertising systems. The study empirically proves that there exists a non-linear but predictable trend between computational expenses (expressed in FLOPs) and model performance as quantified through the $R/R^*$ metric.
3. Efficient Machine Cost Estimation: The paper outlines two effective paradigms for the estimation of machine costs—one based on expert knowledge and another through offline simulation testing. The latter, which implements a standard algorithm, is demonstrated to provide an efficient and reproducible method for resource allocation without requiring actual model deployment.
4. Applications in Real-World Scenarios: Utilizing their findings, the authors demonstrate two practical applications: ROI-constrained model design and multi-scenario resource allocation. By solving optimization problems framed by these scaling laws, they achieve revenue improvements of 0.85% and 2.8%, respectively, exemplifying the tangible benefits of their approach for commercial systems.

Implications and Future Directions

The implications of this study are twofold. Theoretically, it situates neural scaling laws within the context of advertisement retrieval, thereby broadening the applicability of such theoretical constructs beyond traditional NLP and CV domains. Practically, it sets a foundational methodology for optimizing large-scale advertising systems under resource constraints, providing a blueprint for practitioners in the field.

Moving forward, opportunities arise in the enhancement of foundational models employed within recommendation and advertising ecosystems. While the study validated scaling laws within MLP-based architectures, exploring more universally effective models, potentially leveraging architectures akin to transformers in NLP, could unveil further insights into scalability and resource efficiency. Additionally, developing methodologies to extract multidimensional scaling laws that integrate model parameters, data volume, and computational costs might yield even more precise scalability insights essential for dynamic and complex industrial settings.

Developing a robust framework for scaling laws in advertising systems will likely galvanize subsequent innovations, improving both the efficiency of computational resource use and the financial yield from advertising platforms. The work of Wang et al. catalyzes this exploration by providing a clear methodological pathway and demonstrating the immediate benefits of scaling law applications in the field.