RecFlow: An Industrial Full Flow Recommendation Dataset

Abstract: Industrial recommendation systems (RS) rely on the multi-stage pipeline to balance effectiveness and efficiency when delivering items from a vast corpus to users. Existing RS benchmark datasets primarily focus on the exposure space, where novel RS algorithms are trained and evaluated. However, when these algorithms transition to real world industrial RS, they face a critical challenge of handling unexposed items which are a significantly larger space than the exposed one. This discrepancy profoundly impacts their practical performance. Additionally, these algorithms often overlook the intricate interplay between multiple RS stages, resulting in suboptimal overall system performance. To address this issue, we introduce RecFlow, an industrial full flow recommendation dataset designed to bridge the gap between offline RS benchmarks and the real online environment. Unlike existing datasets, RecFlow includes samples not only from the exposure space but also unexposed items filtered at each stage of the RS funnel. Our dataset comprises 38M interactions from 42K users across nearly 9M items with additional 1.9B stage samples collected from 9.3M online requests over 37 days and spanning 6 stages. Leveraging the RecFlow dataset, we conduct courageous exploration experiments, showcasing its potential in designing new algorithms to enhance effectiveness by incorporating stage-specific samples. Some of these algorithms have already been deployed online, consistently yielding significant gains. We propose RecFlow as the first comprehensive benchmark dataset for the RS community, supporting research on designing algorithms at any stage, study of selection bias, debiased algorithms, multi-stage consistency and optimality, multi-task recommendation, and user behavior modeling. The RecFlow dataset, along with the corresponding source code, is available at https://github.com/RecFlow-ICLR/RecFlow.

• The paper introduces a dataset that captures both exposed and unexposed items across six stages of an industrial recommendation pipeline.
• It employs a robust methodology, gathering 38 million interactions from 42,000 users and nearly 9 million items over 37 days.
• Experimental findings demonstrate improvements in key metrics like Recall, NDCG, and AUC by addressing distribution shifts and leveraging stage-specific data.

Overview of "RecFlow: An Industrial Full Flow Recommendation Dataset"

The paper introduces "RecFlow," an expansive dataset designed to address the pragmatic challenges of multi-stage recommendation systems (RS) in industrial settings. Unlike conventional datasets that predominantly focus on the exposure space of RS, RecFlow is comprehensive, capturing both exposed and unexposed items across all stages of the recommendation pipeline. The dataset comprises 38 million interactions from 42,000 users, across nearly 9 million items, and includes 1.9 billion stage samples collected from 9.3 million online requests over 37 days.

Key Contributions

The primary contribution of the RecFlow dataset lies in its representation of the complete flow of an industrial RS, from retrieval to edge ranking. It includes data from all six stages in the pipeline: retrieval, pre-ranking, coarse ranking, ranking, re-ranking, and edge ranking. Notably, it captures unexposed items, which are traditionally overlooked, thereby addressing the distribution shift between offline training and online serving spaces. Additionally, RecFlow also integrates multi-type user feedback, enabling a broader spectrum of RS research, including tasks like multi-task recommendation, user behavior modeling, and the study of selection bias.

Dataset Characteristics

RecFlow is notable for its scale and comprehensiveness. It reflects a pragmatic RS pipeline in its full complexity, with stage-specific samples representing both intermediate and final outputs in RS processing. This allows researchers to explore how to mitigate the discrepancies between the training and serving distributions and to design algorithms that can operate across all stages cohesively.

The dataset is structured around rich feature sets, including user demographics, video metadata, and various user interactions. For instance, it records user feedback not just as discrete actions (like, share) but also as behavioral indicators (e.g., view duration), which can be pivotal for advanced UBM techniques.

In terms of its collection methodology, data privacy is rigorously addressed. All personal identifiers are anonymized, ensuring compliance with international privacy regulations.

Experimental Findings

The authors conduct extensive experiments demonstrating the utility of RecFlow for different RS tasks. For retrieval stages, they explore hard negative mining, showing that incorporating stage-specific unexposed items as negatives improves top-K retrieval metrics. For both coarse ranking and ranking stages, they tackle distribution shift and interplay with subsequent stages using FS-LTR, a framework that incorporates stage-specific hierarchy and consistency, leading to significant improvements in Recall and NDCG metrics.

Furthermore, they explore auxiliary ranking tasks and competitive user behavior modeling to enhance traditional pointwise models. These techniques showcase moderate improvements in both AUC and cross-stage consistency, validating RecFlow’s potential for improving both traditional and emerging RS challenges.

Implications and Future Directions

RecFlow sets a new benchmark for RS datasets by aligning offline modeling with online implementation realities. This alignment is poised to precipitate advancements in multi-stage RS, debiased learning algorithms, and consistency modeling across RS stages. Furthermore, RecFlow's scale and depth lend themselves to exploring large-scale simulations or synthetic model optimizations.

Looking forward, RecFlow opens several avenues for research. It encourages investigations into novel collaborative filtering techniques, online learning paradigms that reduce distribution shifts, and incorporating contextual data beyond user-item interactions. Moreover, the rich feedback information supports multi-objective optimization to balance user satisfaction with business metrics.

In conclusion, RecFlow is a substantial addition to the data resources available for RS research, offering a detailed view of the industrial RS pipeline. Recognizing its foundational role, researchers are likely to build upon this resource to develop next-generation RS methodologies that harmonize offline and online environments, ultimately pushing the field towards more effective and adaptive systems.