MOHPER: Multi-objective Hyperparameter Optimization Framework for E-commerce Retrieval System

Abstract: E-commerce search optimization has evolved to include a wider range of metrics that reflect user engagement and business objectives. Modern search frameworks now incorporate advanced quality features, such as sales counts and document-query relevance, to better align search results with these goals. Traditional methods typically focus on click-through rate (CTR) as a measure of engagement or relevance, but this can miss true purchase intent, creating a gap between user interest and actual conversions. Joint training with the click-through conversion rate (CTCVR) has become essential for understanding buying behavior, although its sparsity poses challenges for reliable optimization. This study presents MOHPER, a Multi-Objective Hyperparameter Optimization framework for E-commerce Retrieval systems. Utilizing Bayesian optimization and sampling, it jointly optimizes both CTR, CTCVR, and relevant objectives, focusing on engagement and conversion of the users. In addition, to improve the selection of the best configuration from multi-objective optimization, we suggest advanced methods for hyperparameter selection, including a meta-configuration voting strategy and a cumulative training approach that leverages prior optimal configurations, to improve speeds of training and efficiency. Currently deployed in a live setting, our proposed framework substantiates its practical efficacy in achieving a balanced optimization that aligns with both user satisfaction and revenue goals.

• The paper presents MOHPER, a multi-objective hyperparameter optimization framework that simultaneously enhances CTR, CTCVR, and conversion rates.
• It employs Bayesian optimization and advanced sampling techniques like TPE to address biases and improve training efficiency in e-commerce retrieval systems.
• Real-world deployment validates MOHPER’s scalability and its ability to outperform traditional methods without incurring additional computational overhead.

MOHPER: Multi-objective Hyperparameter Optimization Framework for E-commerce Retrieval Systems

Introduction

The rise of e-commerce platforms has necessitated the development of sophisticated retrieval systems to match user intents with relevant product listings effectively. Traditional e-commerce search engines primarily rely on click-through rates (CTR) as a proxy for user engagement. However, CTR alone may not accurately reflect purchase intent, leading to unoptimized conversions. The paper introduces MOHPER, a Multi-Objective Hyperparameter Optimization (HPO) framework that leverages Bayesian optimization and advanced sampling methods to jointly optimize CTR, Click-Through Conversion Rate (CTCVR), and relevant business-oriented metrics. MOHPER addresses the limitations of single-objective approaches by integrating Bayesian techniques with new meta-configuration strategies to enhance training speed and efficiency in live settings, aligning user satisfaction with revenue goals.

Figure 1: Overall architecture diagram of the proposed framework, MOHPER.

Problem Formulation and Design

Multi-Objective HPO in Retrieval Systems

MOHPER aims to optimize hyperparameters across multiple objectives simultaneously, considering both engagement metrics (CTR) and conversion metrics (CTCVR, CTCAR). Implementations involve sampling and evaluating configurations using Bayesian sampling techniques like TPE and GP. At each trial, hyperparameter configurations influence retrieval processes via transform functions, adapting search heuristics which can involve ranking strategy modifications or relevance adjustments.

Figure 2: Transform function to modulate query request for a search engine as the change of config HP.

Joint Optimization in E-commerce Context

The framework optimizes not just CTR but also direct metrics like CTCVR and Click-Through Cart-Adding Rate (CTCAR). These metrics mitigate bias and sparsity typical in user interaction logs, allowing a balanced optimization that reflects user engagement and purchase intent in real-time e-commerce scenarios.

Figure 3: Illustration of the e-commerce conversion funnel starting from item impressions, highlighting issues of sample selection bias and data sparsity in user interaction logs.

Hyperparameter Evaluation Stage

After training, MOHPER selects optimal configurations from candidates that exhibit strong overall performance across objectives. These configurations undergo meta-stage evaluations for generalization and are used in cumulative training stages for efficiency improvements.

Experiments

Offline Performance and Comparison

MOHPER's performance was evaluated against baselines (e.g., BM25, Random Search, grid search) using metrics like precision@K and nDCG@K across multiple e-commerce scenarios. Results showed MOHPER achieves balanced optimization over multiple objectives in contrast to single-objective methods that optimize either CTR or CTCVR but not both.

Sampler Analysis

Various sampling methods were evaluated, with TPE consistently yielding superior performance in both single and multi-objective setups, demonstrating its adaptability and efficiency across different retrieval environments.

Figure 4: Performance differences across various sampler configurations on the SERP dataset.

Online Deployment and Feasibility

MOHPER was deployed in real-world settings on OHouse's platform, demonstrating practical application efficacy by optimizing retrieval performance without additional computational overhead. Online A/B testing validated MOHPER's scalability and effectiveness in enhancing e-commerce search results.

Conclusion

MOHPER represents a robust framework for multi-objective hyperparameter optimization tailored to e-commerce search systems, incorporating Bayesian strategies for simultaneous metric optimization. Its practical deployment demonstrates enhancements in balancing user engagement with business goals, showcasing potential scalability and future application in various adaptive e-commerce environments. Through cumulative training and meta-configuration innovations, MOHPER ensures efficient hyperparameter tuning, optimizing retrieval systems in dynamic, real-world scenarios.