Conformal Prediction for Verifiable Learned Query Optimization

Abstract: Query optimization is critical in relational databases. Recently, numerous Learned Query Optimizers (LQOs) have been proposed, demonstrating superior performance over traditional hand-crafted query optimizers after short training periods. However, the opacity and instability of machine learning models have limited their practical applications. To address this issue, we are the first to formulate the LQO verification as a Conformal Prediction (CP) problem. We first construct the CP model and obtain user-controlled bounded ranges for the actual latency of LQO plans before execution. Then, we introduce CP-based runtime verification along with violation handling to ensure performance prior to execution. For both scenarios, we further extend our framework to handle distribution shifts in the dynamic environment using adaptive CP approaches. Finally, we present CP-guided plan search, which uses actual latency upper bounds from CP to heuristically guide query plan construction. We integrated our verification framework into three LQOs (Balsa, Lero, and RTOS) and conducted evaluations on the JOB and TPC-H workloads. Experimental results demonstrate that our method is both accurate and efficient. Our CP-based approaches achieve tight upper bounds, reliably detect and handle violations. Adaptive CP maintains accurate confidence levels even in the presence of distribution shifts, and the CP-guided plan search improves both query plan quality (up to 9.84x) and planning time, with a reduction of up to 74.4% for a single query and 9.96% across all test queries from trained LQOs.

• The paper introduces a Conformal Prediction method for runtime verification of Learned Query Optimizers with formal latency guarantees.
• It details a CP-guided plan search strategy that enhances query plan quality and reduces planning time, improving performance by up to 9.84x.
• The framework adapts to distribution shifts in dynamic workloads, detecting violations early and integrating fallback mechanisms when needed.

Conformal Prediction for Verifiable Learned Query Optimization

The paper "Conformal Prediction for Verifiable Learned Query Optimization" introduces an innovative approach for verifying Learned Query Optimizers (LQOs) using Conformal Prediction (CP), addressing a significant challenge in database query optimization. The work focuses on enhancing the reliability and performance of learning-based query optimization techniques, which have shown promise but face practical barriers due to unpredictability and stability issues.

Introduction to Learned Query Optimization and Its Challenges

Query optimization remains a pivotal yet challenging component of database management systems, translating user queries into efficient execution plans. Traditional query optimizers are predominantly heuristic-based, relying on fixed optimization rules. However, Learned Query Optimizers (LQOs) are increasingly preferred for their adaptability and superior performance in complex query scenarios. Despite their advantages, LQOs face critical challenges, notably the potential for generating suboptimal execution plans during early trainings, susceptibility to catastrophic performance in sparse data scenarios, and performance degradation under workload distribution shifts [neo].

Conformal Prediction-Based Verification Framework

The paper redefines LQO verification as a Conformal Prediction (CP) problem, presenting a robust statistical framework for runtime verification of LQOs with formal guarantees.

• Latency Bounds for LQOs: The authors develop CP-based latency bounds, enabling user-controlled latency constraints and providing predictive latency intervals before executing query plans.
• Runtime Verification: CP-based runtime verification allows for early detection of violations in latency constraints during plan construction, facilitating dynamic violation handling such as fallback mechanisms to traditional query optimizers.
• Adaptive Handling of Distribution Shifts: This work further introduces an adaptive CP framework to manage distribution shifts effectively, ensuring that LQOs maintain performance across changing workloads.

  Figure 1: Architecture of Learned Query Optimization with highlighted ML decision components indicating target verification areas.

CP-Guided Plan Search Strategy

In addition to verification capabilities, the paper proposes a CP-guided plan search strategy. Unlike traditional LQOs that depend on predicted costs, this method utilizes CP-derived latency upper bounds to make more informed optimization decisions during query plan generation. This new approach enhances plan quality and reduces planning time, which is particularly beneficial for large and complex queries.

Performance Evaluation and Experimental Results

The verification and enhancement frameworks were integrated into three LQOs—Balsa, Lero, and RTOS—and tested across standard workloads, such as the JOB and TPC-H benchmarks. The experiments demonstrated several crucial outcomes:

• Accuracy and Efficiency: The CP-based verification methods reliably detected and managed constraint violations, with adaptive CP maintaining high confidence levels amidst distribution shifts.
• Improved Query Plan Quality: The CP-guided plan search improved query plans by up to 9.84x in certain scenarios, with significant reductions in planning latency (up to 74.4% for individual queries and 9.96% overall).
• Reduction in Execution Latency: Handling long-latency queries via the proposed methodology substantially decreased execution times, highlighting its practical efficacy.

Implications and Future Developments

The proposed CP-based frameworks hold tremendous potential for enhancing the robustness and reliability of LQOs in real-world applications. The adaptive CP approach, in particular, presents a scalable solution to the challenges posed by dynamic environments and distribution shifts. As database systems continue to evolve, integrating runtime verification and CP-guided strategies could become integral to future query optimization frameworks, potentially extending beyond LQOs to other learned components in data management systems.

Conclusion

This work represents a significant step forward in addressing the verification challenges of LQOs using Conformal Prediction. By providing a mechanism for runtime verification and latency-bound guidance in query plan generation, the paper paves the way for more trustworthy and efficient database optimization systems, capable of adapting to dynamic and complex workloads while maintaining performance stability and reliability.