Scalable Gaussian Processes with Latent Kronecker Structure

Abstract: Applying Gaussian processes (GPs) to very large datasets remains a challenge due to limited computational scalability. Matrix structures, such as the Kronecker product, can accelerate operations significantly, but their application commonly entails approximations or unrealistic assumptions. In particular, the most common path to creating a Kronecker-structured kernel matrix is by evaluating a product kernel on gridded inputs that can be expressed as a Cartesian product. However, this structure is lost if any observation is missing, breaking the Cartesian product structure, which frequently occurs in real-world data such as time series. To address this limitation, we propose leveraging latent Kronecker structure, by expressing the kernel matrix of observed values as the projection of a latent Kronecker product. In combination with iterative linear system solvers and pathwise conditioning, our method facilitates inference of exact GPs while requiring substantially fewer computational resources than standard iterative methods. We demonstrate that our method outperforms state-of-the-art sparse and variational GPs on real-world datasets with up to five million examples, including robotics, automated machine learning, and climate applications.

• The paper introduces a latent Kronecker structure that enables exact GP inference on incomplete data without resorting to approximate sparse methods.
• It leverages Kronecker products to mitigate the O(n³) complexity, achieving scalable performance on datasets with up to five million examples.
• Empirical results in robotics, AutoML, and climate modeling demonstrate superior accuracy and efficiency over state-of-the-art sparse GP models.

Scalable Gaussian Processes with Latent Kronecker Structure

This paper investigates a new method for applying Gaussian Processes (GPs) to large datasets by introducing the concept of latent Kronecker structure. The primary challenge in utilizing exact Gaussian Processes for large scale data lies in their computational demands, particularly the $\mathcal{O}(n^3)$ complexity linked to solving linear systems with an $n \times n$ kernel matrix. Traditional approaches often rely on sparse approximations or variational methods to manage this complexity, but they come with inherent limitations in model accuracy and may produce overconfident predictions.

The authors propose a method that leverages the Kronecker product to structure kernel matrices efficiently. While Kronecker products offer significant computational acceleration, their application has been limited by the assumption of complete data grids. Real-world data, characterized by missing observations, frequently violates these assumptions, reducing the scalability benefits. To overcome this, the authors introduce the latent Kronecker structure methodology that factors observed data covariance matrices as projections of latent Kronecker products. This approach retains the computational benefits of Kronecker structures while adapting it to partially observed datasets.

The paper's empirical analysis demonstrates the advantages of this method with applications to robotics inverse dynamics, automated machine learning (AutoML), and climate modeling. Notably, the latency Kronecker GP (LKGP) model consistently outperformed state-of-the-art sparse and variational GP models, such as SVGP and VNNGP, in scalability while maintaining accuracy without model approximation. The LKGP approach showed superior performance for datasets with up to five million examples, indicating its potential for effective inference in large-scale applications. Moreover, the memory and time efficiency results validate the theoretical predictions about its scalability.

This latent Kronecker approach has significant implications for the practical application of GPs in scenarios requiring both scalability and precision. It provides a pathway for deploying exact GP models in fields like robotics and climate science, where data may not be fully observed yet demands high accuracy. The methodology expands the available toolkit for machine learning practitioners, offering a scalable solution that avoids the pitfalls of previous GP approximations.

Looking forward, this work paves the way for further exploration into leveraging algebraic structures in machine learning. There is potential for integrating this latent structure approach with other advanced GP models and extending it into domains involving high-dimensional tensor data or complex temporal patterns. Additionally, the paper invites future inquiries into specialized kernels that could enhance the latent Kronecker GP's adaptability to various real-world problems.