- The paper introduces AdaPTS, a framework using "adapters" to transform multivariate data so pre-trained univariate foundation models can forecast each dimension independently.
- Experiments show AdaPTS enhances forecasting accuracy and uncertainty quantification compared to baselines, achieving up to a 10% improvement in accuracy on some datasets.
- AdaPTS allows practitioners to reuse existing univariate models for multivariate tasks, making advanced forecasting more accessible and computationally efficient.
Adapting Univariate Foundation Models to Multivariate Time Series Forecasting
The paper presents a novel framework, referred to as AdaPTS, to address the adaptation of pre-trained univariate foundation models (FMs) for multivariate probabilistic time series forecasting. This study arises from the demonstrated efficacy of pre-trained FMs in univariate time series forecasting, yet acknowledges the complexities posed by multivariate data, particularly concerning feature dependencies and prediction uncertainties.
Summary of Contributions
The primary contribution of this research lies in the introduction of "adapters"—feature-space transformations that project multivariate inputs into a latent space suitable for applying univariate FMs independently on each dimension. Adapters facilitate the leveraging of existing pre-trained models for multivariate tasks, without necessitating the development of models specifically trained for multivariate inputs. The implementation of these adapters is inspired by techniques from representation learning and partially stochastic Bayesian neural networks.
The study advances three key contributions:
- Multivariate FM Adaptation: Development of a method to adapt univariate FMs for multivariate tasks using the AdaPTS framework.
- Theoretical Foundations of Adapters: Examination and analysis of the necessity and effectiveness of adapters through linear transformation approaches and partially stochastic Bayesian networks.
- Empirical Validation: Comprehensive experiments showcasing the performance enhancements in forecasting accuracy and uncertainty quantification when applying the AdaPTS framework compared to baseline methods.
Methodology and Framework
AdaPTS operates through a sequence of steps:
- Feature Transformation: Multivariate time series data is transformed into a latent representation via the adapter.
- Independent Prediction: The transformed data is processed by the FM along each dimension independently.
- Inverse Transformation: Predictions are then reverted to the original feature space, utilizing the invertibility constraint of the adapters.
The experimentation involves various adapter configurations, including linear autoencoders, deep non-linear autoencoders, and probabilistic models such as Variational Autoencoders (VAEs) and Dropout-based models. These configurations were tested against a suite of datasets representing diverse real-world scenarios, including Edge Transformer Temperature dataset (ETTh1), ExchangeRate, Weather, and Illness datasets.
Numerical Results and Analysis
The paper reports substantial advances in forecasting accuracy and uncertainty quantification using AdaPTS. For instance, on the ETTh1 dataset, the introduction of dropout-enhanced linear autoencoders yielded up to a 10% improvement in accuracy for certain forecasting horizons. Similarly, on the Illness dataset, employment of the VAE adapter demonstrated a significant improvement in predictive performance.
Additional experiments demonstrated that AdaPTS can reduce the dimensionality of the feature space while maintaining or improving the forecasting performance, indicating computational efficiency. The inclusion of probabilistic adapters facilitated the quantification of forecasting uncertainty, which was shown to be relatively well-calibrated in shorter prediction horizons, albeit less optimal over longer periods.
Implications and Future Directions
The implications of this research are twofold. Practically, AdaPTS enables practitioners to utilize existing univariate models for multivariate forecasting tasks, thus expanding the applicability of these models without the need for extensive fine-tuning or retraining. Theoretically, it contributes to our understanding of representation learning applied to time series data, particularly in poor-resource settings where full retraining is cost-prohibitive.
Future research directions could include the exploration of other probabilistic inference techniques like Markov Chain Monte Carlo to improve uncertainly estimation, at the potential cost of computational efficiency. Furthermore, AdaPTS can be adapted to exploit not only time series data but also any sequential data where multivariate interactions pose a challenge.
This paper significantly contributes to the field by providing a scalable and modular framework for adapting univariate FMs to complex multivariate time series forecasting tasks, thus supporting a broader utilization of pre-trained models across a variety of applications.