Considering Nonstationary within Multivariate Time Series with Variational Hierarchical Transformer for Forecasting

Abstract: The forecasting of Multivariate Time Series (MTS) has long been an important but challenging task. Due to the non-stationary problem across long-distance time steps, previous studies primarily adopt stationarization method to attenuate the non-stationary problem of the original series for better predictability. However, existing methods always adopt the stationarized series, which ignores the inherent non-stationarity, and has difficulty in modeling MTS with complex distributions due to the lack of stochasticity. To tackle these problems, we first develop a powerful hierarchical probabilistic generative module to consider the non-stationarity and stochastic characteristics within MTS, and then combine it with transformer for a well-defined variational generative dynamic model named Hierarchical Time series Variational Transformer (HTV-Trans), which recovers the intrinsic non-stationary information into temporal dependencies. Being a powerful probabilistic model, HTV-Trans is utilized to learn expressive representations of MTS and applied to forecasting tasks. Extensive experiments on diverse datasets show the efficiency of HTV-Trans on MTS forecasting tasks

• The paper introduces HTV-Trans, a novel model that integrates a hierarchical probabilistic generative module within a Transformer framework to capture nonstationary dynamics.
• It incorporates a balancing parameter that effectively manages both stationary and nonstationary patterns, preserving crucial temporal dependencies for improved long-range forecasts.
• Experimental results on benchmark datasets demonstrate that HTV-Trans outperforms existing Transformer-based models, offering enhanced adaptability to real-world MTS challenges.

Considering Nonstationary within Multivariate Time Series with Variational Hierarchical Transformer for Forecasting

Introduction

The paper "Considering Nonstationary within Multivariate Time Series with Variational Hierarchical Transformer for Forecasting" (2403.05406) addresses the inherent challenges in forecasting multivariate time series (MTS), particularly focusing on the issues of nonstationarity and the stochastic nature of real-world datasets. Historically, methods have leaned towards enforcing stationarity by transforming data to appease models, often at the cost of losing intrinsic temporal dependencies. This work proposes a novel model, the Hierarchical Time series Variational Transformer (HTV-Trans), which integrates a Hierarchical Time Series Probabilistic Generative Module (HTPGM) to effectively model these challenges.

Background and Motivation

Forecasting MTS involves complexities due to correlated temporal dependencies, stochastic elements, and nonstationarity. While recent advances with Transformer models improved forecasting by capturing long-range dependencies, they still suffer from limitations when predicting nonstationary series due to deterministic architectural constraints. Previous solutions often revolved around pre-processing techniques like instance normalization or stationarization, which could lead to over-stationarization, sacrificing the intrinsic properties of the series. The proposed HTV-Trans aims to redress these deficiencies by acknowledging and utilizing the nonstationary and stochastic nature of time series data.

Methodology

HTV-Trans operationalizes its objectives through several key components:

• Hierarchical Generative Module: This module captures multi-scale nonstationary characteristics, allowing the model to maintain complex data patterns and variability over different time horizons. It structures latent variables hierarchically, representing data distributions that recover the nonstationary and non-deterministic features absent in stationary transformations.
• Transformer Integration: By embedding HTPGM within the Transformer framework, HTV-Trans leverages the attention mechanism to model intricate temporal dependencies (Figure 1). The addition of a variational inference mechanism enhances the model's representative capacity for MTS, facilitating both reconstruction and prediction tasks.
• Parameter Balance: The model introduces a parameter $\alpha$ to balance between stationary and non-stationary information, facilitating the recovery of essential temporal structure (Figure 2). This balance is critical for injecting just the right amount of stochasticity back into the series.

  Figure 3: Graphical illustration of different operations of the HTV-Trans: (a) generative process of HTPGM, (b) the inference scheme of HTPGM, (c) the fusion of different scale information and stationarization input series for forecasting.

Experimental Results

The performance evaluation of HTV-Trans was comprehensive, involving multiple datasets such as ETTh1, ETTh2, ETTm1, ETTm2, Weather, and Exchange (Table 1). The results indicated that HTV-Trans consistently outperforms existing Transformer-based models, particularly in scenarios involving long-range forecasting. This superiority is attributed to HTV-Trans's ability to maintain non-stationary dynamics within its latent representations, aiding in better adaptation to complex, real-world data patterns.

Figure 4: The effectiveness evaluation of hierarchical architecture on ETTh1 (left), ETTh2 (middle), and ETTm1 (right) datasets.

Implications and Future Directions

The development of HTV-Trans represents a significant step toward more adaptive and realistic MTS forecasting that acknowledges rather than overlooks the nonstationarity and stochasticity inherent in real data. This model not only enhances forecasting accuracy but also sets a precedent for future explorations into hybrid methods that synergize traditional and deep learning paradigms. Potential extensions of this work could explore more sophisticated fusion techniques within the transformer framework to further enhance adaptability, or alternatively, evaluate the HTV-Trans model across more diverse datasets and real-world applications.

Conclusion

HTV-Trans provides a robust framework for addressing nonstationary and stochastic challenges in MTS forecasting by intricately combining probabilistic generative modeling with advanced attention mechanisms. It stands as a versatile model capable of improving forecasting accuracy across varying domains, asserting the importance of incorporating inherent data variabilities into deep learning architectures. This research holds promise not only for improving time series forecasting techniques but also for inspiring innovative approaches to dealing with the dynamic nature of real-world datasets.