Extreme Adaptive Transformer for Time Series Forecasting

Abstract: Time series forecasting remains challenging when the underlying data contain rare but critical extreme events. This issue is particularly important in hydrologic forecasting, where streamflow distributions are often highly skewed and extreme peaks can have substantial impacts on flood monitoring, water resource management, and early warning systems. Although Transformer-based forecasting models have achieved strong performance by modeling long-range temporal dependencies, they typically treat all time points uniformly and may therefore underrepresent rare extreme patterns. In this paper, we propose the Extreme-Adaptive Transformer (Exformer), a forecasting framework designed to explicitly model temporal dependencies involving both normal and extreme events. Exformer introduces an extreme-adaptive attention mechanism composed of three sparse components: Local, Stride, and Extreme. The Local and Stride components capture short-term and periodic temporal dependencies, respectively, while the Extreme component selectively models event-aware dependencies between normal and extreme streamflow patterns. Experiments on four real-world hydrologic streamflow datasets show that Exformer achieves superior 3-day forecasting performance compared with state-of-the-art baselines. Our findings demonstrate that explicitly incorporating extreme-aware attention improves the forecasting capacity of Transformer models on imbalanced time series with rare but consequential events.

• The paper introduces Exformer, which incorporates an Extreme-Adaptive Attention mechanism to significantly enhance rare event forecasting in time series.
• It utilizes a decomposition pipeline to separate seasonal and trend components and employs patch-based embeddings to efficiently process multivariate data.
• Exformer achieves up to 50% improvement in MAPE and reduced computational cost, demonstrating robust performance on challenging hydrologic datasets.

Extreme-Adaptive Transformer for Time Series Forecasting: An Expert Analysis

Introduction and Motivation

Accurate multivariate time series (MTS) forecasting under rare, severe events is a critical challenge in domains such as hydrology, where rare streamflow peaks profoundly impact flood warning and water resource management. Standard deep learning methods, including canonical Transformer architectures, are generally optimized for average-case performance and tend to underrepresent such extreme values due to their rarity and distributional imbalance. Recent architectures have sought to address this, but prevailing attention mechanisms do not adapt the pattern of attention based on event severity.

This work introduces the Extreme-Adaptive Transformer (Exformer), a model designed to directly target this limitation via its Extreme-Adaptive Attention mechanism, yielding both theoretical innovation and strong empirical results in skewed hydrologic series forecasting.

Exformer Architecture and Methodological Innovations

Exformer is an encoder-only Transformer with a decomposition-based preprocessing pipeline, isolating seasonal and trend components from the input MTS. The seasonal component is modeled by the Transformer encoder, while the trend is linearly predicted. The framework employs patch-based embeddings with dimension-invariant mapping to preserve both the temporal and variable dimensions across channels.

Figure 1: The architecture of the proposed Exformer framework outlines the seasonal-trend decomposition, patch-based embedding, and incorporation of Extreme-Adaptive Attention in the encoder.

At patch granularity, tokens are labeled as normal or extreme via GMM-based outlier detection and thresholding. Patch-level labels drive the construction of an adaptive attention mask, resulting in selective computation of attention only on informative interactions.

Extreme-Adaptive Attention

The Extreme-Adaptive Attention mechanism comprises three components:

• Local: For normal query tokens, attention is restricted to a local temporal window of normal keys, capturing short-range dependencies.
• Stride: Normal queries also attend to other normal keys at fixed temporal intervals, encoding seasonal and periodicity patterns.
• Extreme: For extreme query tokens, attention is exclusively among extreme tokens, regardless of temporal proximity, directly preserving rare but essential dependencies.

The final attention mask is constructed by combining the above in a query-adaptive manner, ensuring that computational resources are focused on the most semantically meaningful relationships.

Figure 2: Comparison of attention masks—Dozer, canonical extreme, and the proposed Extreme-Adaptive Attention—demonstrating conditional masking based on the extremity of tokens.

This design yields sub-quadratic complexity, $\mathcal{O}((w+s)I/p + n_e^2)$, with $w$ and $s$ as small local/stride window sizes and $n_e$ as the number of extreme tokens, significantly reducing FLOPs and memory requirements relative to standard attention.

Empirical Results

Experiments were conducted on four real-world hydrologic datasets (Ross, Saratoga, UpperPen, SFC). Exformer was evaluated against specialized extreme-event predictors (DAN, NEC+, PFformer), generic LSTF Transformers (Informer, FEDformer, iTransformer), and linear baselines (NLinear, DLinear).

Forecasting Performance

Exformer achieved the best or tied-best RMSE on three datasets and the best MAPE across all tasks. For example, on the challenging UpperPen dataset at a 3-day horizon (288 steps), Exformer produced RMSE of 0.96 and MAPE of 0.04, outperforming comparable Transformer variants and extreme-adaptive representation learning models.

Figure 3: Visualization of Exformer forecasts on the Saratoga dataset, showing close tracking of ground-truth, especially for peaks, versus strong baselines DAN and PFformer.

Exformer consistently exhibited up to a 50% reduction in forecast error (MAPE) over PFformer and substantial gains relative to DAN and iTransformer on skewed datasets.

Computational Efficiency

The architecture achieves a superior accuracy-efficiency balance. For a representative configuration (input length 1440, forecasting horizon 288), Exformer required only 31.6 GFLOPs and 240.85 MB of memory, representing $5 \times - 18 \times$ reductions relative to Transformer or LSTM-based baselines, while maintaining parameter counts in an acceptable range (14.3M).

Ablation and Attention Variant Studies

Ablation experiments isolating the Local, Stride, and Extreme components confirm that each aspect contributes positively, but the Extreme component is most critical for rare-event accuracy. Comparative studies with alternative attention mechanisms (Dozer, full, FedAttn, AutoCorr, ProbSparse) establish that Extreme-Adaptive Attention gives strictly superior MAPE/RMSE, especially for imbalanced datasets.

Robustness to Extreme Label Definition

Sensitivity analysis across labeling thresholds for defining extreme events shows Exformer’s performance is robust, with optimal accuracy attained when extremes are conservatively defined; labeling too many points as extreme degrades performance by diluting the anomalous class.

Figure 4: RMSE sensitivity to the threshold parameter used for extreme event labeling, illustrating stability across a broad range and favoring selective, high-threshold settings.

Theoretical and Practical Implications

Explicitly encoding event-aware, query-adaptive attention addresses a major limitation of prior sparse and full attention models. In imbalanced-regime time series with rare but important anomalies, this approach enables direct modeling of interactions that matter most to downstream decisions. The architectural mechanism is data- and domain-agnostic, applicable to other imbalance-intensive forecasting domains beyond hydrology.

Computationally, this design pushes sequence lengths tractable for temporal attention further, facilitating deployment in real-time early warning and resource management systems that demand both high fidelity for outlier events and low latency.

Future Directions

Potential developments building on Exformer include:

• Learnable event-severity scoring instead of binary outlier thresholding.
• Joint inference of outlierness and sequence forecasting within the attention mechanism.
• Exploration of multi-scale and hierarchical extreme labeling for nested rare event architectures.
• Domain adaptation and transfer learning applications for zero-shot extreme event prediction.

Conclusion

Exformer introduces a principled, computationally efficient method for time series forecasting in the presence of extreme imbalance and rare events, leveraging an Extreme-Adaptive Attention mechanism that reallocates modeling capacity towards both general temporal dependencies and crucial extreme patterns. Its empirical and theoretical contributions demonstrate substantial progress toward robust, real-world time series models capable of informing critical decision-making under non-stationarity and skewed target distributions.