Papers
Topics
Authors
Recent
Search
2000 character limit reached

MixLinear: Extreme Low Resource Multivariate Time Series Forecasting with 0.1K Parameters

Published 2 Oct 2024 in cs.LG | (2410.02081v1)

Abstract: Recently, there has been a growing interest in Long-term Time Series Forecasting (LTSF), which involves predicting long-term future values by analyzing a large amount of historical time-series data to identify patterns and trends. There exist significant challenges in LTSF due to its complex temporal dependencies and high computational demands. Although Transformer-based models offer high forecasting accuracy, they are often too compute-intensive to be deployed on devices with hardware constraints. On the other hand, the linear models aim to reduce the computational overhead by employing either decomposition methods in the time domain or compact representations in the frequency domain. In this paper, we propose MixLinear, an ultra-lightweight multivariate time series forecasting model specifically designed for resource-constrained devices. MixLinear effectively captures both temporal and frequency domain features by modeling intra-segment and inter-segment variations in the time domain and extracting frequency variations from a low-dimensional latent space in the frequency domain. By reducing the parameter scale of a downsampled $n$-length input/output one-layer linear model from $O(n2)$ to $O(n)$, MixLinear achieves efficient computation without sacrificing accuracy. Extensive evaluations with four benchmark datasets show that MixLinear attains forecasting performance comparable to, or surpassing, state-of-the-art models with significantly fewer parameters ($0.1K$), which makes it well-suited for deployment on devices with limited computational capacity.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Leo Breiman. Random forests. Machine Learning, 45:5–32, 2001.
  2. Lightts: Lightweight time series classification with adaptive ensemble distillation. Proceedings of the ACM on Management of Data, 1(2):1–27, 2023.
  3. Holt-winters forecasting: some practical issues. Journal of the Royal Statistical Society Series D: The Statistician, 37(2):129–140, 1988.
  4. Tsmixer: An all-mlp architecture for time series forecasting. Transactions on Machine Learning Research, 2023.
  5. Triformer: Triangular, variable-specific attentions for long sequence multivariate time series forecasting–full version. In International Joint Conference on Artificial Intelligence (IJCAI), 2022.
  6. Arima models to predict next-day electricity prices. IEEE Transactions on Power Systems, 18(3):1014–1020, 2003.
  7. Long-term forecasting with tide: Time-series dense encoder. Transactions on Machine Learning Research, 2023.
  8. Forecasting time series with complex seasonal patterns using exponential smoothing. Journal of the American Statistical Association, 106(496):1513–1527, 2011.
  9. P Kingma Diederik. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
  10. Str: A seasonal-trend decomposition procedure based on regression. INFORMS Journal on Data Science, 13(15):2015–13, 2015.
  11. Robusttad: Robust time series anomaly detection via decomposition and convolutional neural networks. In ACM SIGKDD Workshop on Mining and Learning from Time Series (KDD-MiLeTS), 2020.
  12. Everette S Gardner Jr. Exponential smoothing: The state of the art. Journal of Forecasting, 4(1):1–28, 1985.
  13. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020.
  14. Andrew C Harvey. Forecasting, structural time series models and the Kalman filter. Cambridge university press, 1990.
  15. Domain adaptation for time series under feature and label shifts. In International Conference on Machine Learning (ICML), 2023.
  16. Temporal convolutional networks for anomaly detection in time series. Journal of Physics: Conference Series, 1213(4):042050, 2019.
  17. Predicting patient volumes in hospital medicine: A comparative study of different time series forecasting methods. Scientific Report, 2014.
  18. Sparsetsf: Modeling long-term time series forecasting with 1k parameters. In International Conference on Machine Learning (ICML), 2024.
  19. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations (ICLR), 2021.
  20. Trend filtering: empirical mode decompositions versus l1 and hodrick–prescott. Advances in Adaptive Data Analysis, 3(01n02):41–61, 2011.
  21. Web-search ranking with initialized gradient boosted regression trees. In Proceedings of Machine Learning Research(PMLR), 2011.
  22. A unified nonlinear stochastic time series analysis for climate science. Scientific Reports, 7(1):44228, 2017.
  23. Modeling inertial sensor errors using autoregressive (ar) models. Navigation, 51(4):259–268, 2004.
  24. Gradient boosting machines, a tutorial. Frontiers in Neurorobotics, 7:21, 2013.
  25. Logtrans: Providing efficient local-global fusion with transformer and cnn parallel network for biomedical image segmentation. In High Performance Computing and Communications (HPCC), 2022.
  26. A time series is worth 64 words: Long-term forecasting with transformers. International Conference on Learning Representations (ICLR), 2023.
  27. Forecasting monthly sales retail time series: a case study. In 2017 IEEE 19th Conference on Business Informatics (CBI), volume 1, pp.  1–6. IEEE, 2017.
  28. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
  29. Ceveland Robert, B. Stl: A seasonal-trend decomposition procedure based on loess. Journal of Official Statistics, 6:3–73, 1990.
  30. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR), 2022.
  31. Recent advances in recurrent neural networks. arXiv preprint arXiv:1801.01078, 2017.
  32. Financial time series forecasting with deep learning: A systematic literature review: 2005–2019. Applied Soft Computing, 90:106181, 2020.
  33. Ryan J Tibshirani. Adaptive piecewise polynomial estimation via trend filtering. The Annals of Statistics, 42(1):285–3, 2014.
  34. Comparison of svm and ls-svm for regression. In International Conference on Neural Networks and Brain (ICNNB), 2005.
  35. Pandemic velocity: Forecasting covid-19 in the us with a machine learning & bayesian time series compartmental model. PLoS Computational Biology, 17(3):e1008837, 2021.
  36. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems (NeurIPS), 2021.
  37. Timesnet: Temporal 2d-variation modeling for general time series analysis. International Conference on Learning Representations (ICLR), 2022.
  38. Learning in the frequency domain. In IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR), 2020.
  39. Fits: Modeling time series with 10⁢k10𝑘10k10 italic_k parameters. In International Conference on Learning Representations (ICLR), 2024.
  40. Are transformers effective for time series forecasting? In Association for the Advancement of Artificial Intelligence (AAAI), 2023.
  41. Auto tcl: Automated time series contrastive learning with adaptive augmentations. In International Joint Conference on Artificial Intelligence (IJCAI), 2023.
  42. Parametric augmentation for time series contrastive learning. In International Conference on Learning Representations (ICLR), 2024.
  43. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Association for the Advancement of Artificial Intelligence (AAAI), 2021.
  44. Film: Frequency improved legendre memory model for long-term time series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), 2022a.
  45. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning (ICML), 2022b.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Authors (3)

  1. Aitian Ma 
  2. Dongsheng Luo 
  3. Mo Sha 

Collections

Sign up for free to add this paper to one or more collections.

Sign Up