Papers
Topics
Authors
Recent
Search
2000 character limit reached

NuwaTS: a Foundation Model Mending Every Incomplete Time Series

Published 24 May 2024 in cs.LG and cs.AI | (2405.15317v3)

Abstract: Time series imputation is critical for many real-world applications and has been widely studied. However, existing models often require specialized designs tailored to specific missing patterns, variables, or domains which limits their generalizability. In addition, current evaluation frameworks primarily focus on domain-specific tasks and often rely on time-wise train/validation/test data splits, which fail to rigorously assess a model's ability to generalize across unseen variables or domains. In this paper, we present \textbf{NuwaTS}, a novel framework that repurposes Pre-trained LLMs (PLMs) for general time series imputation. Once trained, NuwaTS can be applied to impute missing data across any domain. We introduce specialized embeddings for each sub-series patch, capturing information about the patch, its missing data patterns, and its statistical characteristics. By combining contrastive learning with the imputation task, we train PLMs to create a versatile, one-for-all imputation model. Additionally, we employ a plug-and-play fine-tuning approach, enabling efficient adaptation to domain-specific tasks with minimal adjustments. To evaluate cross-variable and cross-domain generalization, we propose a new benchmarking protocol that partitions the datasets along the variable dimension. Experimental results on over seventeen million time series samples from diverse domains demonstrate that NuwaTS outperforms state-of-the-art domain-specific models across various datasets under the proposed benchmarking protocol. Furthermore, we show that NuwaTS generalizes to other time series tasks, such as forecasting. Our codes are available at https://github.com/Chengyui/NuwaTS.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815, 2024.
  2. Accurate medium-range global weather forecasting with 3d neural networks. Nature, 619(7970):533–538, 2023.
  3. Multi-patch prediction: Adapting llms for time series representation learning. ArXiv, abs/2402.04852, 2024.
  4. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  5. Msgnet: Learning multi-scale inter-series correlations for multivariate time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 11141–11149, 2024.
  6. Brits: Bidirectional recurrent imputation for time series. Advances in neural information processing systems, 31, 2018.
  7. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  8. Saits: Self-attention-based imputation for time series. Expert Systems with Applications, 219:119619, 2023.
  9. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36, 2024.
  10. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations, 2022.
  11. Visual prompt tuning. In European Conference on Computer Vision, pages 709–727. Springer, 2022.
  12. Time-LLM: Time series forecasting by reprogramming large language models. In The Twelfth International Conference on Learning Representations, 2024.
  13. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  14. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
  15. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023.
  16. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In International Conference on Learning Representations, 2018.
  17. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61–68, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  18. Unitime: A language-empowered unified model for cross-domain time series forecasting. CoRR, abs/2310.09751, 2023.
  19. Largest: A benchmark dataset for large-scale traffic forecasting. Advances in Neural Information Processing Systems, 36, 2024.
  20. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024.
  21. Autotimes: Autoregressive time series forecasters via large language models. arXiv preprint arXiv:2402.02370, 2024.
  22. Naomi: Non-autoregressive multiresolution sequence imputation. Advances in neural information processing systems, 32, 2019.
  23. E²gan: End-to-end generative adversarial network for multivariate time series imputation. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 3094–3100. International Joint Conferences on Artificial Intelligence Organization, 7 2019.
  24. Adversarial joint-learning recurrent neural network for incomplete time series classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(4):1765–1776, 2020.
  25. End-to-end incomplete time-series modeling from linear memory of latent variables. IEEE transactions on cybernetics, 50(12):4908–4920, 2019.
  26. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2023.
  27. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
  28. S2ip-llm: Semantic space informed prompt learning with llm for time series forecasting. ArXiv, abs/2403.05798, 2024.
  29. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
  30. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  31. Test: Text prototype aligned embedding to activate llm’s ability for time series. ArXiv, abs/2308.08241, 2023.
  32. Joint modeling of local and global temporal dynamics for multivariate time series forecasting with missing values. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5956–5963, 2020.
  33. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34:24804–24816, 2021.
  34. Unsupervised representation learning for time series with temporal neighborhood coding. In International Conference on Learning Representations, 2020.
  35. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  36. mice: Multivariate imputation by chained equations in r. Journal of statistical software, 45:1–67, 2011.
  37. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
  38. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The eleventh international conference on learning representations, 2022.
  39. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34:22419–22430, 2021.
  40. Temporal regularized matrix factorization for high-dimensional time series prediction. Advances in neural information processing systems, 29, 2016.
  41. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121–11128, 2023.
  42. Multimodal pathway: Improve transformers with irrelevant data from other modalities, 2024.
  43. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning, pages 27268–27286. PMLR, 2022.
  44. One fits all: Power general time series analysis by pretrained lm. Advances in neural information processing systems, 36, 2024.
  45. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43–76, 2021.

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free