Papers
Topics
Authors
Recent
Search
2000 character limit reached

Large Language Model Guided Knowledge Distillation for Time Series Anomaly Detection

Published 26 Jan 2024 in cs.LG and cs.AI | (2401.15123v1)

Abstract: Self-supervised methods have gained prominence in time series anomaly detection due to the scarcity of available annotations. Nevertheless, they typically demand extensive training data to acquire a generalizable representation map, which conflicts with scenarios of a few available samples, thereby limiting their performance. To overcome the limitation, we propose \textbf{AnomalyLLM}, a knowledge distillation-based time series anomaly detection approach where the student network is trained to mimic the features of the LLM-based teacher network that is pretrained on large-scale datasets. During the testing phase, anomalies are detected when the discrepancy between the features of the teacher and student networks is large. To circumvent the student network from learning the teacher network's feature of anomalous samples, we devise two key strategies. 1) Prototypical signals are incorporated into the student network to consolidate the normal feature extraction. 2) We use synthetic anomalies to enlarge the representation gap between the two networks. AnomalyLLM demonstrates state-of-the-art performance on 15 datasets, improving accuracy by at least 14.5\% in the UCR dataset.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Usad: Unsupervised anomaly detection on multivariate time series. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3395–3404, 2020.
  2. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4183–4192, 2020.
  3. A review on outlier/anomaly detection in time series data. ACM Computing Surveys (CSUR), 54(3):1–33, 2021.
  4. Tempo: Prompt-based generative pre-trained transformer for time series forecasting. arXiv preprint arXiv:2310.04948, 2023.
  5. Neural contextual anomaly detection for time series. arXiv preprint arXiv:2107.07702, 2021.
  6. Imdiffusion: Imputed diffusion models for multivariate time series anomaly detection. arXiv preprint arXiv:2307.00754, 2023.
  7. Graph-augmented normalizing flows for anomaly detection of multiple time series. arXiv preprint arXiv:2202.07857, 2022.
  8. Graph neural network-based anomaly detection in multivariate time series. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 4027–4035, 2021.
  9. Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXiv:2106.14112, 2021.
  10. Timegpt-1. arXiv preprint arXiv:2310.03589, 2023.
  11. Monash time series forecasting archive. arXiv preprint arXiv:2105.06643, 2021.
  12. Unsupervised model selection for time-series anomaly detection. arXiv preprint arXiv:2210.01078, 2022.
  13. Large language models are zero-shot time series forecasters. arXiv preprint arXiv:2310.07820, 2023.
  14. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  15. Local evaluation of time series anomaly detection algorithms. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 635–645, 2022.
  16. An empirical survey of data augmentation for time series classification with neural networks. Plos one, 16(7):e0254841, 2021.
  17. Anomalybert: Self-supervised transformer for time series anomaly detection using data degradation scheme. arXiv preprint arXiv:2305.04468, 2023.
  18. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728, 2023.
  19. Large models for time series and spatio-temporal data: A survey and outlook. arXiv preprint arXiv:2310.10196, 2023.
  20. Anomaly detection in time series with robust variational quasi-recurrent autoencoders. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 1342–1354. IEEE, 2022.
  21. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
  22. Towards a rigorous evaluation of time-series anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7194–7201, 2022.
  23. Staged: A spatial-temporal aware graph encoder-decoder for fault diagnosis in industrial processes. IEEE Transactions on Industrial Informatics, 2023.
  24. Prototype-oriented unsupervised anomaly detection for multivariate time series. In International Conference on Machine Learning, pages 19407–19424. PMLR, 2023.
  25. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730, 2022.
  26. A multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoder. IEEE Robotics and Automation Letters, 3(3):1544–1551, 2018.
  27. Deep one-class classification. In International conference on machine learning, pages 4393–4402. PMLR, 2018.
  28. Rethinking assumptions in deep anomaly detection. arXiv preprint arXiv:2006.00339, 2020.
  29. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5):756–795, 2021.
  30. Multiresolution knowledge distillation for anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14902–14912, 2021.
  31. Timeseries anomaly detection using temporal hierarchical one-class network. Advances in Neural Information Processing Systems, 33:13016–13026, 2020.
  32. Memto: Memory-guided transformer for multivariate time series anomaly detection. arXiv preprint arXiv:2312.02530, 2023.
  33. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2828–2837, 2019.
  34. Test: Text prototype aligned embedding to activate llm’s ability for time series. arXiv preprint arXiv:2308.08241, 2023.
  35. Unraveling theanomaly’in time series anomaly detection: A self-supervised tri-domain solution. arXiv preprint arXiv:2311.11235, 2023.
  36. Deep contrastive one-class time series anomaly detection. In Proceedings of the 2023 SIAM International Conference on Data Mining (SDM), pages 694–702. SIAM, 2023.
  37. Current time series anomaly detection benchmarks are flawed and are creating the illusion of progress. IEEE Transactions on Knowledge and Data Engineering, 2021.
  38. Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.02186, 2022.
  39. Anomaly transformer: Time series anomaly detection with association discrepancy. arXiv preprint arXiv:2110.02642, 2021.
  40. Dcdetector: Dual attention contrastive representation learning for time series anomaly detection. arXiv preprint arXiv:2306.10347, 2023.
  41. Self-supervised learning for time series analysis: Taxonomy, progress, and prospects. arXiv preprint arXiv:2306.10125, 2023.
  42. Beatgan: Anomalous rhythm detection using adversarially generated time series. In IJCAI, volume 2019, pages 4433–4439, 2019.
  43. Pull & push: Leveraging differential knowledge distillation for efficient unsupervised anomaly detection and localization. IEEE Transactions on Circuits and Systems for Video Technology, 2022.
  44. Detecting multivariate time series anomalies with zero known label. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 4963–4971, 2023.
  45. One fits all: Power general time series analysis by pretrained lm. arXiv preprint arXiv:2302.11939, 2023.
  46. Label-free multivariate time series anomaly detection. IEEE Transactions on Knowledge and Data Engineering, 2024.
Citations (6)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 2 likes about this paper.

Sign Up for Free