Self-supervised Contrastive Representation Learning for Semi-supervised Time-Series Classification

Abstract: Learning time-series representations when only unlabeled data or few labeled samples are available can be a challenging task. Recently, contrastive self-supervised learning has shown great improvement in extracting useful representations from unlabeled data via contrasting different augmented views of data. In this work, we propose a novel Time-Series representation learning framework via Temporal and Contextual Contrasting (TS-TCC) that learns representations from unlabeled data with contrastive learning. Specifically, we propose time-series-specific weak and strong augmentations and use their views to learn robust temporal relations in the proposed temporal contrasting module, besides learning discriminative representations by our proposed contextual contrasting module. Additionally, we conduct a systematic study of time-series data augmentation selection, which is a key part of contrastive learning. We also extend TS-TCC to the semi-supervised learning settings and propose a Class-Aware TS-TCC (CA-TCC) that benefits from the available few labeled data to further improve representations learned by TS-TCC. Specifically, we leverage the robust pseudo labels produced by TS-TCC to realize a class-aware contrastive loss. Extensive experiments show that the linear evaluation of the features learned by our proposed framework performs comparably with the fully supervised training. Additionally, our framework shows high efficiency in the few labeled data and transfer learning scenarios. The code is publicly available at \url{https://github.com/emadeldeen24/CA-TCC}.

• The paper presents TS-TCC, a novel contrastive framework that integrates temporal and contextual contrasting for effective self-supervised time-series representation learning.
• It extends the approach to semi-supervised learning with CA-TCC, leveraging few labeled samples and supervised contrastive loss to form class-aware representations.
• The method achieves competitive accuracy and macro F1-scores compared to fully supervised techniques, demonstrating robust performance across diverse real-world time-series applications.

Self-supervised Contrastive Representation Learning for Semi-supervised Time-Series Classification

The paper presents an innovative approach to time-series representation learning through a novel framework, Time-Series representation learning via Temporal and Contextual Contrasting (TS-TCC). This framework aims to enhance the ability to learn from unlabeled time-series data, a critical task given the scarcity of labeled datasets in real-world applications. The authors extend this framework to semi-supervised learning scenarios with Class-Aware TS-TCC (CA-TCC), which further improves upon the learned representations by utilizing a few labeled samples.

The TS-TCC framework leverages contrastive learning techniques adapted for the unique characteristics of time-series data. It introduces two major components: temporal and contextual contrasting. Temporal contrasting focuses on preserving temporal relations within time-series data by employing innovative time-series-specific data augmentations. These include weak and strong data augmentations to create varied views of the data. The augmentations facilitate a challenging cross-view prediction task, where temporal dependencies are leveraged to learn more robust representations.

In contextual contrasting, TS-TCC utilizes the inherent contextual information within a time-series sample to maximize agreements between different contexts of the same sample, which ensures discriminative representations are learned.

Key Technical Contributions and Results

The contributions of this paper lie in the unique way TS-TCC addresses the peculiar temporal nature of time-series data. By integrating temporal dependencies into both the augmentation and contrasting processes, the framework surpasses conventional self-supervised methods that primarily focus on image data properties. The framework shows competitive performance compared to fully supervised training, leveraging mechanisms that are not overly reliant on extensive labeled data.

The paper further expands on the utility of TS-TCC in semi-supervised contexts through CA-TCC. This variant incorporates pseudo labels generated from the few available labeled samples to facilitate class-aware representation learning. The use of supervised contrastive loss in CA-TCC allows the model to form positive pairs of samples sharing the same class, an advantage over traditional contrastive learning where such semantic information is absent.

The results demonstrate substantial improvements over existing self-supervised and semi-supervised techniques, with TS-TCC reliably achieving high accuracy and macro F1-scores across a diverse set of time-series datasets. Moreover, the frameworks manage to maintain high performance even when only a minimal fraction of labeled samples is available, showcasing their effectiveness and robustness.

Implications and Future Directions

The TS-TCC and CA-TCC frameworks offer promising pathways for tackling the challenges of learning representations from time-series data with limited labeled examples. The methodological advancements presented here suggest several practical applications in domains heavily reliant on time-series data, such as healthcare, finance, and IoT-based monitoring systems.

In terms of theoretical implications, the successful integration of temporal contrasting with contextual contrasting embodies a significant step forward in adapting self-supervised learning paradigms to non-image domains. This opens avenues for further exploration into contrastive learning strategies that can generalize effectively across a multitude of temporal data representations.

Looking forward, future work may investigate the scalability of these approaches to even more complex and diverse time-series problems. Additionally, exploring other forms of contrastive learning and augmentation techniques, as well as fine-tuning the balance between weak and strong augmentations, could yield even more precise representations. Finally, the integration of domain-specific knowledge into these frameworks might further enhance their effectiveness and broaden their applicability across various real-world contexts.