Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning

Abstract: Self-Supervised Contrastive Learning has proven effective in deriving high-quality representations from unlabeled data. However, a major challenge that hinders both unimodal and multimodal contrastive learning is feature suppression, a phenomenon where the trained model captures only a limited portion of the information from the input data while overlooking other potentially valuable content. This issue often leads to indistinguishable representations for visually similar but semantically different inputs, adversely affecting downstream task performance, particularly those requiring rigorous semantic comprehension. To address this challenge, we propose a novel model-agnostic Multistage Contrastive Learning (MCL) framework. Unlike standard contrastive learning which inherently captures one single biased feature distribution, MCL progressively learns previously unlearned features through feature-aware negative sampling at each stage, where the negative samples of an anchor are exclusively selected from the cluster it was assigned to in preceding stages. Meanwhile, MCL preserves the previously well-learned features by cross-stage representation integration, integrating features across all stages to form final representations. Our comprehensive evaluation demonstrates MCL's effectiveness and superiority across both unimodal and multimodal contrastive learning, spanning a range of model architectures from ResNet to Vision Transformers (ViT). Remarkably, in tasks where the original CLIP model has shown limitations, MCL dramatically enhances performance, with improvements up to threefold on specific attributes in the recently proposed MMVP benchmark.

• The paper introduces a Multistage Contrastive Learning (MCL) framework that mitigates feature suppression through feature-aware negative sampling and cross-stage representation integration.
• It demonstrates significant accuracy improvements in unimodal tasks (e.g., CIFAR feature recognition from 0.10 to 0.87) and enhanced performance in multimodal models such as CLIP.
• The methodology adapts flexibly to various parameters, offering a promising strategy to enrich learning representations with previously unlearned features.

Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning

Introduction

The paper "Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning" introduces a Multistage Contrastive Learning (MCL) framework to tackle the problem of feature suppression in contrastive learning models such as SimCLR and CLIP (2402.11816). Feature suppression refers to the tendency of contrastive learning models to capture limited information, which adversely affects downstream task performance. The proposed MCL framework progressively learns unlearned features and preserves well-learned features by utilizing feature-aware negative sampling across multiple stages.

Figure 1: Images from Trifeature with different shapes and textures have high similarity in the SimCLR space.

Multistage Contrastive Learning Framework

Feature Suppression Challenge

Contrastive learning maximizes the similarity between an anchor and its positive samples while increasing separation among dissimilar samples [jaiswal2020survey]. However, recent studies have identified that standard contrastive learning often misses substantial portions of input information, causing feature suppression [robinson2021can]. For example, in unimodal settings, the SimCLR model fails to differentiate between different shapes and textures, resulting in visually indistinguishable representations (Figure 1). In multimodal settings, CLIP encounters similar issues with orientation and direction, leading to systematic failures in distinguishing between semantically different images.

Feature-aware Negative Sampling

To mitigate feature suppression, MCL introduces feature-aware negative sampling, where negative samples are selected from clusters assigned in preceding stages (Figure 2). At each stage, the model learns new features distinct from those already learned by ensuring that the anchor and its negative samples have identical feature-aware pseudo labels. This approach forces the model to explore previously unlearned features, progressively enhancing the learned representations.

Figure 2: Overview of the Multistage Contrastive Learning (MCL) Framework. The final representations are the concatenation of representations from each stage.

Cross-stage Representation Integration

After multistage training, MCL employs cross-stage representation integration, where representations from all stages are concatenated to preserve well-learned features. This concatenated representation ensures comprehensive feature retention and improves downstream task performance by maintaining information from each learned stage.

Experiments and Results

Unimodal Settings

In experiments on unobvious datasets such as Trifeature and CIFAR-MNIST, MCL demonstrated significant improvement over baseline models (Table 1). For instance, in CIFAR-MNIST, MCL improved CIFAR feature recognition accuracy from 0.10 to 0.87, showcasing its efficacy in reducing feature suppression.

Multimodal Settings

Testing MCL-enhanced CLIP models on the MMVP benchmark revealed improved performance across multiple image-text pairing tasks and attributes (Table 2). MCL improved average accuracy from 20.0 to 32.6 in ViT models, validating its practicality and effectiveness in multimodal contrastive learning environments.

Figure 3: In each stage, the top 3 most similar samples to the anchor, demonstrating the model's shifting focus from color, texture, to shape across different stages of MCL.

Dynamic Aspect Analysis

In analyzing the impact of training stages (N), the number of clusters (K) in K-means, and temperature ($\tau$) settings, MCL consistently showed better adaptability and robustness across diverse configurations (Table 3). This adaptability underscores its broad applicability and potential to integrate into existing contrastive learning frameworks without requiring substantial computational changes.

Figure 4: Linear evaluation accuracy for each stage individually before cross-stage representation integration.

Conclusion

The proposed Multistage Contrastive Learning framework successfully addresses feature suppression in contrastive learning models by facilitating the acquisition of previously unlearned features through feature-aware negative sampling and cross-stage representation integration. MCL demonstrates robust performance improvements across both unimodal and multimodal settings, presenting a promising direction for enhancing contrastive learning methodologies and their downstream applications in AI. Future work may explore advanced cross-stage integration techniques and extend MCL's applicability to larger-scale models like CLIP and other architectures within the AI domain.