Contrastive Vision-Language Pre-training with Limited Resources

Abstract: Pioneering dual-encoder pre-training works (e.g., CLIP and ALIGN) have revealed the potential of aligning multi-modal representations with contrastive learning. However, these works require a tremendous amount of data and computational resources (e.g., billion-level web data and hundreds of GPUs), which prevent researchers with limited resources from reproduction and further exploration. To this end, we propose a stack of novel methods, which significantly cut down the heavy resource dependency and allow us to conduct dual-encoder multi-modal representation alignment with limited resources. Besides, we provide a reproducible baseline of competitive results, namely ZeroVL, with only 14M publicly accessible academic datasets and 8 V100 GPUs. Additionally, we collect 100M web data for pre-training, and achieve comparable or superior results than state-of-the-art methods, further proving the effectiveness of our methods on large-scale data. We hope that this work will provide useful data points and experience for future research in contrastive vision-language pre-training. Code is available at https://github.com/zerovl/ZeroVL.

• The paper proposes novel contrastive techniques that alleviate dataset bias through debiased sampling and coin flipping mixup.
• The paper optimizes computation with decoupled gradient accumulation and auxiliary speedups to enable training on limited hardware.
• The paper demonstrates that a model trained on 14 million image-text pairs achieves competitive performance with large-scale models.

Contrastive Vision-Language Pre-training with Limited Resources

The research described in the paper focuses on reducing the resource dependency of dual-encoder models for vision-language pre-training, primarily using contrastive learning. The authors seek an efficient solution by proposing novel methods that allow competitive performance with limited datasets and computational power.

Context and Motivation

Vision-LLMs such as CLIP and ALIGN have demonstrated robust performance by leveraging large-scale datasets and significant computational resources. However, this high entry barrier limits broader experimentation and innovation, especially for researchers constrained by resources. This paper addresses a crucial need: advancing vision-language multi-modal learning without the requisite of several hundred million image-text pairs and vast arrays of GPUs or TPU cores.

Approach

The paper introduces several techniques to alleviate data and computation constraints:

1. Data Utilization:
   • Debiased Sampling: To counter dataset bias and maximize varied data utilization, the authors propose crafting batches that only include samples from a single dataset. This strategy discourages the model from learning dataset-specific biases, focusing instead on universal semantic concepts.
   • Coin Flipping Mixup: Building on mixup strategies, this method applies augmentation to either the image or text modality randomly within a given batch to interpolate training instances. This leads to better generalization and robustness by providing the model with more varied training data.
2. Computational Optimization:
   • Decoupled Gradient Accumulation: By structuring gradient calculations to work insynchronously with data access across batches, this method effectively mimics the effects of an enormous batch size on limited hardware, making gradient computation feasible even with accumulations from smaller sub-batches dispersed over reduced hardware.
   • Additional Speedups: Techniques like TokenDrop and auxiliary encoders further cut down the training time, enabling efficient training without sacrificing the model's end performance.

Results and Analysis

The implementation of these optimizations yields a robust multi-modal model, ZeroVL, which attains results comparable to leading models trained with substantially more data and computational horsepower. With just 14 million image-text pairs and utilizing 8 V100 GPUs, ZeroVL aligns closely with, or even surpasses, the results of CLIP and ALIGN models in certain retrieval tasks. Furthermore, when trained with 100 million pairs, ZeroVL approaches the performance of models utilizing several billion pairs, confirming the utility of the authors' methods on larger datasets.

Implications and Future Directions

The ability to produce high-performing models using limited resources democratizes research in the field, allowing small labs and individual researchers to engage in model development and extend the exploration of vision-language tasks. It also highlights the potential to revisit existing large-resource models and optimize them to a more feasible computation scale.

The findings suggest various avenues for future work:

• Further refinement of bias elimination techniques, potentially integrating with domain adaptation frameworks.
• Exploration of hybrid models that incorporate elements of both dual- and single-encoder systems to further streamline efficiency.
• Application of these concepts to other domains, such as audio-visual learning or robotics, where resource-efficient multi-modal learning can have significant benefits.

In conclusion, this paper presents a thoughtful and productive path towards resource-efficient dual-encoder vision-LLMs, holding promise for both immediate application and ongoing research development in AI.