Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment

Abstract: Recent contrastive multimodal vision-LLMs like CLIP have demonstrated robust open-world semantic understanding, becoming the standard image backbones for vision-language applications. However, recent findings suggest high semantic similarity between well-trained unimodal encoders, which raises a key question: Is there a plausible way to connect unimodal backbones for vision-language tasks? To this end, we propose a novel framework that aligns vision and language using frozen unimodal encoders. It involves selecting semantically similar encoders in the latent space, curating a concept-rich dataset of image-caption pairs, and training simple MLP projectors. We evaluated our approach on 12 zero-shot classification datasets and 2 image-text retrieval datasets. Our best model, utilizing DINOv2 and All-Roberta-Large text encoder, achieves 76(\%) accuracy on ImageNet with a 20-fold reduction in data and 65-fold reduction in compute requirements compared multi-modal alignment where models are trained from scratch. The proposed framework enhances the accessibility of multimodal model development while enabling flexible adaptation across diverse scenarios. Code and curated datasets are available at \texttt{github.com/mayug/freeze-align}.

• The paper demonstrates that unimodal encoders can be efficiently aligned using lightweight projectors to achieve competitive zero-shot multimodal performance.
• It details a methodology that employs CKA for selecting encoder pairs and curates a concept-rich dataset for robust image-text semantic correspondence.
• Results show a 76% ImageNet accuracy and enhanced retrieval, localization, and multilingual capabilities while reducing data and compute requirements drastically.

Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment

Introduction

The paper "Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment" (2409.19425) addresses the challenge of connecting unimodal vision and language encoders to perform zero-shot vision-language tasks. The main motivation lies in leveraging the robust capabilities of unimodal encoders, particularly vision encoders like DINOv2 and language encoders like All-Roberta-Large, which possess high semantic similarities in their embedding spaces. By utilizing these existing powerful models, the proposed approach aims to align vision and language modalities using projection layers only, thereby circumventing the need for extensive retraining or using large multimodal datasets.

Methodology

Encoder and Data Selection

The authors introduce a structured framework for selecting encoders and curating datasets. The selection process involves using Centered Kernel Alignment (CKA) to measure the semantic similarity between vision and LLMs, guiding the choice of encoder pairs with potential for efficient alignment. For datasets, the approach focuses on creating a concept-rich dataset of image-caption pairs by balancing class coverage and ensuring high-quality semantic correspondence between image and text modalities. The dataset accrues a broad range of concepts essential for zero-shot domain transfer capabilities.

Projectors and Training

The alignment relies on lightweight multilayer perceptron (MLP) projectors, trained using a standard contrastive loss, between frozen unimodal models. The projectors are applied to both local tokens and global embeddings (CLS tokens) in a residual manner, enhancing the cross-modal alignment. This architectural choice captures global and local encoder information efficiently, crucial for maintaining the quality of unimodal features and ensuring robust performance across zero-shot tasks.

Results and Analysis

Zero-shot Classification and Retrieval

The proposed method achieves a noteworthy accuracy of 76% on ImageNet, outperforming comparable models trained on considerably larger datasets such as 400 million image-caption pairs. This performance is achieved with a 20-fold reduction in data and 65-fold reduction in computing requirements. Moreover, the model demonstrates superior results across zero-shot text and image retrieval tasks, reflecting the effectiveness of the alignment strategy. The approach not only matches conventional CLIP models but also leverages enhanced unimodal encoder features, leading to improved retrieval accuracies.

Zero-shot Localization and Multilingual Capabilities

In addition to classification and retrieval, the model exhibits promising results in semantic segmentation tasks, outperforming other baseline models due to DINOv2's superior localization features. By facilitating easy swapping of text encoders, the framework supports multilingual scenarios efficiently, as demonstrated in image-text retrieval tasks across several languages, even those not directly trained with multilingual data.

Implications and Future Directions

The research presents significant implications for developing multimodal models with reduced resource requirements, making such frameworks more accessible for varied applications. Practically, the methodology allows for rapid adaptation to new tasks or languages by swapping encoders and retraining only lightweight projectors. The findings suggest a path towards more dynamic and scalable AI systems that do not rely heavily on extensive training datasets or compute power. Future developments could explore finer-grain alignment techniques or extend the framework to accommodate additional modalities, thereby broadening its applicability and increasing efficiencies in multimodal model training and deployment.

Conclusion

The paper successfully demonstrates that competitive multimodal performance can be achieved by aligning well-trained unimodal encoders using projection layers alone. This powerful approach balances efficiency and adaptability, highlighting a promising direction for scalable multimodal AI research without the exorbitant costs traditionally associated with training vast multimodal models.