HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding

Abstract: The rapid advance of LLMs has catalyzed the development of Vision-LLMs (VLMs). Monolithic VLMs, which avoid modality-specific encoders, offer a promising alternative to the compositional ones but face the challenge of inferior performance. Most existing monolithic VLMs require tuning pre-trained LLMs to acquire vision abilities, which may degrade their language capabilities. To address this dilemma, this paper presents a novel high-performance monolithic VLM named HoVLE. We note that LLMs have been shown capable of interpreting images, when image embeddings are aligned with text embeddings. The challenge for current monolithic VLMs actually lies in the lack of a holistic embedding module for both vision and language inputs. Therefore, HoVLE introduces a holistic embedding module that converts visual and textual inputs into a shared space, allowing LLMs to process images in the same way as texts. Furthermore, a multi-stage training strategy is carefully designed to empower the holistic embedding module. It is first trained to distill visual features from a pre-trained vision encoder and text embeddings from the LLM, enabling large-scale training with unpaired random images and text tokens. The whole model further undergoes next-token prediction on multi-modal data to align the embeddings. Finally, an instruction-tuning stage is incorporated. Our experiments show that HoVLE achieves performance close to leading compositional models on various benchmarks, outperforming previous monolithic models by a large margin. Model available at https://huggingface.co/OpenGVLab/HoVLE.

• The paper introduces a unified embedding module that merges vision and language, eliminating the need for separate modality-specific encoders.
• It employs a multi-stage training strategy—distillation, alignment, and instruction tuning—to enhance robust multimodal representation.
• The model outperforms existing monolithic VLMs by achieving a 15-point improvement on 17 benchmarks, demonstrating its significant efficiency.

HoVLE: Advancing Monolithic Vision-LLMs with Holistic Vision-Language Embedding

The paper presents "HoVLE," a monolithic vision-LLM designed to efficiently handle both visual and textual inputs by employing a Holistic Vision-Language Embedding (HoVLE). This model represents a departure from traditional compositional vision-LLMs (VLMs) which typically integrate pre-trained vision and language encoders separately, often leading to intricate architectures and the necessity of modality-specific processing paths.

Core Contributions and Model Architecture

HoVLE introduces a holistic embedding module that aligns both image and text inputs into a unified embedding space, allowing the model to leverage a LLM's (LLM) capabilities for interpreting visual data alongside textual data. This innovation is pivotal in overcoming the common limitations faced by existing monolithic VLMs, which often suffer from a degradation of language capabilities when adapting pre-trained LLMs for vision tasks.

The architecture of HoVLE circumvents the need for modality-specific encoders by:

1. Utilizing a shared embedding module that processes image patches and text tokens together, projecting them into a unified space.
2. Maintaining the language proficiency of tuned LLMs, thereby retaining strong textual understanding while extending visual capabilities through a shared space interpretation.

Training Strategy and Implementation

HoVLE employs a sophisticated multi-stage training strategy to imbue the holistic embedding module with robust vision and language encoding capacities. The training consists of:

• Distillation Stage: The model is initialized by distilling knowledge from pre-trained vision and LLMs, using a random combination of images and text tokens, which fosters general representation capabilities without requiring image-text pairs.
• Alignment Stage: This phase aligns the diverse modalities via auto-regressive training, ensuring cohesive vision-language understanding by leveraging multimodal data.
• Instruction Tuning: The final stage fine-tunes the model using multi-modal instruction data, enhancing its ability to follow diverse task instructions and improve performance across various benchmarks.

Performance Evaluation

HoVLE is evaluated against a wide spectrum of benchmarks, showcasing performance competitive with leading compositional VLMs across 17 multimodal tasks. It notably surpasses previous monolithic models significantly, evidencing the effectiveness of its holistic approach. On MMBench, a comprehensive multi-modal benchmark, HoVLE outperformed preceding models by approximately 15 points, solidifying its efficacy.

Implications and Future Directions

The introduction of HoVLE offers significant insights into the potential of monolithic VLMs. It demonstrates that simplifying the VLM architecture by removing modality-specific pathways does not inherently lead to performance compromises, provided that a robust holistic embedding is employed. This advancement suggests promising pathways for more unified model architectures in AI, which could facilitate more efficient deployment and broader application scopes.

Future developments may explore scaling HoVLE to leverage larger datasets and further refinements in embedding alignment strategies, potentially enhancing model utility in even more sophisticated vision-language tasks. Additionally, innovations in training techniques, focusing on minimizing computational demands while optimizing learning efficacy, could further advance the field of vision-language integration.