Vision-Speech Models: Teaching Speech Models to Converse about Images

Abstract: The recent successes of Vision-LLMs raise the question of how to equivalently imbue a pretrained speech model with vision understanding, an important milestone towards building a multimodal speech model able to freely converse about images. Building such a conversational Vision-Speech model brings its unique challenges: (i) paired image-speech datasets are much scarcer than their image-text counterparts, (ii) ensuring real-time latency at inference is crucial thus bringing compute and memory constraints, and (iii) the model should preserve prosodic features (e.g., speaker tone) which cannot be inferred from text alone. In this work, we introduce MoshiVis, augmenting a recent dialogue speech LLM, Moshi, with visual inputs through lightweight adaptation modules. An additional dynamic gating mechanism enables the model to more easily switch between the visual inputs and unrelated conversation topics. To reduce training costs, we design a simple one-stage, parameter-efficient fine-tuning pipeline in which we leverage a mixture of image-text (i.e., "speechless") and image-speech samples. We evaluate the model on downstream visual understanding tasks with both audio and text prompts, and report qualitative samples of interactions with MoshiVis. Our inference code will be made available, as well as the image-speech data used for audio evaluation.

• The paper introduces MoshiVis, a Vision-Speech Model that successfully integrates visual understanding into a speech model to enable real-time conversation about images.
• MoshiVis addresses data scarcity for Vision-Speech Models using a mixed training strategy combining image-text and image-speech data, alongside efficient architectural adaptations.
• The model demonstrates effective text-to-audio transfer for visual tasks and maintains real-time capability, offering a foundation for scalable, interactive multimodal AI systems.

Vision-Speech Models: Enhancing Conversational Speech Models with Visual Understanding

The integration of multimodal capabilities into large-scale models has seen significant advancements in recent years, driven by the success of Vision-LLMs (VLMs). This paper discusses the exploration of extending these capabilities to include speech, presenting a comprehensive methodology for augmenting pre-trained speech models with visual inputs to create a Vision-Speech Model (VSM). Specifically, the paper introduces MoshiVis, a model designed to converse about images in real-time, built upon an existing speech LLM, Moshi.

Methodology

To achieve the goal of a fully capable VSM, three primary challenges are addressed:

1. Data Scarcity: VLMs usually benefit from abundant image-text data. However, image-speech datasets are less common and often limited. The authors tackle this through a mixed data training strategy that incorporates speechless data (image-text pairs without audio) with a smaller proportion of image-speech data. This strategy leverages existing VLM resources while maintaining alignment with the speech modality.
2. Real-time Inference: The requirement for real-time interaction necessitates optimized compute and memory efficiency. MoshiVis uses cross-attention based adaptation modules that integrate image tokens into the speech LLM. These modules are designed to be lightweight and use cached computations for efficiency.
3. Preserving Speech Features: Maintaining the conversational abilities of the base speech model, including prosodic features such as tone, is crucial. The model employs a dynamic gating mechanism within the cross-attention modules to allow selective inclusion of visual inputs, thereby supporting seamless transitions between image discussions and unrelated conversation topics.

Training and Evaluation

MoshiVis is trained using a one-stage fine-tuning pipeline that capitalizes on a mix of image-text and synthetically-generated image-speech data. For evaluation, the model is tested on tasks such as image captioning and visual question answering, using both text and audio prompts. The results demonstrate that even with minimal audio data, the model can effectively engage in visual tasks, indicating strong transfer of information from text to speech modalities. Furthermore, qualitative assessments highlight the model's ability to carry out contextually rich dialogues about images while maintaining low latency.

Key Findings

1. Text-to-Audio Transfer: Even with predominantly text-based supervision during training, the MoshiVis model exhibits substantial visual understanding when evaluated with audio prompts. The transfer is particularly effective in scenarios with balanced text and audio training data, achieving competitive results on benchmarks.
2. Contextual Gating: The gating mechanism shows promise in managing the integration of visual information, aiding the model’s capacity to discern when to focus on visual input versus maintaining a general conversational context.
3. Real-time Capability: The model’s design is successful in retaining real-time interaction capability, with only a slight increase in latency compared to the unaugmented speech model, proving the feasibility of integrating vision and speech without compromising performance.

Implications and Future Directions

The introduction of MoshiVis is a notable step towards comprehensive multimodal models capable of seamless interaction across visual, textual, and auditory domains. Its development and the strategies employed suggest several implications for the future of AI and its applications:

• Scalability and Accessibility: By utilizing existing resources efficiently, this approach can be adapted to other models and modalities, potentially expanding the reach of AI technologies in accessible formats.
• Enhanced Multimodal Interaction: The continuous refinement of such models could lead to more natural and intuitive human-computer interactions, beneficial in areas like virtual assistants, education, and customer service.
• New Research Opportunities: The challenges overcome in this work open avenues for further research into the dynamic management of multiple data streams and real-time processing in large-scale models.

In conclusion, the paper presents a thorough examination of the integration of visual understanding into speech models, providing a foundation for future exploration in the multimodal AI domain. The methodologies and findings offer insights into developing efficient, scalable, and interactive systems that adhere to the increasing demand for more capable and adaptable artificial intelligence models.