Qwen2-Audio Technical Report

Abstract: We introduce the latest progress of Qwen-Audio, a large-scale audio-LLM called Qwen2-Audio, which is capable of accepting various audio signal inputs and performing audio analysis or direct textual responses with regard to speech instructions. In contrast to complex hierarchical tags, we have simplified the pre-training process by utilizing natural language prompts for different data and tasks, and have further expanded the data volume. We have boosted the instruction-following capability of Qwen2-Audio and implemented two distinct audio interaction modes for voice chat and audio analysis. In the voice chat mode, users can freely engage in voice interactions with Qwen2-Audio without text input. In the audio analysis mode, users could provide audio and text instructions for analysis during the interaction. Note that we do not use any system prompts to switch between voice chat and audio analysis modes. Qwen2-Audio is capable of intelligently comprehending the content within audio and following voice commands to respond appropriately. For instance, in an audio segment that simultaneously contains sounds, multi-speaker conversations, and a voice command, Qwen2-Audio can directly understand the command and provide an interpretation and response to the audio. Additionally, DPO has optimized the model's performance in terms of factuality and adherence to desired behavior. According to the evaluation results from AIR-Bench, Qwen2-Audio outperformed previous SOTAs, such as Gemini-1.5-pro, in tests focused on audio-centric instruction-following capabilities. Qwen2-Audio is open-sourced with the aim of fostering the advancement of the multi-modal language community.

• The paper introduces Qwen2-Audio, a large-scale audio-language model that combines an audio encoder based on Whisper-large-v3 with the Qwen-7B language framework for unified audio-text interaction.
• Its training methodology replaces complex hierarchical tags with natural language prompts and employs supervised fine-tuning and direct preference optimization to boost instruction-following ability.
• Evaluation demonstrates state-of-the-art performance with significant WER improvements (1.6% test-clean, 3.6% test-other) and advances in speech translation, emotion recognition, and vocal sound classification.

Technical Overview of Qwen2-Audio: A Large-Scale Audio-LLM

The paper introduces Qwen2-Audio, a sophisticated large-scale audio-LLM developed for audio analysis and interaction. The model integrates advanced capabilities for processing varied audio inputs, simplifying pre-training processes, and optimizing model outputs for enhanced instruction-following ability. The research focuses on scaling the instruction-following abilities of Qwen2-Audio without relying on complex hierarchical tags, thereby simplifying the pre-training phase with natural language prompts and expanding the dataset size for more comprehensive learning.

Model Design and Training Methodology

Qwen2-Audio's architecture is composed of an audio encoder and a LLM. The audio encoder is based on the Whisper-large-v3 model, effectively pre-processing audio data by converting raw waveforms into mel-spectrograms. The LLM is built upon the Qwen-7B framework, culminating in an overall model size of 8.2 billion parameters. The training methodology unfolds across three stages—pre-training with natural language prompts, supervised fine-tuning via instruction-based techniques, and Direct Preference Optimization (DPO) to align model behavior with human preferences.

During pre-training, the reliance on hierarchical tags was replaced with language prompts, which improved generalization capability. The model accommodates two interaction modes—Audio Analysis for offline audio examination and Voice Chat for real-time interaction. These modes operate seamlessly without explicit user switching, enabling Qwen2-Audio to interpret both audio and text inputs simultaneously.

Evaluation and Performance Metrics

The paper details an extensive evaluation regimen, illustrating Qwen2-Audio's robust performance across several datasets and tasks, including Automatic Speech Recognition (ASR), Speech-to-Text Translation (S2TT), Speech Emotion Recognition (SER), and Vocal Sound Classification (VSC). Using metrics such as Word Error Rate (WER) and BLEU scores, Qwen2-Audio consistently outperformed previous models, establishing new benchmarks in understanding and interpreting diverse audio signals.

Key results highlighted include WER improvements to 1.6% and 3.6% on the Librispeech test-clean and test-other datasets, respectively, and surpassing benchmarks in the Speech-to-Text Translation task across multiple language pairs. The model demonstrated substantial gains in tasks like SER and VSC, with notable improvements over existing counterparts, further solidifying its SOTA capabilities when evaluated through objective metrics from the AIR-Bench.

Implications and Future Directions

Qwen2-Audio's advancements signify a substantial leap in multi-modal language modeling, targeting more natural and expansive audio interactions without the limitations of traditional tagging systems. The model's proficiency presents potential applications across multimedia analysis, intelligent voice assistants, and automated audio transcription systems, offering pathways for enhanced human-computer interaction.

The paper implicitly suggests directions for future exploration, notably in further scaling model parameters and dataset sizes and integrating broader language understanding capabilities to support even more complex audio-visual task scenarios. The open-source nature of Qwen2-Audio invites contributions from the community, potentially accelerating innovations in the multi-modal and audio analysis domains of AI research. As the AI field progresses towards more holistic integrations of language and audio interactions, models like Qwen2-Audio set the foundational benchmarks for subsequent advancements.