WavChat: A Survey of Spoken Dialogue Models

Abstract: Recent advancements in spoken dialogue models, exemplified by systems like GPT-4o, have captured significant attention in the speech domain. Compared to traditional three-tier cascaded spoken dialogue models that comprise speech recognition (ASR), LLMs, and text-to-speech (TTS), modern spoken dialogue models exhibit greater intelligence. These advanced spoken dialogue models not only comprehend audio, music, and other speech-related features, but also capture stylistic and timbral characteristics in speech. Moreover, they generate high-quality, multi-turn speech responses with low latency, enabling real-time interaction through simultaneous listening and speaking capability. Despite the progress in spoken dialogue systems, there is a lack of comprehensive surveys that systematically organize and analyze these systems and the underlying technologies. To address this, we have first compiled existing spoken dialogue systems in the chronological order and categorized them into the cascaded and end-to-end paradigms. We then provide an in-depth overview of the core technologies in spoken dialogue models, covering aspects such as speech representation, training paradigm, streaming, duplex, and interaction capabilities. Each section discusses the limitations of these technologies and outlines considerations for future research. Additionally, we present a thorough review of relevant datasets, evaluation metrics, and benchmarks from the perspectives of training and evaluating spoken dialogue systems. We hope this survey will contribute to advancing both academic research and industrial applications in the field of spoken dialogue systems. The related material is available at https://github.com/jishengpeng/WavChat.

• The paper provides a comprehensive survey on spoken dialogue models, detailing the evolution from traditional to modern architectures.
• The paper highlights multi-stage training processes, including modality alignment, supervised fine-tuning, and emerging reinforcement learning methods.
• The paper analyzes real-time interaction improvements through streaming and duplex communication techniques for effective human-computer dialogue.

WavChat: A Survey of Spoken Dialogue Models

Spoken dialogue systems have become a key interface for human-computer interaction, evolving significantly from early voice assistants to today's sophisticated systems. This survey, titled "WavChat: A Survey of Spoken Dialogue Models," examines the technological advancements in spoken dialogue models, particularly focusing on the dichotomy between cascaded and end-to-end paradigms. The paper categorizes these systems based on their ability to understand and generate speech without intermediary processes, and it outlines the state-of-the-art methodologies, core technologies, and future research directions in the domain.

Evolution of Spoken Dialogue Models

The history of spoken dialogue models encompasses a wide range of technological advances aimed at improving the natural interaction between humans and machines. The progression from traditional systems such as dGSLM and AudioGPT to more contemporary models like GPT-4o and Moshi highlights significant advancements in model intelligence and real-time interaction capabilities.

Figure 1: A timeline of existing spoken dialogue models in recent years.

Paradigms in Spoken Dialogue Systems

Spoken dialogue systems are primarily distinguished into two paradigms: cascaded and end-to-end models.

• Cascaded Models: These are characterized by a multi-step process involving ASR, LLMs, and TTS components. Although they benefit from leveraging LLMs for language processing, they encounter limitations in latency and interaction capability due to their complex architecture.
• End-to-End Models: These models aim for direct comprehension and generation of speech without dependency on text as a central intermediary. While they face challenges in aligning speech modalities with existing text-based frameworks, they offer improved latency and interaction simplicity.

  Figure 2: A general overview of current spoken dialogue systems, categorized into cascaded and end-to-end models.

Architectural Innovations

The paper discusses architectural paradigms, such as:

• Text Output Methods: Traditionally involve text-based LLMs extended to speech with minimal adaptation, but require external systems for speech synthesis.
• Chain-of-Modality Methods: These rely on generating text before speech, improving content quality at the expense of increased latency.
• Parallel Generation Methods: Offering reduced latency while maintaining quality by simultaneously generating text and speech tokens.

  Figure 3: Categorization Diagram of Spoken Dialogue Model Architectural Paradigms.

Multi-Stage Training Approaches

The training of spoken dialogue models is an intricate process involving:

• Modality Alignment: Adapting text models to understand and generate speech through multimodal training.
• Supervised Fine-Tuning: Utilizing instruction and dialogue datasets to improve conversational abilities.
• Reinforcement Learning: Though currently underexplored, it presents future potential for refining dialogue capabilities.

  Figure 4: Diagram of Multi-stage Training Steps.

Streaming and Duplex Communication

A significant focus is on enabling real-time, full-duplex dialogue capabilities:

• Streaming Models: These reduce latency through techniques like causal convolution and attention mechanisms, enhancing the fluidity of interactions.
• Duplex Systems: Allow simultaneous listening and speaking, mirroring realistic human interactions.

Figure 5: Simplex: One-way communication, and the direction is fixed.

Training Resources and Evaluation

The survey highlights the importance of comprehensive training datasets and evaluation benchmarks essential for advancing spoken dialogue models:

• Datasets: Encompass resources targeted towards specific tasks within the models, such as audio understanding and multimodal capabilities.
• Evaluation Methods: Address speech quality, intelligence, context learning, and interaction proficiency, with evolving benchmarks aiming to standardize assessments across diverse models.

Conclusion

The "WavChat: A Survey of Spoken Dialogue Models" provides a thorough exploration of current technologies, methodologies, and future research trajectories in spoken dialogue systems. The paper underscores ongoing challenges such as modality alignment and interaction fidelity while charting a path forward that leverages advanced model architectures and training techniques to improve human-computer interaction efficacy.