Integrating Facial Generation into Full-Duplex Spoken Dialogue Systems

Abstract: Full-duplex spoken dialogue models, such as Moshi, enable natural, low-latency voice conversations. However, they remain limited to the audio modality, lacking the facial expressions that are integral to human communication. We present Moshi-Face, the first full-duplex dialogue model that jointly processes the user's audio and facial input while simultaneously generating speech and facial motion. We first construct a vector-quantized variational autoencoder (VQ-VAE) as a face codec that encodes 3D head meshes extracted from facial videos into compact discrete tokens, referred to as face tokens, and conversely reconstructs 3D meshes from these tokens. We then extend Moshi with a Face Transformer module that generates face tokens non-autoregressively, enabling Moshi-Face to produce synchronized audio and face tokens in real time. Experiments show that Moshi-Face achieves audiovisual alignment at low latency while preserving the dialogue quality of the original audio-only model.

• The paper introduces Moshi-Face, a novel system that seamlessly integrates 3D facial generation into full-duplex spoken dialogue.
• It employs a VQ-VAE based face codec and a non-causal Face Transformer to simultaneously process and generate synchronized audio and facial tokens.
• Experimental results demonstrate strong audiovisual alignment with low LSE-D and high coherence, highlighting its potential for natural multimodal human-computer interaction.

Integrating Synchronized Facial Generation into Full-Duplex Spoken Dialogue: The Moshi-Face Framework

Motivation and Background

Natural human dialogue is inherently bidirectional and multimodal, relying on both verbal and rich nonverbal cues—such as facial expressions, head movements, and lip synchronization—to convey intent and achieve communicative nuance. While prior multimodal dialogue systems have incorporated facial synthesis, they have been predominantly restricted to turn-based architectures, precluding natural simultaneity and real-time back-channeling ([ji2024wavchat], [arora2025landscape]). Recent full-duplex dialogue frameworks, such as Moshi ([defossez2024moshi]), enable real-time, streaming bidirectional voice interaction but have thus far been confined to the audio modality alone. This work addresses a fundamental gap: the lack of systems that can process and generate both auditory and visually realistic facial signals in full-duplex conversation.

System Architecture: Moshi-Face

Moshi-Face extends Moshi to enable real-time joint processing and generation of 3D facial motion in addition to audio. The core components are:

1. Face Codec: A VQ-VAE quantization pipeline encodes 3D head meshes (FLAME topology) into compact discrete face tokens and enables efficient reconstruction. The codec temporally aligns with Moshi’s audio tokens (12.5 Hz) by downsampling 25 fps meshes, using $N=8$ face token streams per speaker for fine-grained control. Optimization combines L1 reconstruction, quantization, and velocity losses.
2. Face Transformer: A non-causal transformer module generates $N$ face tokens synchronously per timestep, conditioned on the current state of the main RQ-Transformer, the spoken text, and audio embeddings. Unlike autoregressive decoders, all $N$ tokens are predicted in parallel, reflecting their independence after quantization and promoting streamability. The Face Transformer leverages both system and user multimodal context with position-aware queries.
3. RQ-Transformer: Retains Moshi’s large Language Modeling backbone (7B parameters) for text and audio generation. Outputs at each timestep feed into the Face Transformer, ensuring audiovisual synchronization across modalities.

   Figure 1: Moshi-Face architecture—VQ-VAE face codec (a), multimodal token streams (b), and non-causal Face Transformer generation (c).

Dataset and Training Protocol

Moshi-Face required large-scale multimodal conversational data aligned at the frame level. The construction process involved:

• Corpus Selection: A subset (180 hours, ~3,400 dialogues) of the Seamless Interaction dataset ([agrawal2025seamless]) was used, providing time-aligned audio, transcripts, and video. High-fidelity 3D meshes were extracted per frame using the VHAP pipeline ([qian2024gaussianavatars]) at 25 fps with FLAME topology (5,143 vertices/frame).
• Tokenization and Codec Pretraining: All text, audio, and facial motion were tokenized with Moshi and the new face codec. The face codec was pretrained with 70 hours of mesh data, comparing codebook sizes and embedding dimensionalities to balance reconstruction error and codebook utilization.
• Two-Stage Model Training: The RQ-Transformer was initialized from a pre-trained Moshi checkpoint. Initially, only the Face Transformer was trained (with RQ-Transformer frozen), then all modules were jointly fine-tuned to optimize the composite loss across text, audio, and face modalities.

Experimental Evaluation

Face Codec Quality

Evaluation considered Mean Vertex Error (MVE), Lip Vertex Error (LVE), and normalized codebook Perplexity. Increasing codebook size from 128 to 256 improved both MVE (down to approximately $9.9 \times 10^{-3}$) and LVE (down to $11.77 \times 10^{-3}$), while maintaining high codebook utilization ($0.66$ perplexity with $K$=256, $C$=128). These results indicate high-fidelity reconstruction and prevention of codebook collapse.

Multimodal Generation Performance

Moshi-Face was benchmarked against four baselines:

• Moshi (audio-only, unmodified)
• Moshi-ft (fine-tuned audio-only on new dataset)
• Reconstructed face (ground-truth reconstruction, upper-bound)
• Random face (random face tokens, lower-bound).

Key metrics:

• Audiovisual Synchronization: Measured by LSE-D (lower is better) and LSE-C (higher is better) using SyncNet ([chung2016out], [prajwal2020lip]). Under teacher-forced input, Moshi-Face achieved LSE-D = 8.76 (close to reconstructed upper-bound: 8.53) and LSE-C = 0.14. Autoregressive free-run settings maintained alignment (LSE-D = 11.0).
• Speech Naturalness and Semantic Quality: UTMOS ([saeki2022utmos]) judged mean opinion scores for speech quality, and LLM-as-a-Judge ([zheng2023judging], [abe2026effects]) provided language understanding and generation scores for coherence, naturalness, relevance, and overall dialogue quality. While raw speech naturalness (UTMOS) for Moshi-Face was modest (1.75 vs. Moshi’s 3.08), coherence (3.79) was superior to both the original and fine-tuned baselines, and naturalness/relevance were competitive.

  Figure 2: Example Moshi-Face dialogue output: the model generates facial motion with both lip synchronization and natural head movement, aligned with speech.

Ablation Studies

• Face Transformer Pretraining: Omitting Stage 1 increased LSE-D to 9.53 (teacher-forced), confirming the advantage of initializer stages for stable learning.
• No Full Fine-Tuning: Only training Face Transformer sharply increased LSE-D (11.8) and degraded LLMAJ scores, indicating that full-system joint training is necessary for effective multimodal generation.
• Removal of $t-1$ face token input: Improved some metrics (LSE-D) but worsened UTMOS and LSE-C, suggesting temporal dependencies in face token streams have a regularizing effect.

Implications and Future Directions

Moshi-Face demonstrates the technical feasibility of tightly synchronized, real-time audiovisual generation in a full-duplex dialogue system. By elevating the interaction from audio-only to multimodal, user experience is expected to more closely mirror face-to-face human communication, with potential benefits for engagement and comprehension in conversational AI applications. The model’s factorized structure allows for effective adaptation to additional modalities, such as gaze, gesture, or emotional state.

The authors propose transitioning from the current non-causal VQ-VAE face codec to a low-latency, causal streaming codec, which is critical for robust deployment in latency-sensitive applications. Further, systematic human evaluations targeting perceptual realism and psychological impact are needed. Future research may also investigate transferability across domains, low-resource adaptation, multi-party extensions, and integration with advanced agentic planning modules.

Conclusion

This work presents Moshi-Face, a first-of-its-kind full-duplex dialogue model capable of real-time, synchronized generation and processing of speech and 3D facial motion. Through a scalable, discrete-token VQ-VAE codec and an efficient non-causal Face Transformer, Moshi-Face achieves strong audiovisual alignment while maintaining dialogue quality. This system sets a new standard for naturalistic, multimodal AI conversational agents and serves as a foundation for future research in multimodal, low-latency human-computer interaction.