MM-Sonate: Multimodal Flow-Matching Transformer

• MM-Sonate is a multimodal flow-matching transformer framework that jointly synthesizes identity-preserving speech and high-fidelity, temporally synchronized videos.
• It integrates video and audio instructions with explicit phoneme timings using an MM-DiT backbone and a flow-matching objective to enhance lip sync and intelligibility.
• The framework employs an efficient timbre injection and noise-based negative conditioning approach to enable robust zero-shot voice cloning across diverse content.

MM-Sonate denotes a multimodal flow-matching transformer framework for controllable, temporally aligned audio–video generation with integrated zero-shot voice cloning. It is designed to jointly synthesize identity-preserving speech and high-fidelity video, establishing state-of-the-art results in speech intelligibility, lip synchronization, and cross-modal alignment. MM-Sonate unifies conditioning on video instructions, audio instructions, and fine-grained, explicitly timed phoneme sequences, and incorporates a parameter-efficient timbre injection mechanism to enable zero-shot speaker cloning within a single, unified audio–video generation model (Qiang et al., 4 Jan 2026).

1. Model Architecture and Input Encoding

MM-Sonate is built on a multimodal diffusion transformer ("MM-DiT") backbone. The input representation comprises three components:

• Video Instruction: A scene-level description specifying desired visual content.
• Audio Instruction: High-level attributes (e.g., “female voice, happy tone”) for the desired audio domain.
• Phoneme Sequence: Obtained by Grapheme-to-Phoneme (G2P) conversion, enforcing precise linguistic and temporal alignment.

All textual conditioning (instructions, phonemes) is embedded via a shared text encoder (Qwen2.5-7B for instructions; a dedicated 512-dimensional encoder for phonemes). In dialogues, utterances are demarcated with [S0], [S1], and phoneme streams are concatenated.

Audio and video signals are mapped into continuous latent spaces using pre-trained VAEs: Mel-VAE at 43 Hz for audio and a 3D causal VAE at approximately 3 Hz for video. The MM-DiT backbone comprises 32 layers of cross-/self-attention blocks (feedforward dimension 4096), yielding a shared multisensory latent space. At each diffusion step $t$, the network processes: text conditioning (instruction/phoneme embeddings), noised audio latents, and noised video latents. Modality fusion and temporal dependencies are handled by multi-head cross-modal attention.

2. Flow-Matching Objective and Generation Technique

The denoising network $\mathbf{s}_t(\mathbf{z}_t; t, \mathbf{c})$ is trained under a flow-matching objective to model an optimal transport path from Gaussian noise to the data distribution in the joint latent space:

$$\mathcal{L}_{\mathrm{flow}} = \mathbb{E}_{t,\,\mathbf{x}_0,\,\mathbf{z}_t} \Big\| \mathbf{s}_t(\mathbf{z}_t; t, \mathbf{c}) - \nabla_{\mathbf{z}_t}\log p_t(\mathbf{z}_t \mid \mathbf{x}_0) \Big\|^2$$

where $$\mathbf{z}_t = t\,\mathbf{x}_1 + (1-t)\,\mathbf{x}_0$$ interpolates between noisy inputs $\mathbf{x}_0$ and clean latents $\mathbf{x}_1$.

MM-Sonate introduces a noise-based negative conditioning scheme for classifier-free guidance (CFG). The unconditional branch, instead of simply suppressing conditions, replaces the speaker embedding $S$ with a negative embedding $S_{\mathrm{neg}} = f_{\mathrm{spk}}(n)$, derived by encoding natural white noise. The final guided score interpolates between conditional and (noise-based) unconditional branches:

$$\tilde{\epsilon} = \epsilon_{\mathrm{uncond}} + w\,(\epsilon_{\mathrm{cond}} - \epsilon_{\mathrm{uncond}})$$

Empirically, this approach regularizes acoustic fidelity and enhances the trade-off between speech quality and lip sync, outperforming Gaussian-noise-based alternatives.

3. Timbre Injection and Zero-Shot Voice Cloning

A pre-trained speaker encoder $f_{\mathrm{spk}}$ obtains a global utterance-level timbre embedding $S \in \mathbb{R}^{B \times D}$, which is upsampled along the time axis to match phoneme feature sequences $P \in \mathbb{R}^{B \times L \times D}$ and injected additively:

$P' = P + \mathrm{Upsample}(S)$

For multi-speaker scenarios, individual speaker embeddings $S_i$ and learned ID embeddings $I_i$ are combined per sequence:

$P_i = P_i + S_i + I_i,\quad i \in \{0, 1\}$

When reference audio is omitted (text-to-video-audio, T2VA; text-image-to-video-audio, TI2VA settings), $S$ is set to zero, defaulting to generic generation modes. This mechanism enables near-instantaneous, high-fidelity zero-shot voice cloning, preserving identity while maintaining precise phoneme-level synchronization.

4. Training Procedure and Data Strategy

MM-Sonate undergoes unified multi-task training over 100 million video–audio–caption–transcript examples covering speech, music, and ambient sounds.

Key training procedures:

• Data Preprocessing: Raw speech/music is denoised, dry vocals extracted, and a synthetic neutral-style speech set is synthesized via DMP-TTS. Speaker verification by WavLM ensures identity-pure samples.
• Stochastic Modality Masking: Randomly masks out inputs during training (audio/image/text), simulating all conditional generation modalities (T2VA, TI2VA, TA2VA, TIA2VA) and approximating the conditional-unconditional gap for CFG.
• Optimization: Sole loss is $\mathcal{L}_{\mathrm{flow}}$. No adversarial or explicit alignment losses are used, as strict lip synchronization arises solely from phoneme-level conditioning and unified modeling. Optimization uses Adam, weight decay, and cosine learning-rate schedules across $\sim$10^8$ steps.

5. Empirical Performance and Benchmark Results

MM-Sonate demonstrates superior performance on established benchmarks for audio–video synthesis, lip sync, and voice cloning.

Task/Metric	MM-Sonate	Best Baseline	Note
Audio Fréchet Distance (FD)	1.43	1.50	Lower is better
Audio KL Divergence	1.16	1.19	Lower is better
Speech Intelligibility (WER)	0.020	0.035 (Ovi)	Lower is better
Lip Sync (SyncNet Conf.)	6.51	4.28	Higher is better
Speaker Similarity (SIM-o EN)	0.604	0.599 (M3-TTS)	On par
Speaker Similarity (SIM-o ZH)	0.691	N/A	-
Pass Rate (single-spk, EN)	85.0%	N/A	At cloning

Additional findings:

• Natural-noise negative conditioning yields WER 2.06% vs 3.17% for Gaussian baseline, with no appreciable loss of lip-sync.
• Speaker cloning passes at ZH 91.3%, EN 85.0%; multi-speaker dialogue cloning at ZH 81.3%, EN 75.0%.
• Qualitative analysis reveals strict phoneme–viseme alignment, cross-species voice cloning, and frame-accurate musical and sound effect generation.

6. Strengths, Limitations, and Ethical Considerations

MM-Sonate's strengths include being the first unified model to support zero-shot voice cloning within audio–video generation; achieving state-of-the-art lip synchronization and intelligibility; and high versatility, covering speech, singing, dialogues, and sound effects.

Limitations:

• Clip duration is limited to 3–15 seconds due to context modeling constraints.
• Occasional loss of fine video texture or audio high-frequency details under strong compression.
• Lip–audio desynchronization occurs in rare, extreme head poses or occlusion scenarios.
• Voice cloning quality degrades with poor reference audio.

Recognized ethical issues include deepfake and impersonation risks; all outputs are watermarked, and access is managed through controlled APIs with input/output safety classifiers. There is ongoing work on consent verification for reference voices and public-figure usage restrictions. These mitigations are necessary for responsible deployment (Qiang et al., 4 Jan 2026).

7. Comparative Context and Impact

MM-Sonate represents a significant convergence in multisensory content generation, addressing prior limitations such as the inability to perform joint zero-shot voice cloning and semantic misalignment in unified models. By linking phoneme-anchored linguistic control with identity-specific timbre, MM-Sonate avoids the temporal misalignment typical of cascaded (audio-first, video-second) approaches and matches the fidelity of specialized text-to-speech models, even under the harder multimodal generation task.

Its benchmarks indicate superiority in both standard metrics and new evaluation regimes (e.g., VerseBench for AV alignment, SongEval for music), establishing new baselines for future unified audio–video synthesis research. The methodology demonstrates that a single conditional generative model can robustly handle fine-grained, identity-aware, and temporally synchronized multimodal content in zero-shot scenarios (Qiang et al., 4 Jan 2026).