Precise spatio-temporal alignment in joint audio–video generation

Establish methods that achieve precise spatio-temporal alignment between generated audio and video in joint text-to-audio-video generation systems, ensuring tightly synchronized audiovisual outputs during speech and broader soundscape synthesis.

Background

The paper surveys recent progress in joint audio–video generation across commercial and open-source systems and notes that, despite architectural advances (e.g., dual-stream and single-tower models), achieving tight synchronization between modalities is still not fully resolved.

Within this context, the authors explicitly identify precise spatio-temporal alignment as an open challenge, pointing to the underexplored nature of synchronized speech-video synthesis and complete soundscapes, which motivates their dual-stream MMDiT design with bidirectional cross-attention and RoPE scaling.