DITTO: Diffusion Inference-Time T-Optimization for Music Generation

Abstract: We propose Diffusion Inference-Time T-Optimization (DITTO), a general-purpose frame-work for controlling pre-trained text-to-music diffusion models at inference-time via optimizing initial noise latents. Our method can be used to optimize through any differentiable feature matching loss to achieve a target (stylized) output and leverages gradient checkpointing for memory efficiency. We demonstrate a surprisingly wide-range of applications for music generation including inpainting, outpainting, and looping as well as intensity, melody, and musical structure control - all without ever fine-tuning the underlying model. When we compare our approach against related training, guidance, and optimization-based methods, we find DITTO achieves state-of-the-art performance on nearly all tasks, including outperforming comparable approaches on controllability, audio quality, and computational efficiency, thus opening the door for high-quality, flexible, training-free control of diffusion models. Sound examples can be found at https://DITTO-Music.github.io/web/.

• The paper introduces a framework that leverages differentiable feature matching and latent optimization to precisely control text-to-music outputs without retraining.
• The method employs gradient checkpointing to mitigate memory overhead while achieving state-of-the-art controllability and computational efficiency in music generation tasks.
• The results demonstrate significant improvements in metrics such as Frechet Audio Distance and CLAP score, opening avenues for flexible, real-time music editing.

Advanced Inference-Time Control of Text-to-Music Diffusion Models with DITTO

This paper introduces DITTO, an innovative framework designed for controlling pre-trained text-to-music diffusion models at inference time through optimization of initial noise latents. The proposed methodology leverages differentiable feature matching losses to achieve targeted music outputs, extending the boundaries of music generation tasks such as inpainting, outpainting, and musical structure control.

Diffusion models have been a cornerstone in generative tasks across various domains, notably in text-to-image and text-to-audio transformations. However, much like their image and video counterparts, audio diffusion models have predominantly provided high-level control, leaving room for more nuanced and precise manipulation. Traditional approaches, such as the training-intensive ControlNet, rely on extensive datasets and pre-defined control signals, which bind the model to a fixed control setup post-training. DITTO diverges from these methods by offering training-free, fine-grained control through noise latent optimization, presenting a compelling alternative for music generation without modifying the underlying model parameters.

At the core of DITTO is the concept of optimizing the initial noise latents $\bm{x}_T$ through gradient checkpointing. The paper outlines a compelling use of gradient checkpointing to mitigate memory overhead, thus optimizing through the diffusion sampling process without requiring model fine-tuning. This technique enables the exploration of a variety of applications, ranging from intensity and melody control to novel tasks such as looping and musical structure control.

Numerical assessments underscore DITTO's superiority, indicating state-of-the-art performance across a spectrum of music generation tasks. The results show DITTO's enhanced control and its efficiency gains, achieving superior performance metrics over frameworks like MultiDiffusion and DOODL, particularly in controllability and computational efficiency. DITTO’s setup achieves robust generation results across various metrics, including Frechet Audio Distance (FAD) and CLAP score, when evaluated against conventional methods using the MusicCaps dataset.

A notable aspect of DITTO is its grounding in solid theoretical principles, substantiated by experiments that highlight the control semantic properties latent in the diffusion model's random initialization. These experiments provide insight into the expressive capacity of diffusion models, presenting an exciting direction for further research into the latent space's influence over low-frequency content intrinsic to music pieces.

Furthermore, the potential implications of DITTO are substantial. Practically, it facilitates more accessible and flexible music generation and editing, offering artists and creators finely-tuned control without extensive computational resources. Theoretically, DITTO paves the way for novel exploration into inference-time model manipulation.

In conclusion, DITTO presents a significant advancement in the field of diffusion-based music generation, providing a powerful, resource-efficient framework that bridges the gap between high-level control paradigms and the intricate, stylized demands of music creation. Future developments could explore real-time applications and broaden this framework’s adaptability to diverse control tasks in generative models.