Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction

Abstract: Autoregressive next-token prediction with the Transformer decoder has become a de facto standard in LLMs, achieving remarkable success in NLP at scale. Extending this paradigm to audio poses unique challenges due to its inherently continuous nature. We research audio generation with a causal LLM (LM) without discrete tokens. We leverage token-wise diffusion to model the continuous distribution of the next continuous-valued token. Our approach delivers significant improvements over previous discrete solution, AudioGen, achieving 20% and 40% relative gains on AudioCaps in Frechet Audio Distance (FAD) and Kullback-Leibler (KL) divergence, respectively. Additionally, we propose a novel masked next-token prediction task that incorporates masked prediction into the causal LM framework. On AudioCaps, the innovation yields 41% and 33% relative FAD improvements over AudioGen Base (285M) and AudioGen Large (1B) models, respectively, and is on par with the state-of-the-art (SOTA) diffusion models. Furthermore, we achieve these results with significantly fewer parameters -- 193M for our Base and 462M for our Large models.