LLark: A Multimodal Instruction-Following Language Model for Music

Abstract: Music has a unique and complex structure which is challenging for both expert humans and existing AI systems to understand, and presents unique challenges relative to other forms of audio. We present LLark, an instruction-tuned multimodal model for \emph{music} understanding. We detail our process for dataset creation, which involves augmenting the annotations of diverse open-source music datasets and converting them to a unified instruction-tuning format. We propose a multimodal architecture for LLark, integrating a pretrained generative model for music with a pretrained LLM. In evaluations on three types of tasks (music understanding, captioning, reasoning), we show that LLark matches or outperforms existing baselines in music understanding, and that humans show a high degree of agreement with its responses in captioning and reasoning tasks. LLark is trained entirely from open-source music data and models, and we make our training code available along with the release of this paper. Additional results and audio examples are at https://bit.ly/llark, and our source code is available at https://github.com/spotify-research/llark .

• The paper introduces LLark, a novel multimodal model combining a generative audio encoder and Llama 2 to enhance music comprehension.
• The paper presents an innovative data pipeline that augments music metadata into unified Query-Response pairs for instruction tuning across key musical tasks.
• The paper demonstrates competitive performance in tasks such as key identification and captioning, validated by human evaluations and ablation studies.

LLark: A Multimodal Instruction-Following LLM for Music

Introduction

The paper "LLark: A Multimodal Instruction-Following LLM for Music" introduces a novel multimodal model tailored for music comprehension. LLark integrates an innovative instruction-tuning dataset with a unique data augmentation strategy to facilitate musical understanding, captioning, and reasoning. It leverages a generative audio encoder combined with a LLM to execute a variety of complex musical tasks effectively.

Model Architecture

LLark's architecture involves three main components: the generative audio encoder (Jukebox), the LLM (Llama 2), and a projection module for aligning audio representations with the LLM. Jukebox, being a generative model, processes audio into rich, temporally-varying embeddings, which are subsequently transformed by a linear projection layer. These embeddings are integrated with Llama 2's language processing capabilities, enabling LLark to perform audio-text transformations effectively. The use of Jukebox as a generative model for audio embedding, rather than a typical contrastive model, represents a distinct choice that enhances LLark's ability to handle diverse musical attributes.

Figure 1: Overview of LLark. Given audio input and text instructions, LLark can answer a variety of queries, including music understanding, music captioning, and reasoning queries.

Data Pipeline

The data pipeline is pivotal for LLark’s performance. The authors introduce a method to create a comprehensive multimodal dataset by augmenting existing music datasets with metadata and generating Query-Response pairs using LLMs. The process converts diverse musical annotations into a unified instruction-tuning framework across three key task families: Music Understanding, Captioning, and Reasoning.

Figure 2: The core LLark data pipeline. Left: The metadata augmentation procedure. Right: Query-Response generation from augmented data via LLM for the three task families considered in this work.

Evaluation

LLark is evaluated on its ability to perform music understanding tasks such as key identification, tempo estimation, genre classification, and instrument identification. It achieves competitive performances that are close to state-of-the-art models fine-tuned on specific tasks. Moreover, LLark's capabilities in music captioning and reasoning tasks are assessed using both human and machine evaluations. It demonstrates superior performance when compared to existing multimodal models that are not specifically tuned for music.

Figure 3: Win rates of LLark vs. existing captioning models on test data.

In audio-text matching tasks for reasoning evaluation, LLark's outputs were preferred due to their enhanced musical detail, highlighting the effectiveness of the data augmentation strategy.

Figure 4: Audio-Text matching rates of non-expert human evaluators across 10 reasoning tasks.

Ablation Studies and Scaling

Ablation studies emphasize the importance of the specific components of LLark. Replacing the audio encoder or LLM results in diminished performance, particularly highlighting the crucial role of the generative audio encoder in capturing the temporal dynamics of music.

Figure 5: Ablation studies for the audio encoder.

A dataset scaling study further underscores the model's efficiency, showing diminishing returns beyond a certain scale, thereby validating the sufficiency of a diverse yet moderate-sized dataset.

Limitations and Future Work

Though LLark sets a new benchmark in multimodal music understanding, it is limited by the context window of its audio encoder and relies heavily on open-source data. The model's performance could potentially benefit from the inclusion of private music datasets and improved feature extraction techniques. Furthermore, the development of higher-quality evaluation benchmarks for music tasks is needed to facilitate more robust assessments of future models.

Conclusions

LLark demonstrates significant advancements in the domain of music AI, showcasing a model capable of understanding, describing, and reasoning about music with high precision and flexibility. Its unique combination of instruction tuning with generative audio processing paves the way for future research and application in diverse musical scenarios.