MusicGen-small Benchmark

Updated 28 January 2026

MusicGen-small benchmark is a systematic evaluation framework for text-conditional music generation models using both objective and subjective metrics.
It utilizes various datasets such as MusicBench, MusicCaps, and MTG-Jamendo to assess performance through metrics like FAD, KL divergence, and CLAP.
The benchmark underpins research into model compression, efficient deployment, and refining metric correlation with human musical judgments.

MusicGen-small Benchmark refers to the systematized evaluation of the “MusicGen-small” model—a Transformer-based, text-conditional music generation system—across several public, semi-public, and synthetic datasets using standardized objective and subjective metrics. MusicGen-small benchmarks underpin quantitative and qualitative comparisons both within the MusicGen family and with other state-of-the-art text-to-music (TTM) systems, serving as a reference point for model analysis, data and compute efficiency studies, and research into model compression and deployment.

1. Model Architecture and Training Regime

MusicGen-small has been implemented in several configurations, most notably as a single-stage autoregressive Transformer LM over discrete EnCodec token streams (Copet et al., 2023, Moschopoulos et al., 2024, Grötschla et al., 23 Jun 2025). The canonical “MusicGen-small” variant is characterized by:

Parameterization: 300–350 M (as published (Copet et al., 2023, Grötschla et al., 23 Jun 2025)), but in some compression analysis papers, a larger 557.6 M parameter variant is reported (Moschopoulos et al., 2024), comprising T5-base (109.6 M), a 24-layer, 16-head Transformer (419.6 M), and EnCodec neural audio codec (28.4 M).
Tokenization and Interleaving: Audio is encoded as 4–9 streams of RVQ tokens (e.g., $K=4$ codebooks, $M=2048$ per codebook) at a frame rate (e.g., 50 Hz at 32 kHz sample rate), and interleaved for sequential decoding via an efficient “delay” pattern (1-step codebook delay per stream) (Copet et al., 2023).
Conditioning: Utilizes a T5-base text encoder with classifier-free guidance (20%) and, optionally, unsupervised chromagram codes for melody conditioning.
Training: Models use up to 20 000 h of licensed music, with large batch sizes (e.g. 192, random 30 s crops), AdamW optimization, and mixed-precision/FlashAttention (Copet et al., 2023). Some public benchmarks restrict training to 457 h of CC-licensed data (MTG-Jamendo) to assess data efficiency (Lee et al., 21 Jan 2026).
Fine-tuning: When reported, MusicGen-small is either used “off-the-shelf” or fine-tuned on benchmark-specific data for improved vocal rendering and style adaptation (Moschopoulos et al., 2024).

2. Benchmark Datasets and Evaluation Protocols

MusicGen-small’s evaluation spans a spectrum of datasets and protocols:

MusicBench (Moschopoulos et al., 2024): Derived from MusicCaps with pitch/tempo/volume augmentation and enhanced captions. The test set (“Test A”) comprises 400 prompts (192 with vocals, 208 without). Evaluations utilize the AudioLDM-eval toolkit to extract VGGish (for FAD) and PANNs (for KL divergence).
MusicCaps (Copet et al., 2023): 5,500 ten-second prompts, with a genre-balanced subset of 1,000 samples for human evaluation. In-domain tests utilize 528 held-out tracks.
MusicEval (Liu et al., 18 Jan 2025): 2,748 clips from 31 models over 384 prompts spanning mood, structure, and instrumentation (pop/classical focused), rated by 14 experts on overall musical impression and text alignment.
MTG-Jamendo-Balanced (Grötschla et al., 23 Jun 2025, Lee et al., 21 Jan 2026): Prompts mined as triplets of tags (genre/instrument/mood), with public data for fully open-source benchmarking.
Other Human Preference Studies (Grötschla et al., 23 Jun 2025): 6,000 samples over 500 prompts, enabling statistically robust pairwise human comparison among 12–13 leading systems.

Evaluation is performed both on automatically derived metrics (section 3) and multi-stage human-listening protocols incorporating pairwise preference judgments and expert Likert-scale annotation.

3. Objective Metrics

MusicGen-small benchmarks rely on a suite of standardized metrics:

Fréchet Audio Distance (FAD):

$\text{FAD} = \|\mu_r - \mu_g\|_2^2 + \mathrm{Tr}\bigl(\Sigma_r + \Sigma_g - 2\,(\Sigma_r\,\Sigma_g)^{1/2}\bigr)$

where $\mu_r,\Sigma_r$ and $\mu_g,\Sigma_g$ are the mean and covariance of reference and generated embeddings, typically VGGish but also PANNs, CLAP, and EnCodec latents (Copet et al., 2023, Moschopoulos et al., 2024, Grötschla et al., 23 Jun 2025, Grötschla et al., 23 Jun 2025).

Kullback–Leibler Divergence (KL):

$\text{KL}(P\|Q) = \sum_i P(i)\,\log\frac{P(i)}{Q(i)}$

for feature-space histograms $P$ (reference) and $Q$ (generated)—often computed over PANNs classifier outputs (Copet et al., 2023, Moschopoulos et al., 2024).

CLAP Score: Cosine similarity between CLAP-encoded (text, audio) embeddings:

$\text{CLAP} = \cos(e_{text}, e_{audio})$

Higher CLAP indicates better text-audio alignment (Copet et al., 2023, Wang et al., 31 Aug 2025, Grötschla et al., 23 Jun 2025).

Perplexity (PPL): Log-perplexity on held-out codebook tokens, for linguistic modeling (Copet et al., 2023).
Composite Metrics: FAD-CLAP-MA (FAD on music-trained CLAP features) provides improved correlation with human judgment over standard FAD (Grötschla et al., 23 Jun 2025).
Human Metrics: Mean Opinion Scores (MOS) for overall quality and prompt relevance (typically 1–100 or 1–5 scales), and Bradley–Terry/Elo rankings derived from pairwise comparisons (Copet et al., 2023, Grötschla et al., 23 Jun 2025, Liu et al., 18 Jan 2025).

Benchmarks report statistical dispersion (e.g., mean ± std) and often include correlation (Pearson $r$ , Spearman $\rho$ ) between objective metrics and human preference to assess metric validity.

4. Performance Results across Benchmarks

MusicGen-small’s performance must be contextualized across dataset, model size, and reference system:

Model/Setup	Dataset/Task	FAD (↓)	KL (↓)	CLAP (↑)	Human Rank	Params	Reference
MusicGen-small (pretrained)	MusicBench (Test A)	$3.85\pm0.11$	$1.33\pm0.01$	–	–	557.6 M	(Moschopoulos et al., 2024)
MusicGen-small (fine-tuned)	MusicBench	$2.73\pm0.05$	$1.20\pm0.01$	–	–	557.6 M	(Moschopoulos et al., 2024)
TinyTTM (distilled)	MusicBench	$3.66\pm0.14$	$1.32\pm0.01$	–	–	89.2 M	(Moschopoulos et al., 2024)
MusicGen-small (300 M)	MusicCaps (10 s)	$3.1$	$1.28$	$0.31$	$78.4\pm1.3$ MOS	300 M	(Copet et al., 2023)
MusicGen-small (MTG-Jamendo)	Bench. Pref. Study	FAD-VGG $1.28$	FAD-PANN $0.95$	CLAP-MA $0.27$	$948.3$ Elo (11/12)	350 M	(Grötschla et al., 23 Jun 2025)
MusicGen-small	MusicEval	(run protocol)	(run protocol)	(run protocol)	(mean 5-point Likert)	300–350 M	(Liu et al., 18 Jan 2025)
MusicGen-small	TinyMusician Bench.	$6.49$ (FAD)	–	$0.303$	–	–	(Wang et al., 31 Aug 2025)

On MusicBench, MusicGen-small lags fine-tuned variants if not adapted to vocal-rich prompts, but its baseline performance is closely matched by knowledge-distilled and compressed students at <100 M parameters (Moschopoulos et al., 2024).
On canonical MusicCaps benchmarking, MusicGen-small significantly outperforms earlier systems such as Riffusion or Mousai, both in FAD and subjective ratings; scaling to larger parameter counts (1.5 B, 3.3 B) yields further—but diminishing—quality gains (Copet et al., 2023).
In human preference studies with rich, genre-diverse long-tail prompt sets, MusicGen-small occupies an intermediate rank, trailing behind recent diffusion and hybrid systems, with notable failure modes in jazz and ambient prompts (Grötschla et al., 23 Jun 2025).

5. Protocols for Small-Model and Data-Efficient Benchmarking

A significant research focus is reproducing or exceeding MusicGen-small’s benchmark quality with compressed or data-efficient models:

Knowledge Distillation: TinyTTM (Moschopoulos et al., 2024) and TinyMusician (Wang et al., 31 Aug 2025) directly distill MusicGen-small, incorporating bidirectional and skewed KL losses, layer reduction (LM: 24→7 or fewer), and aggressive encoder/decoder compression. Components such as T5-base are distilled/fine-tuned to span-MLM or further compressed to int8/float16 prec.
Quantization: Adaptive mixed-precision quantization in TinyMusician assigns int8 to text encoders, float16 to autoregressive decoders, float32 to codecs, balancing footprint, latency, and audio fidelity (Wang et al., 31 Aug 2025).
State-Space Model (SSM) Substitution: SSMs (Prefix SiMBA, Mamba-2) replace Transformer blocks at similar parameter budgets (≈300 M), achieving competitive FAD/KL/CLAP scores with 1/10–1/50th of data and compute (Lee et al., 21 Jan 2026).
Benchmarking Protocol: MusicEval (Liu et al., 18 Jan 2025) prescribes reproducible generation on 100–384 prompts, standardized audio formatting, expert Likert annotation (1–5, 5 raters per clip), calculation of CLAP-based automatic scores, and rigorous statistical reporting.

These protocols guarantee result comparability and facilitate the “slotting in” of new compressed or efficient models against the established MusicGen-small baseline.

6. Metric Validity and Human Alignment

Comprehensive studies highlight the variable correspondence between objective TTM metrics and human judgment:

Best Correlates: FAD-PANN and FAD-CLAP-Audio exhibit the strongest correlation with human musical preference ( $\rho\approx0.76$ and $0.58$; Pearson $r$ similar), but FAD on music-trained CLAP features (“FAD-CLAP-MA”) is the single best overall predictor for MusicGen-small ( $\rho\approx0.39$ ) (Grötschla et al., 23 Jun 2025).
Limitations: Genre-specific and long-range failure modes persist, notably in styles requiring macro-structural modeling (jazz, ambient), suggesting model capacity and representation bottlenecks (Grötschla et al., 23 Jun 2025).
Protocol Best Practices: Robust small-model benchmarking mandates combined usage of automated metrics and at least 1,000 human preference comparisons, open-source prompt/clip archives, and statistical reporting (Pearson $r$ , Spearman $\rho$ , p-values, boxplots, scatter comparisons) (Grötschla et al., 23 Jun 2025, Liu et al., 18 Jan 2025).

A plausible implication is that simple FAD scores are insufficient for nuanced comparison; system benchmarking should always include human studies and maximize metric diversity.

7. Practical Impact and Emerging Research Directions

The MusicGen-small benchmark fulfills four pivotal roles in TTM research:

Reference Standard: Provides a reproducible and well-characterized baseline for ablation, compression, and new architecture studies by constraining size, data, and evaluation setup.
Deployment Feasibility: Serves as a gateway for research into on-device and low-latency TTM, enabling the community to investigate trade-offs when model size and inference cost are tightly constrained (Moschopoulos et al., 2024, Wang et al., 31 Aug 2025).
Metric Analysis: Underpins the development of improved TTM metrics, including advances in music-trained CLAP models and multi-objective losses, directly tied to observed correlations with expert ratings (Grötschla et al., 23 Jun 2025, Liu et al., 18 Jan 2025).
Open-Source and Data Efficiency: Motivates training-efficient architectures (SSMs), low-compute pipelines, and the use of public data, expanding the accessibility of competitive TTM research (Lee et al., 21 Jan 2026).

Open questions include the integration of SSMs and hybrid attention architectures for full-model efficiency, optimal prompt and caption construction, and direct use of human-feedback loops in model training and evaluation.

Key References: