Music Deepfake Detection

• Music deepfake detection is the process of identifying whether a musical recording has been artificially generated or manipulated using AI.
• Research integrates diverse datasets, specialized benchmarks, and models like CNNs, GATs, and foundation models to tackle challenges from polyphonic and adversarial audio conditions.
• Evaluations focus on metrics such as Equal Error Rate and cross-domain generalization, emphasizing robustness, augmentation-aware training, and interpretability.

Music deepfake detection is the task of algorithmically determining whether a musical audio recording—either instrumental, full mix, or singing voice—has been synthesized, manipulated, or otherwise generated by artificial intelligence rather than recorded from a bona fide human performance. With the proliferation of generative music and singing-voice synthesis models, this problem now presents acute challenges for copyright enforcement, performer attribution, and auditory content integrity. Unlike speech deepfakes, musical deepfakes pose unique detection difficulties: strong background accompaniments, broad pitch ranges, complex rhythmic and harmonic structures, and often adversarial signal processing to evade forensics, all contribute to the need for specialized detection frameworks and benchmarks.

1. Distinctive Challenges and Task Formulation

Music deepfake detection departs fundamentally from speech deepfake detection. Synthesized singing voice appears within polyphonic, accompaniment-rich contexts, obscuring synthetic artefacts and introducing covariates absent in speech. Vocal synthesis covers broader pitch, intentional vibrato, and non-lexical voice gestures. Detection models must therefore discriminate not only between natural and synthetic timbral and prosodic patterns, but also contend with substantial confounds from backing tracks, genre-typical effects, and a wide diversity of generation methods (Zang et al., 2023, Zhang et al., 2024).

Formally, given an audio sample $x$ (e.g., a musical waveform or fixed-length segment), the task is to infer a binary label $y \in \{0,1\}$ (bona fide vs. deepfake). A scoring function $s(x) \in \mathbb{R}$ is trained so that high $s(x)$ values indicate deepfakes. The canonical evaluation metric is the Equal Error Rate (EER): the operating point $\tau^*$ where the False Acceptance Rate (FAR) equals the False Rejection Rate (FRR), i.e.,

$$\mathrm{EER} = \mathrm{FAR}(\tau^*) = \mathrm{FRR}(\tau^*),$$

with

$$\mathrm{FAR}(\tau) = \mathbb{P}[s(x)\ge\tau\mid y=1], \quad\mathrm{FRR}(\tau) = \mathbb{P}[s(x)<\tau\mid y=0].$$

This controls for threshold-dependent variance and balances both types of classification error (Phukan et al., 2024).

2. Datasets and Benchmark Frameworks

The availability of high-fidelity, annotated audio datasets is foundational for music deepfake detection research. The SingFake dataset establishes the first curated in-the-wild corpus for singing voice detection (28.93 h of bona fide singing, 29.40 h AI-generated, 40 singers, multi-lingual, with both source mixtures and separated vocals) (Zang et al., 2023). The SVDD Challenge 2024 extends this paradigm with controlled and wild tracks (CtrSVDD and WildSVDD), representing both studio-quality and social-media derived singing, using up to 14 synthesis/conversion models and diverse language, performer, and genre splits (Zhang et al., 2024). For general music (beyond vocals), the FakeMusicCaps dataset offers ∼10k ten-second clips from text-to-music platforms and human-controlled baselines, while FakeSound and FakeSound2 provide manipulation-localized, frame-wise annotated corpora for general audio, with direct applicability to instrument- or mix-level music detection (Sunday, 3 May 2025, Xie et al., 21 Sep 2025, Xie et al., 2024).

Benchmark protocols segment data by seen vs. unseen performer, generator model, genre, and codec, quantifying cross-domain generalization and real-world transfer (Zang et al., 2023, Zhang et al., 2024, Sroka et al., 7 Jul 2025). Both binary classification and fine-grained localization of manipulated regions are considered, with compositional metrics such as segment-level F1 and composite “Score” balancing identification and localization (Xie et al., 21 Sep 2025, Xie et al., 2024).

3. Model Architectures and Foundations

A variety of model families have been explored:

• Feature-Space CNNs and ResNets: Standard mel-spectrogram or linear/mel-frequency cepstral coefficients (MFCC, LFCC) with deep residual networks form a strong baseline for both general musical audio and singing detection (Zang et al., 2023, Sunday, 3 May 2025).
• Graph Attention Networks (GAT): Architectures such as AASIST leverage time–frequency relationships and integrated attention over spectro-temporal graphs (Zang et al., 2023). SingGraph extends this with spectral–temporal node fusion and max-graph operations for robust separation of singing and instrumental branches (Chen et al., 2024).
• Self-Supervised Large Audio Models: Foundation models pretrained for speech (Wav2vec2, x-vector, Whisper, Unispeech-SAT) and music (MERT, music2vec) provide transferable embeddings, with notable success for speaker recognition models (x-vector) in singing detection. Carefully designed fusion models such as FIONA synchronize complementary music and speech representations, achieving SOTA EERs (e.g., 13.74% on the SVDD CtrSVDD benchmark) (Phukan et al., 2024).
• Multibranch Feature Networks: MFAAN combines MFCC, LFCC, and Chroma-STFT in parallel CNN branches, fusing spectral, timbral, and pitch-class energy, with substantial performance increases over single-view baselines (e.g., 98.9% accuracy, 0.04% EER on “in-the-wild” audio) (Krishnan et al., 2023).

In musical deepfakes, backbone choices are substantially determined by the target region (e.g., isolated vocal vs. song mixture), the available augmentation strategies (RawBoost, beat-matching), and whether secondary modalities (e.g., lyrics, video) are available (Chen et al., 2024).

4. Evaluation, Robustness, and Limitations

Music deepfake detectors achieve near-perfect accuracy on test data when the distribution of generators and augmentations at training and test match (e.g., >99% accuracy, ROC AUC ≃ 0.999 on FMA with amplitude spectrograms) (Afchar et al., 2024). However, generalization beyond seen generators, under covariate shift, or when adversarial signal manipulations are applied remains problematic. Pitch shifting (±2 semitones), time-stretching, low-bitrate re-encoding (MP3/AAC), and simple white noise can reduce classifier accuracy to near chance, as shown for both neural (Transformer, ResNet) and classical feature pipelines (Sroka et al., 7 Jul 2025, Afchar et al., 2024, Sunday, 3 May 2025).

Generalization to unseen synthesis models is often poor; detectors trained solely on one generator “family” rarely transfer to another (e.g., Encodec→DAC; inter-family transfer near zero) (Afchar et al., 2024). For singing voice, even state-of-the-art speech countermeasures, when directly applied, show EERs of 45–58%—close to random guessing—until substantially retrained on in-domain music data (Zang et al., 2023).

Binary detection alone provides an incomplete assessment; explainability, manipulation localization, and source-type traceability are essential in operational settings. Benchmarks such as FakeSound2 extend evaluation to include traceability of manipulation types/sources, with models scoring F1_segment ≈ 97% in-domain but ≥20 pp lower in out-of-domain trials (Xie et al., 21 Sep 2025). Interpretability remains primitive: saliency mapping around suspect regions can highlight synthetic passages (Afchar et al., 2024).

5. Advanced Techniques: Foundation Model Fusion and Augmentation

Empirical evidence indicates that foundation models trained on speaker recognition (e.g., x-vector) outperform music-oriented self-supervised models (e.g., MERT) for singing voice deepfake detection, likely due to enhanced sensitivity to micro-intonation, pitch, and timbre variability—attributes strongly manipulated or lost in synthetic singing (Phukan et al., 2024). The FIONA fusion framework aligns and gates outputs from x-vector and MERT models, regularized with Centered Kernel Alignment (CKA) loss, substantially improving classifier calibration and EER over simple concatenation or individual models.

Augmentation-aware training is critical. Training on adversarially perturbed data (random pitch/time shifts, reverb, noise) increases detector robustness to real-world distribution shifts. For singing, domain-aware augmentations such as RawBoost (colored noise injection in vocals) and beat-matching (instrumental time-alignment) further improve detection under cross-codec and cross-language/genre splits, with relative EER reductions of up to 37% observed for hard conditions (Chen et al., 2024).

6. Open Problems and Research Outlook

Major open challenges persist:

• Domain Generalization: Current models overfit generator artifacts or codec idiosyncrasies, with poor transfer to new synthesis techniques, styles, or out-of-domain performance (Afchar et al., 2024, Xie et al., 21 Sep 2025).
• Adversarial Robustness: Simple signal manipulations can catastrophically erase classifier confidence. Robust, augmentation-driven training and domain-invariant objective formulations are required (Sroka et al., 7 Jul 2025, Li et al., 2024).
• Localization and Attribution: Binary classification is insufficient for human-in-the-loop workflows; methods must localize and explain detected manipulations, enabling content audit and forensic recourse (Xie et al., 21 Sep 2025, Afchar et al., 2024).
• Benchmarking and Dataset Availability: Broader, more diverse music deepfake corpora are needed—spanning genres, instrumental/vocal compositions, generation paradigms, and associated metadata for multimodal fusion (Li et al., 2024).

Recommended research directions include:

• Multi-task and one-class anomaly modeling for generalization to open-set attacks,
• Multimodal fusion with lyrics, video, and symbolic score for semantic consistency (Li et al., 2024),
• Structured augmentation and adversarial training to induce feature invariance,
• Development of explainable frameworks supporting forensic and legal auditability (Xie et al., 21 Sep 2025, Afchar et al., 2024).

7. Summary Table: Representative Datasets and Methods

Resource	Content & Split	Notable Model/Result
SingFake (Zang et al., 2023)	40 singers, 58h (real+fake), 5+ langs	Speech CMs: EER >45%; retrained: EER 5–23%
SVDD (Ctr/Wild) (Zhang et al., 2024)	164 singers, 14 gen. methods, wild in-the-field splits	Baselines: RawWave+GAT EER ≈10%
FakeMusicCaps (Sunday, 3 May 2025)	10,746 clips, TTM platforms + human	ResNet-18: 88% accuracy, 84% F1
FakeSound2 (Xie et al., 21 Sep 2025)	6 manip types ×12 sources × >300k ex.	EAT–ResNet–LSTM: F1 97% (in-dom), 79% (OOD)
FIONA (Phukan et al., 2024)	47.6h real/260.3h fake (CtrSVDD)	x-vector+MERT: SOTA EER 13.74%
SingGraph (Chen et al., 2024)	SingFake splits (solo, codecs, lang)	SOTA EER: 4.01% (seen), 6.30% (codecs)

These resources collectively define the state of the art and the reference point for ongoing research in music deepfake detection.