Supervised Music Deepfake Detectors

• Supervised music deepfake detectors are systems trained on paired authentic and synthetic audio to distinguish real from AI-manipulated music.
• They utilize advanced feature extraction methods and various neural architectures, including CNNs, Transformers, and graph-based models, for accurate detection.
• Empirical evaluations using metrics like EER and accuracy highlight robustness challenges against augmentations, codec shifts, and out-of-distribution samples.

Supervised music deepfake detectors are machine learning systems, typically trained with labeled pairs of real and synthetic (deepfake) audio, that are designed to distinguish between authentic musical content and content generated or manipulated by artificial intelligence. These detectors are critical for mitigating the risks associated with unauthorized artistic replication, copyright infringement, and the erosion of trust in music distribution ecosystems. The following sections synthesize state-of-the-art architectures, training paradigms, empirical findings, vulnerabilities, and research directions characterizing this rapidly evolving research field.

1. Benchmarks and Data Regimes

Supervised detector development hinges on carefully curated datasets containing paired authentic and synthetic musical recordings, as well as rigorous test splits probing generalization.

Singing-Voice Benchmarks:

The SingFake benchmark (Zang et al., 2023) is the leading resource for singing-voice deepfake detection, featuring 28.93 hours of bonafide and 29.4 hours of deepfake song clips from 40 real–AI singer pairs. Its splits—train, validation, and progressively challenging test sets (T01: seen singer, T02: unseen singers, T03: codec-shifted, T04: unseen language/genre)—allow for assessment of generalization along axes of singer identity, codec perturbation, and musical style. Labels are binary (bonafide=1, deepfake=0), and the average clip length is 13.75 seconds.

General Music Deepfake Benchmarks:

Datasets such as FakeMusicCaps (Sunday, 3 May 2025)—with over 10,000 human and deepfake 10s audio clips from a mix of TTM generators (MusicGen, AudioLDM2, MusTaNGo, etc.)—and the FMA "medium" split (Afchar et al., 2024), transformed via multiple decoders (e.g., neural codecs, conventional vocoder pipelines), underlie the supervised training and evaluation of deepfake music detectors beyond the vocal domain.

Test Regimes and Preprocessing:

Evaluation splits on these datasets induce regime shifts through, for example, application of audio augmentations (tempo stretching, pitch shifting, codec and noise corruptions) (Sunday, 3 May 2025, Sroka et al., 7 Jul 2025), unseen model families (Afchar et al., 2024), or out-of-distribution language and genre (e.g., Persian, Hip-Hop in SingFake T04).

2. Model Architectures and Feature Engineering

The canonical pipeline for supervised music deepfake detection includes front-end feature extraction followed by a deep classifier. The most effective systems exploit both local spectral cues and higher-order musical semantics.

Feature Extraction:

• Spectrogram Variants: Magnitude spectrograms and log-Mel spectrograms are standard (Sunday, 3 May 2025, Afchar et al., 2024, Zang et al., 2023). STFT is parameterized as $X_{m,k} = \sum_n x[n] w[n-mH] e^{-j 2\pi k n/N}$, with subsequent Mel or decibel transformations.
• Cepstral Features: MFCC, LFCC, and CQCC are also evaluated as inputs, with LFCC serving as a common speech–music baseline (Zang et al., 2023, Sharma et al., 31 Jan 2025).
• Learned Representations: Self-supervised embeddings from models such as Wav2Vec2 and Whisper—especially "noise-variant" encodings from Whisper—offer enhanced discriminative power over handcrafted features (Sharma et al., 31 Jan 2025).

Neural Classifiers:

• Convolutional Neural Networks (CNNs): Multi-layer 2D CNNs, including ResNet18/34 backbones, are widely used for spectrogram input and provide strong baseline performance (Sunday, 3 May 2025, Afchar et al., 2024).
• Graph Neural Networks: AASIST leverages a graph attention mechanism over time–frequency patches; SingGraph fuses MERT (pitch/rhythm) and Wav2Vec2 (lyrics) via heterogeneous spectral–temporal graph attention (Zang et al., 2023, Chen et al., 2024).
• Transformers: SpecTTTra-models tokenize Mel-spectrograms and process tokens via self-attention, yielding improved performance in large-scale settings (Sroka et al., 7 Jul 2025, Li et al., 2024).
• Lightweight CNNs (LCNNs): Used for fast initial discrimination in triage pipelines (Salvi et al., 20 Oct 2025).

Hybrid and Pipeline Architectures:

Multi-stage frameworks, such as a discriminator followed by an identification model (e.g., LCNN followed by ECAPA-TDNN for singer ID), enhance both detection and forensic attribution on high-quality forgeries (Salvi et al., 20 Oct 2025).

3. Training Objectives, Protocols, and Evaluation Metrics

Supervised detectors are trained with binary or multi-class cross-entropy losses:

$$\mathcal{L}_{\mathrm{CE}} = -\frac{1}{N} \sum_{i=1}^N \left[ y_i \log p_i + (1-y_i) \log(1-p_i) \right]$$

where $p_i$ is the probability assigned to the positive class (e.g., bonafide, deepfake, or class c for singer ID).

Supervised Regimes:

• Speech Countermeasure Transfer: Pretrained speech deepfake detectors (e.g., on ASVspoof2019-LA) perform poorly when applied out-of-domain to singing, but retraining on music data yields substantial EER improvements (from 50–60% down to ≤10%–32% on unseen singers) (Zang et al., 2023).
• Augmentation: RawBoost (additive stationary noise), beat-matching, tempo and pitch shifts, mixing in background music/noise, and on-the-fly pitch shifting serve to diversify the training distribution and foster generalization (Chen et al., 2024, Salvi et al., 20 Oct 2025, Sunday, 3 May 2025).

Key Metrics:

• Equal Error Rate (EER): Operating point where false acceptance and rejection rates intersect, standard in both singing-voice and music detection (Zang et al., 2023, Sharma et al., 31 Jan 2025, Chen et al., 2024).
• Accuracy, Precision, Recall, F1, Specificity: Routine for general music detection studies (Sunday, 3 May 2025, Sroka et al., 7 Jul 2025).
• ROC–AUC: Area under the ROC curve, particularly when calibration or discrimination performance matters (Li et al., 2024).

4. Empirical Performance, Robustness, and Generalization

Tables: Representative EER/Accuracy Results

Method / Features	Vocals EER	Mixtures EER	Key Test (T04/T03)	Source
Whisper-Medium + ResNet34	4.86%	9.45%	13.05–18.16%	(Sharma et al., 31 Jan 2025)
Wav2Vec2 + AASIST (mix)	8.23%	13.62%	42.77%	(Zang et al., 2023)
SingGraph (full)	6.23%	n/a	6.30% (T03)	(Chen et al., 2024)
ResNet18 (FakeMusicCaps, clean)	88.5% acc	—	—	(Sunday, 3 May 2025)
CNN/Spectrogram (FMA)	up to 99.8% acc	—	~0% on OOD decoders	(Afchar et al., 2024)
SpecTTTra-α (clean, Suno)	96% acc	—	–28pp under pitch shift	(Sroka et al., 7 Jul 2025)

Observations:

• State-of-the-art models based on Whisper encodings and deep residual classifiers achieve the lowest reported EERs on challenging SingFake splits, consistently outperforming LFCCs, MFCCs, spectrograms, and earlier Wav2Vec2 or AASIST baselines (Sharma et al., 31 Jan 2025).
• SingGraph’s integration of music-specific (MERT) and language-specific (Wav2Vec2) features with graph attention yields compounded gains, especially for unseen singers and codecs (Chen et al., 2024).
• General music deepfake detectors, while achieving high accuracy on in-distribution or known decoder sets (e.g., 99.8% on FMA-derived fakes (Afchar et al., 2024)), display catastrophic generalization failure to unseen generation pipelines (accuracy ≈ 0% inter-family).
• Small audio manipulations—pitch shift ±2 semitones, time-stretch, low-bitrate encoding, light noise—cause dramatic drops (5–30 percentage points) in accuracy and EER; in some cases, fake detection accuracy on manipulated samples collapses to 0% (Sunday, 3 May 2025, Sroka et al., 7 Jul 2025, Afchar et al., 2024).
• Triaging pipelines (e.g., LCNN discriminator preceding singer ID) boost robustness by focusing recognition resources on high-quality fakes (Salvi et al., 20 Oct 2025).

5. Limitations, Vulnerabilities, and Failure Modes

Supervised music deepfake detectors face several fundamental and practical challenges:

• Instrumental Interference: In mixtures, strong background music can mask synthesis artifacts, degrading signal-to-artifact contrast and EER (Zang et al., 2023).
• Source Separation Artifacts: Imperfect vocal extraction introduces confounds, especially for minority genres or complex musical mixtures (Zang et al., 2023, Chen et al., 2024).
• Codec and Augmentation Robustness: Most systems are sensitive to benign pitch/tempo/codecs, suffering major performance loss when faced with real-world editing or adversarial augmentation (Sroka et al., 7 Jul 2025, Sunday, 3 May 2025, Afchar et al., 2024).
• Generalization Deficits: Classifiers trained on one generator family do not transfer to others—table 3 of (Afchar et al., 2024) demonstrates nearly zero transfer accuracy outside of intra-family settings.
• Spurious Correlations: Over-reliance on spectral, pitch, or silence artifacts results in high false positive rates for legitimate music containing atypical edits or silence (Sroka et al., 7 Jul 2025).
• Unseen Language, Genre, and Singer Diversity: All approaches exhibit EER jumps (≳40–50% in some splits) when evaluated on wholly out-of-distribution genres and languages (Zang et al., 2023, Chen et al., 2024, Sharma et al., 31 Jan 2025).
• Adversarial Evasion: Re-encoding, noise injection, or remastering can defeat detection unless such augmentations are anticipated during training (Afchar et al., 2024, Sunday, 3 May 2025).

6. Research Directions and Perspectives

Algorithmic Innovations:

• Genre- and language-agnostic representation learning aims to disentangle musicological style from synthetic cues (Zang et al., 2023, Chen et al., 2024).
• Interference-robust and multi-branch architectures explicitly segregate accompaniment from vocals, as in SingGraph or multi-stream pipelines (Chen et al., 2024).
• Noise-variant SSL features: Whisper encodings, which paradoxically retain information about background and synthetic artifact, constitute a powerful, more robust basis compared to traditional speech features (Sharma et al., 31 Jan 2025).

Training Paradigms:

• Augmentation-averse training—incorporating pitch, tempo, codec, and silence augmentations to immunize detectors against trivial editing (Sroka et al., 7 Jul 2025, Zang et al., 2023).
• End-to-end separation plus detection: Joint modeling of source separation and fake detection, possibly via multi-objective training regimes, to mitigate separation-induced artifacts (Zang et al., 2023).
• Multi-modality: Fusing lyrics/text features with audio (as in SONICS) or employing multimodal transformers (Li et al., 2024) for richer semantic grounding.

Forensics, Attribution, and Explainability:

• Patch-wise attribution maps or sliding-window confidence heatmaps indicate which regions are responsible for fake classification (Afchar et al., 2024).
• Singer ID triage: Two-stage pipelines that first filter low-fidelity forgeries before attempting forensic identification show both improved accuracy and increased robustness against low-level attacks (Salvi et al., 20 Oct 2025).
• Transparency and auditing recommendations call for public reporting of detector error rates, especially false positive rates on legitimate content (Sunday, 3 May 2025).

Deployment and Societal Considerations:

• Continual Learning: Periodic updating on newly observed manipulations or OOD generators is needed to maintain detector fidelity but care is required to balance recall and specificity (Sunday, 3 May 2025).
• Regulatory and rights-protection implications: The proliferation of deepfakes mandates not only technical detection, but synergy with watermarking, provenance verification, and policy-level interventions (Afchar et al., 2024).

7. Comparative Summary Table

Detector Name / Paper	Core Feature	Model Type	Robustness (Aug/Gen)	Notable Weakness / Limitation
SingFake baselines (Zang et al., 2023)	Raw/LFCC/Spectrogram	AASIST, ResNet, W2V2+AASIST	Weak-to-moderate	Large EER increase for T04/OOD cases
Whisper-based (Sharma et al., 31 Jan 2025)	Whisper encoding	CNN, ResNet-34	Moderate	Still nontrivial degradation unseen genres
SingGraph (Chen et al., 2024)	MERT+W2V2+GNN	Multi-stream graph	Improved	May inherit limits of separation
SONICS SpecTTTra-α (Sroka et al., 7 Jul 2025)	Log-Mel	Conv+Transformer	Low (to OOD/aug)	Collapses under simple pitch/time edits
ResNet18 (FakeMusicCaps) (Sunday, 3 May 2025)	Mel spectrogram	ResNet18	Low	10–20pp accuracy drop, doubled FPR w/aug
FMA CNN (Afchar et al., 2024)	Magnitude spec	6-layer CNN	Very low to OOD/deco	No inter-family generalization
LCNN+ECAPA-TDNN (Salvi et al., 20 Oct 2025)	Log-mel	LCNN → ECAPA-TDNN pipeline	N/A (ID task)	Fails on low-fidelity fakes

The state of the art for supervised music deepfake detection is characterized by efficacious in-domain performance and innovative model architectures, yet remains limited by brittleness to basic edits, poor OOD generalization, and the complexity of musical diversity. Ongoing research must address these vulnerabilities through deeper augmentation, principled representation learning, and cross-modal integration while ensuring transparency and domain-aware deployment practices.