Separation-Enhanced Joint Learning

• The paper introduces a framework that embeds explicit source separation within an end-to-end anti-spoofing model to discern manipulated audio components.
• The architecture combines mixture detection, dual-branch classification, and fusion of speech and environmental cues to enable five-class discrimination.
• Empirical results show strong performance on curated datasets while highlighting challenges in generalizing to unseen spoofing generators, especially for environmental sounds.

A separation-enhanced joint learning framework is an architectural paradigm that structures audio anti-spoofing systems to exploit explicit source separation as an intermediate step within an end-to-end model. This approach has been introduced to address the challenge of detecting component-level spoofing in complex audio mixtures, where either foreground speech, background environmental sounds, or both may be independently manipulated via state-of-the-art generative models. The separation-enhanced joint learning framework, together with the CompSpoofV2 dataset, forms the basis of the Environment-Aware Speech and Sound Deepfake Detection Challenge (ESDD2), targeting scenarios in which conventional monolithic detectors are insufficient (Zhang et al., 12 Jan 2026).

1. Motivation and Role in Component-Level Audio Spoofing

Conventional audio anti-spoofing systems have generally operated under the assumption that input recordings are globally bona fide or globally spoofed. With the maturation of text-to-speech (TTS), voice-conversion (VC), and environmental sound synthesis models, individual components within a recording can be attacked independently. This necessitates approaches that not only discern spoofing at the whole-utterance level but also disentangle and analyze each component for possible generation artifacts. A separation-enhanced joint learning framework explicitly integrates signal separation to improve sensitivity to component-level manipulations and mitigate masking effects, where unspoofed segments obscure artifacts in manipulated regions.

2. Baseline Architecture and Workflow

The separation-enhanced joint learning baseline consists of a staged sequence of neural modules:

1. Mixture Detection: A binary classifier $M(\cdot)$ determines if the input $x(t)$ is an unaltered recording (“original,” class 0) or contains any form of mixture (classes 1–4).
2. Source Separator: The separator $S(\cdot)$ estimates two latent waveforms, $\hat{y}_\mathrm{speech}(t)$ and $\hat{y}_\mathrm{env}(t)$, corresponding to foreground speech and environmental background respectively.
3. Dual-Branch Classification:
   • A speech anti-spoofing classifier $C_s(\cdot)$ processes $\hat{y}_\mathrm{speech}(t)$.
   • An environment classifier $C_e(\cdot)$ processes $\hat{y}_\mathrm{env}(t)$.
4. Fusion and Decision: The mixture detection score, speech component spoof score, and environmental component spoof score are fused via a fully connected layer, producing a five-way softmax over the mutually exclusive classes outlined in CompSpoofV2.

The end-to-end training objective is the weighted sum of separation loss ($L_{\mathrm{sep}}$) and five-class cross-entropy classification loss ($L_{\mathrm{cls}}$):

$$L_\mathrm{total} = L_\mathrm{sep} + \lambda L_\mathrm{cls}$$

where $L_\mathrm{sep}$ is typically the negative scale-invariant signal-to-distortion ratio (SI-SDR) between estimated and ground-truth separated waveforms; $L_\mathrm{cls}$ is

$$L_\mathrm{cls} = -\sum_{i=0}^{4} y_{\mathrm{true},i} \log p_i(x;\theta)$$

with $p_i$ the softmax probability for class $i$, $y_\mathrm{true}$ the one-hot ground truth, and $\lambda$ a tunable weight (set to 0.1 in the baseline). This end-to-end training ensures that separation preserves the acoustic cues relevant to both speech and environment spoofing.

3. Relationship with the CompSpoofV2 Dataset

CompSpoofV2 is expressly constructed to facilitate and benchmark the performance of algorithms like separation-enhanced joint learning. The dataset comprises over 250,000 curated audio clips (totaling approximately 283 hours), partitioned evenly into five classes reflecting all combinations of bona fide and spoofed speech/environment components. Spoofing engines include Tacotron2-style TTS systems, VC systems leveraging autoencoder plus FiLM-conditioning, and GAN-based soundscape synthesizers. The five classes are:

ID	Class Name	Speech	Environment
0	original	genuine	genuine
1	bonafide_bonafide	genuine	different genuine
2	spoof_bonafide	spoofed	genuine
3	bonafide_spoof	genuine	spoofed
4	spoof_spoof	spoofed	spoofed

Table 1. CompSpoofV2 class definitions as in (Zhang et al., 12 Jan 2026).

A separation-enhanced joint learning model enables explicit detection of nuanced artifacts in both channels and directly supports the five-way structure of CompSpoofV2.

4. Data Flow and Training Objectives

Each sample is processed as follows: The system first uses the mixture detector to distinguish between isolated and mixed classes. For mixed inputs, the separator outputs two time-domain signals; these are simultaneously subject to anti-spoofing detection in their respective branches. The joint classification module integrates all three scores, producing a five-class posterior. Ground truth for both separation and classification is available in training, with validation and test splits leveraging “new generated” environmental samples to evaluate generalization under zero-day attacks.

The combined separation and classification training loss encourages the model to produce separations with high fidelity to the original components while maintaining discriminative cues for spoofing detection. This design addresses potential loss of salient cues that may otherwise arise if separation and classification were optimized independently.

5. Evaluation Methodology and Baseline Performance

Evaluation in the ESDD2 challenge framework is primarily based on the macro-averaged F1 score:

$$\text{Macro-F1} = \frac{1}{5} \sum_{i=0}^{4} \text{F1}_i$$

with

$$\text{F1}_i = \frac{2P_i R_i}{P_i + R_i}, \quad P_i = \frac{\text{TP}_i}{\text{TP}_i+\text{FP}_i}, \quad R_i = \frac{\text{TP}_i}{\text{TP}_i+\text{FN}_i}$$

where $P_i$ and $R_i$ denote the precision and recall for class $i$. Diagnostic equal-error-rate (EER) metrics are also reported for original detection, speech spoof detection, and environment spoof detection, though leaderboard ranking is solely determined by Macro-F1.

On the CompSpoofV2 validation set, the baseline yields:

• Original EER: 0.31%
• Speech EER: 1.72%
• Environment EER: 37.66%
• Macro-F1: 0.9462

However, on held-out evaluation and test splits with previously unseen spoof generators, the Macro-F1 drops to approximately 0.62. This result highlights the elevated complexity of component-level spoof detection, particularly for environmental sound manipulations. A plausible implication is that source separation and cross-domain generalization remain unsolved challenges in robust spoof detection frameworks employing separation modules (Zhang et al., 12 Jan 2026).

6. Metadata, Organization, and Operational Considerations

Each audio instance is annotated with a comprehensive JSON schema, including audio ID, class ID, separate genuineness flags for speech and environment, provenance and model details for each component, SNR levels employed during mixing, and technical details such as sampling rate and bit depth. The dataset is organized hierarchically by split and class (train, val, eval, test; classes 0–4), with associated CSV and JSON metadata for streamlined integration into training and evaluation pipelines.

The data is released for non-commercial research under challenge rules, prohibiting redistribution and use of supplementary synthetic audio except by prior approval. This ensures comparability and reproducibility of methods employing the separation-enhanced joint learning strategy.

7. Significance and Challenges

The separation-enhanced joint learning framework represents a domain-specific adaptation of end-to-end learning that leverages explicit separation to address the unique challenges of detecting sophisticated generation-based audio manipulations. Key advantages include improved detection in settings where artifact masking and component interplay preclude the use of monolithic detectors, and a principled mechanism for integrating both separation and classification cues into a single optimization objective. Nevertheless, the observed performance degradation on generalized, previously unseen generators—especially for environmental spoofing—underscores the current methodological limitations and motivates future research in separation robustness, disentanglement, and cross-domain anti-spoofing generalization (Zhang et al., 12 Jan 2026).