Multi-Task Transformer for Speech Deepfake Detection

• The paper introduces multi-task transformer architectures, such as SFATNet-4 and DFALLM, that integrate deep temporal modeling with prosody, formant, and voicing analysis for speech deepfake detection.
• It employs dual encoders, specialized prediction heads, and multi-head pooling to achieve improved EER and AUC metrics while reducing parameter count and training time.
• The approach enhances generalization to unseen spoofing methods through joint supervision and interpretable attention, despite challenges with lossy codec robustness.

A multi-task transformer for speech deepfake detection is a neural architecture that simultaneously addresses several subtasks relevant to the forensic analysis of speech audio. Central to these systems is the integration of deep temporal modeling (via transformer modules) with explicit modeling of speech prosody, formant structure, and voicing to both detect synthetic speech and provide interpretable rationales for decisions. Recently, two representative paradigms have emerged, typified by models such as SFATNet-4 (“Multi-Task Transformer for Explainable Speech Deepfake Detection via Formant Modeling”) (Negroni et al., 21 Jan 2026) and DFALLM (“DeepFake Audio LLM”) (Li et al., 9 Dec 2025). These models advance the field through architectural refinements, multi-task learning protocols, and explicit explainability mechanisms.

1. Foundational Model Designs

State-of-the-art multi-task architectures for speech deepfake detection integrate several processing blocks to maximize detection accuracy, generalization, and interpretability.

SFATNet-4 processes audio after resampling to 16 kHz and amplitude normalization, and computes the short-time Fourier transform (STFT) with 2.064 s windows (256 frequency bins, 128 time frames). Inputs are segmented along the time axis only, producing one token per frame. Dual encoders (one for magnitude, one for phase) of 8-layer transformers each (with 8 heads and embedding dimension 512) encode the time tokens. Outputs from both encoders are concatenated and projected for downstream decoding. The model supports three parallel prediction heads: (1) a multi-formant decoder regresses F₀, F₁, F₂ values with physiological constraints; (2) a voicing decoder predicts binary voiced/unvoiced labels at the frame level; (3) a synthesis predictor (transformer encoder–decoder) performs framewise deepfake/bona fide classification and aggregates results with multi-head pooling for final detection.

DFALLM proposes a modular pipeline: an audio encoder (pretrained Whisper or Wav2Vec2-BERT) maps the audio waveform to latent sequences, which are projected into the embedding space of a pretrained LLM (LLM, e.g., Qwen2.5-0.5B). These are concatenated with a prompt embedding and processed by the LLM, which autoregressively outputs answers to multi-task prompts (binary detection, spoof attribution, and spoof localization).

2. Multi-Task Learning Objectives and Losses

Multi-task training leverages shared representations across related subtasks to improve generalization and interpretability.

In SFATNet-4, the losses for detection (binary cross-entropy), voicing (binary cross-entropy), and formant regression (mean squared error on log-scaled, standardized F₀, F₁, F₂) are combined:

$$L_\text{total} = \lambda_d L_\text{det} + \lambda_v L_\text{voicing} + \lambda_f L_\text{formant}$$

with $\lambda_d=1.0$, $\lambda_v=0.3$, and $\lambda_f=0.3$, and the voicing mask restricts the formant loss to voiced frames. SFATNet-4 thus enforces physiologically meaningful priors and supports joint supervision.

DFALLM follows a similar multi-task strategy, combining

• detection (binary cross-entropy: $L_\text{det}$),
• attribution (N-way cross-entropy: $L_\text{attr}$),
• localization (squared error on start/end times: $L_\text{loc}$), with total loss $$L_\text{total} = \alpha L_\text{det} + \beta L_\text{attr} + \gamma L_\text{loc}$$, typically with $\alpha=\beta=\gamma=1$.

3. Explainability and Attribution Mechanisms

Explainability is obtained in SFATNet-4 through per-frame attention weights, derived from the multi-head pooling step of the synthesis predictor. This enables precise identification of time regions that influence the fake/real decision. Additionally, the model quantifies the relative contribution of voiced versus unvoiced frames to the detection output, by summing normalized attention weights across the two masked regions:

$C_\text{voiced} = \sum_{t: v_\text{mask}_t=1} w_t, \qquad C_\text{unvoiced} = \sum_{t: v_\text{mask}_t=0} w_t$

This supports direct interpretability of whether the classifier’s prediction relies on regions traditionally more susceptible to synthesis artifacts.

In DFALLM, explainability is mediated by the prompt-driven LLM: tasks can be formulated as free-form queries (e.g., “Is this audio fake or real?”, “Label the start and end times of spoof segments”), and the output can be parsed accordingly. The model also enables time-stamp localization of spoof regions and can perform spoof attribution.

4. Training Protocols and Evaluation Datasets

Both models rely on established training and evaluation protocols. SFATNet-4 is trained on ASVspoof 5 (merged train+dev, oversampled for class balance, no augmentation). In-domain evaluation uses ASVspoof 5 eval (clean). The out-of-domain evaluation includes In-the-Wild, FakeOrReal, and TIMIT-TTS (with VidTIMIT real added for coverage).

DFALLM aggregates data from ASVspoof2019 LA, SpoofCeleb, MLAADv6, ReplayDF, DFADD, AISHELL3, ADD2023, GigaSpeech, CNCeleb, and PartialSpoof, with ≈170,000 total training samples and multi-task splits for detection, attribution, and localization. Training uses the AdamW optimizer (SFATNet-4: lr=1e-4, batch=256; DFALLM: lr=1e-5, batch by tokens), with learning rate schedules and early stopping.

5. Empirical Results, Ablations, and Scalability

Key empirical results for SFATNet-4 demonstrate substantial improvements over earlier architectures:

Dataset	SFATNet-4 (EER, AUC)	SFATNet-3 (EER, AUC)
ASVspoof 5	4.41 %, 98.89 %	8.85 %, 96.69 %
In-the-Wild	17.29 %, 89.17 %	19.70 %, 85.20 %
FakeOrReal	20.33 %, 85.03 %	21.08 %, 81.01 %
TIMIT-TTS	20.93 %, 84.49 %	18.59 %, 83.36 %
Average	15.74 %, 89.40 %	17.06 %, 86.57 %

SFATNet-4 reduces the parameter count from 64.7M to 41.8M and training time per epoch from approximately 60 minutes to 15 minutes (NVIDIA A40). Codec robustness (without data augmentation) is less competitive under severe compression (e.g., MP3: EER 40.9%, AUC 64.9%).

SFATNet-4 ablation studies reveal that time-only segmentation affords 4× faster training and superior interpretability compared to spatiotemporal patching. Removing either the multi-formant or voicing decoder leads to degraded formant regression performance and weaker generalization. Multi-head pooling is essential for per-frame explainability; class-token pooling eliminates interpretability.

DFALLM achieves binary detection accuracies up to 99.15% (in-domain, Wav2Vec2-BERT audio-only), with out-of-domain average of 94.07% when coupled with Qwen2.5-0.5B. Multi-task detection, attribution, and localization are addressed simultaneously. Notably, the model’s primary bottleneck is the audio encoder; increasing LLM size beyond 0.5B has negligible gain. Higher frame rates (50 Hz vs. 12.5 Hz) marginally improve localization and OOD accuracy, especially for short-duration artifacts.

6. Generalization, Task Coverage, and Current Limitations

Multi-task transformer architectures achieve stronger generalization to unseen spoofing methods compared to single-task or monolithic approaches. SFATNet-4 is comparatively lightweight, provides frame-level prosodic and voicing supervision, and yields interpretable decisions at minimal cost in performance or latency. DFALLM extends generalization to out-of-domain tasks—including spoof attribution and time-stamp localization—by leveraging pretraining on vast, diverse speech corpora and efficient prompt-based transfer.

Observed limitations include sensitivity to aggressive lossy codecs (in the absence of dedicated augmentation), and a generalization bottleneck rooted in the pretraining objective and capacity of the audio encoder rather than the LLM head. Models that omit voicing/formant supervision or interpretability-driven pooling mechanisms incur penalty in generalization, especially under domain shift or for low-resource spoof types.

A plausible implication is that future advances will focus on integrating physiologically-informed submodules, robust data augmentation for codec and channel variability, and further harmonization of speech-specific and LLM-style multi-modal modeling for forensic speech analysis. Recent trends also suggest that modestly sized LLMs, paired with powerful acoustic encoders and judicious prompt design, are sufficient for maintaining state-of-the-art deepfake detection and generalization (Li et al., 9 Dec 2025, Negroni et al., 21 Jan 2026).