Wavelet Prompt-Tuned XLSR-AASIST

• The paper introduces a parameter-efficient framework that integrates wavelet prompt tuning with frozen XLSR and AASIST to detect deepfake audio spoofing attacks.
• It utilizes learnable standard and wavelet prompt tokens injected into transformer layers, coupled with a multi-system speaker verification ensemble using score fusion.
• Empirical results show state-of-the-art performance with significant EER reductions across diverse benchmarks while updating less than 1% of model parameters.

Wavelet Prompt-Tuned XLSR-AASIST is a parameter-efficient end-to-end countermeasure framework designed for detecting generative audio spoofing attacks while retaining speaker verification performance, with particular strength in simultaneously capturing fine time–frequency artifacts invariant to content type. This approach leverages the frozen XLSR (cross-lingual self-supervised representation learning) transformer, augmenting it via the targeted injection of learnable prompt tokens and specially constructed wavelet-domain prompt embeddings, and cascades the resulting deepfake detection output with a multi-system speaker verification ensemble. Key research contributions come from advances in wavelet prompt tuning, prompt-domain sparsification strategies, and integrated anti-spoofing speaker verification, as demonstrated on benchmarks including WildSpoof 2026, SpoofCeleb, Deepfake-Eval-2024, and large-scale all-type deepfake detection settings (Farhadipour et al., 24 Jan 2026, Xuan et al., 6 Oct 2025, Xie et al., 9 Apr 2025).

1. Underlying Architecture and Model Integration

The Wavelet Prompt-Tuned XLSR-AASIST (WPT-XLSR-AASIST) framework consists of three principal modules:

1. Front-end: A frozen XLSR-53 large transformer model pretrained on cross-lingual audio, with a CNN feature extractor followed by numerous transformer layers (e.g., $l=24$), each with hidden size $d$ (typically $d=1024$).
2. Prompt and Wavelet Prompt Injection: At each transformer block $\ell$, the hidden representations $H_{\ell-1} \in \mathbb{R}^{T \times D}$ are prepended with $N$ standard prompt tokens $P = \{p_i\}_{i=1}^N$ and $M$ wavelet prompt tokens $W = \{w_j\}_{j=1}^M$, creating $$\widetilde{H}_{\ell-1} = [p_1; \ldots; p_N; w_1; \ldots; w_M; H_{\ell-1}] \in \mathbb{R}^{(N+M+T)\times D}$$. These extended embeddings feed into the transformer's multi-head self-attention and feed-forward sublayers.
3. Back-end: AASIST, a dual-graph spectro-temporal attention network, that pools and encodes the output of the prompt-augmented XLSR stack, producing a bona fide versus spoof logit for countermeasure scoring.

The overall system is cascaded in a Spoofing-Aware Speaker Verification (SASV) paradigm. The first stage (countermeasure) discards spoofed utterances, while the second stage (ASV) fuses ensemble speaker similarity scores, ultimately accepting only those samples passing both thresholds (Farhadipour et al., 24 Jan 2026).

2. Wavelet Prompt Tuning Mechanism

Wavelet Prompt Tuning enhances standard prompt tuning by injecting a learned set of prompt tokens that are explicitly structured or transformed in the wavelet domain to promote time–frequency localization:

• Prompt Tokens: Standard prompt tokens $P = \{p_i \in \mathbb{R}^D\}$ are randomly initialized and learned per transformer layer via backpropagation.
• Wavelet Prompt Tokens: $W = \{w_j \in \mathbb{R}^D\}$ are also randomly initialized but are intended to focus on specific time-frequency resolutions.

Theoretical underpinning: Each $w_j$ can be interpreted as a trainable linear combination of dilated and shifted wavelet basis functions, $$w_j \approx \sum_{(a,b)\in S_j} \alpha_{j,a,b}\,\psi_{a,b}(t)$$, but in practice, they are simple learnable vectors with no explicit analytic wavelet structure. Training encourages specialization for detecting artifacts at different spectral scales (Farhadipour et al., 24 Jan 2026).

Variants such as WaveSP-Net (Xuan et al., 6 Oct 2025) and WPT-SSL (Xie et al., 9 Apr 2025) further refine this mechanism, either by:

• Performing explicit/discrete wavelet transforms (DWT) on parts of the prompt embedding matrix, sparsifying coefficients via random Bernoulli masks, and learning both wavelet analysis and synthesis filters (“wavelet-domain sparse prompt tuning”).
• Applying single-level 2D Haar DWT on wavelet prompt tokens, then stacking LL/LH/HL/HH subbands to form the final prompt set at each layer.

In these variations, prompt embeddings are the sole trainable parameters alongside back-end classifiers—core XLSR weights remain frozen, yielding high parameter efficiency.

3. Training Protocols and Parameter Efficiency

Wavelet Prompt-Tuned XLSR-AASIST is optimized for binary cross-entropy (BCE) or weighted cross-entropy loss: $$L = -\sum_{i\in\text{batch}}\Big[\,\omega_\text{pos}\, y_i\,\log \hat{y}_i + \omega_\text{neg}\,(1 - y_i)\,\log (1 - \hat{y}_i)\,\Big]$$ where class weights $\omega_\text{pos}$, $\omega_\text{neg}$ account for class imbalance. Training details include:

• Optimizer: Adam
• Learning Rate: $1\times 10^{-4}$ (mainline WPT), $5\times 10^{-4}$ for WPT-SSL and WaveSP-Net, with cosine-annealing for WPT-XLSR-AASIST.
• Batch Size: 64 (WPT-XLSR-AASIST), 32 (WPT-SSL/WaveSP-Net)
• Epochs: 15 for WPT-XLSR-AASIST, 20–50 for others depending on co-training and early stopping protocols.

Parameter counts demonstrate the method’s efficiency: full XLSR fine-tuning updates all $\sim 300$M weights; prompt-based methods reduce learnable parameter count by over two orders of magnitude (0.69M for WPT-XLSR-AASIST, $<$1.5% of model size) (Xie et al., 9 Apr 2025).

4. Integration with Speaker Verification and Score Fusion

Wavelet Prompt-Tuned XLSR-AASIST countermeasure is integrated into a two-stage SASV pipeline:

Stage	Operation	Output
Countermeasure	WPT-XLSR-AASIST scores utterance	$s_{\text{CM}}(x)$ (spoof vs. bona fide)
ASV ensemble	ResNet34, ResNet293, WavLM-ECAPA-TDNN	$s_1, s_2, s_3$ (similarity scores)
Z-score norm	$(s_i - \mu_i)/\sigma_i$ for cohort statistics	$z_i$
Score fusion	$s_{\text{ASV}} = \frac{1}{3}(z_1+z_2+z_3)$	Fused ASV score
Final accept	$s_{\text{CM}} \geq \theta_{\text{CM}}$ and $s_{\text{ASV}} \geq \theta_{\text{ASV}}$	Output label

Both components are trained on large curated datasets: SpoofCeleb (containing bona fide/spoof pairs) for the countermeasure, and VoxCeleb2 with ArcFace loss for ASV backbones; no domain-adversarial adaptation losses are used, highlighting the pure in-domain fitting approach (Farhadipour et al., 24 Jan 2026).

5. Empirical Results and Cross-Domain Robustness

On in-domain benchmarks such as WildSpoof 2026, the overall system achieves:

• Spoof Detection (WPT-XLSR-AASIST): EER = 0.16%
• ASV Ensemble: EER = 2.25%
• SASV Joint EER: 2.08%
• Macro a-DCF: 0.2017 (across four unseen test splits)

In all-type deepfake audio benchmarks (speech, sound, singing, music), WPT-XLSR-AASIST achieves a mean average EER of 3.58%, surpassing 315M-parameter full fine-tuned systems (4.98% EER) with only 0.69M trainable parameters (Xie et al., 9 Apr 2025).

However, cross-domain testing (e.g., on ASVspoofF5, ASV 2022) reveals significant generalization gaps:

• ASVspoofF5: a-DCF = 0.4088
• ASV 2022: a-DCF = 0.3252

This highlights ongoing challenges in robust transfer to new data distributions, spoof genres, and recording conditions, indicating a limitation of in-domain prompt-tuning and a promising direction for domain-aware or multi-corpus extensions (Farhadipour et al., 24 Jan 2026).

6. Analytical Insights: Universal Artifact Detection via Wavelet Tokens

Analysis of t-SNE embeddings and self-attention distributions demonstrates that wavelet prompt tuning creates type-invariant feature spaces and attention mechanisms:

• Standard fine-tuning: Feature clusters are strongly type-dependent.
• WPT: Model’s self-attention consistently prioritizes the HH (diagonal high-frequency) subband token, regardless of content type, capturing universal generator artifacts.

A plausible implication is that WPT acts as a data-driven "artifact filter" for neural audio generation artifacts, enabling superior universal deepfake detection across modalities (speech, sound, singing, music) (Xie et al., 9 Apr 2025). Ablations confirm that high-frequency wavelet subbands and learnable filter adaptation are essential—removing them causes substantial EER increases.

7. Comparisons, Variants, and Future Outlook

Wavelet Prompt-Tuned XLSR-AASIST stands in contrast to other prompt-based and Fourier-based tuning strategies:

Variant	Prompt Structure	Param. Eff.	Key Result
PT-XLSR-AASIST	Standard prompt tokens	0.69M (<0.5%)	AVG EER (all-type) = 6.74%
FT-XLSR-AASIST	Full fine-tune	315.89M (full model)	AVG EER (all-type) = 4.98%
WPT-XLSR-AASIST	Standard + wavelet tokens	0.69M (<0.5%)	AVG EER (all-type) = 3.58%
WaveSP-Net	Sparse, DWT-transformed	4.15M (1.3%)	EER (SpoofCeleb) = 0.13%

WaveSP-Net (Xuan et al., 6 Oct 2025) further improves efficiency with partial wavelet-token processing and bidirectional Mamba back-end classifiers, showing state-of-the-art results at fractionally increased parameter cost.

Future research will likely focus on domain-aware prompt adaptation, multi-corpus prompt co-training, and more adaptive, hierarchical wavelet packet schemes to improve robustness to novel generation techniques and unseen conditions (Farhadipour et al., 24 Jan 2026, Xuan et al., 6 Oct 2025, Xie et al., 9 Apr 2025).