Spotting tell-tale visual artifacts in face swapping videos: strengths and pitfalls of CNN detectors

Abstract: Face swapping manipulations in video streams represents an increasing threat in remote video communications, due to advances in automated and real-time tools. Recent literature proposes to characterize and exploit visual artifacts introduced in video frames by swapping algorithms when dealing with challenging physical scenes, such as face occlusions. This paper investigates the effectiveness of this approach by benchmarking CNN-based data-driven models on two data corpora (including a newly collected one) and analyzing generalization capabilities with respect to different acquisition sources and swapping algorithms. The results confirm excellent performance of general-purpose CNN architectures when operating within the same data source, but a significant difficulty in robustly characterizing occlusion-based visual cues across datasets. This highlights the need for specialized detection strategies to deal with such artifacts.

• The paper demonstrates that CNN detectors achieve near-perfect accuracy (B-ACC > 99%) on matched datasets but falter in cross-dataset scenarios.
• The paper employs controlled datasets (GOTCHA and FOWS) and tests five CNN architectures to assess the impact of occlusion-induced artifacts on detection performance.
• The paper finds that feature attribution via GradCAM++ reveals detectors often neglect occlusion regions, indicating the need for specialized, anomaly-aware models.

Overview of "Spotting tell-tale visual artifacts in face swapping videos: strengths and pitfalls of CNN detectors" (2506.16497)

This work presents an in-depth empirical study of CNN-based detectors tasked with identifying face-swapping manipulations in video streams. The focus centers on the efficacy of using occlusion-induced visual artifacts—arising when subjects perform deliberate occluding actions with their hands or objects during capture—as cues for automated detection in remote video communication scenarios. The analysis leverages two datasets: an existing, challenge-response benchmark (GOTCHA) and a newly introduced corpus (FOWS), both systematically capturing occlusion events.

Context and Motivation

The rapid proliferation of real-time face-swapping systems has heightened security concerns in settings such as remote video calls and digital identity proofing. While human observers can often notice tell-tale artifacts during occlusions, it remains unclear to what extent CNN-based detectors can automate such recognition—especially across diverse datasets and manipulation pipelines. The study thus addresses the critical question of detector generalizability and the limitations inherent to purely data-driven approaches.

Dataset Development and Experimental Design

The FOWS dataset is designed with explicit challenge-based occlusions: participants perform hand and object occlusions in controlled spatial patterns, with both genuine and manipulated versions produced via state-of-the-art face swapping algorithms (SimSwap, GHOST, FaceDancer). Complementary data from the GOTCHA corpus provides comparative variety, as it includes different users and swapping engines (DeepFaceLab, FSGAN).

Frames are categorized into occlusion (occ) and non-occlusion (no-occ) subsets using a combination of automated (Google Mediapipe's BlazeFace) and manual verification. This enables the controlled investigation of occlusion artifacts as detection cues.

The experimental protocol involves:

• Model Benchmarks: Five CNN architectures (MobileNetV2, EfficientNetB4, XceptionNet, plus two task-specific variants pre-trained on DFDC and FaceForensics++).
• Evaluation Modes: Intra-dataset (train/test within one dataset) and cross-dataset (train on one, test on the other) evaluations, further divided into occlusion and non-occlusion frame splits.
• Metrics: Balanced Accuracy (B-ACC), Area Under Curve (AUC), and Equal Error Rate (EER) for robust performance estimation in the presence of class imbalance.

Empirical Findings

1. Intra-Dataset Detection

When trained and tested within the same dataset and occlusion category, all evaluated CNNs yield near-perfect results (B-ACC > 99% in most occlusion settings). These results confirm that local, dataset-specific visual artifacts provide strong signals for manipulation detection.

2. Cross-Dataset Generalization

A pronounced drop in effectiveness emerges in all cross-dataset scenarios. B-ACC scores drop by roughly half compared to intra-dataset settings, and accuracies on specific dataset partitions are markedly unstable. Notably, AUC and EER occasionally remain relatively high, suggesting that model output scores retain some separation, but their operating thresholds become misaligned, underscoring poor calibration and unreliable decision-making on unseen data.

3. Feature Attribution and Interpretability

GradCAM++ visualizations reveal that models often do not focus on occlusion regions (where the most reliable artifacts reside) but instead attend to non-occlusion regions or generic face areas. Cross-category testing further reveals that detectors trained on non-occlusion frames can generalize to occlusion frames (and vice versa) with negligible performance degradation; this indicates that learned cues are not exclusive to occlusion artifacts, raising concerns about robustness and specificity.

4. Pretraining and Specialized Models

Models originally trained on large generic fake datasets (DFDC, FaceForensics++) demonstrate no consistent improvement over lighter, general-purpose CNN baselines in both intra- and cross-dataset scenarios. This suggests limited transferability of learned representations for occlusion-specific artifact detection.

Implications and Prospective Directions

The results highlight limitations in current data-driven, CNN-based detectors for practical face-swap detection under real-world conditions:

• Generalization Challenge: Purely data-driven CNNs, even with state-of-the-art architectures and pretraining, overfit to intra-dataset signals and do not robustly generalize to novel acquisition pipelines, participants, or manipulation engines.
• Salient Cues: Despite the intuitive value of occlusion-based "challenge-response" artifacts, current models fail to reliably exploit them without explicit supervision or architectural guidance.
• Operational Risks: Deployment of uncalibrated models, trained on limited data sources, is likely to induce high false positive/negative rates in the field, particularly for security-sensitive applications (e.g., video-based KYC).

Promising future research directions include:

• Development of specialized detectors with explicit occlusion-awareness (e.g., multi-branch architectures combining occlusion detection and artifact analysis).
• Incorporation of one-class/anomaly detection paradigms, focusing on modeling genuine (unmanipulated) data distribution to better accommodate varied attack types and reduce dependence on labeled fake data.
• Introduction of attention mechanisms or region-guided supervision to enforce model focus on physically plausible artifact regions during training.
• Exploration of domain adaptation or meta-learning strategies to alleviate dataset-specific overfitting and boost cross-domain transferability.

Conclusions

This study delivers a thorough empirical assessment of existing CNN-based manipulation detectors in the context of challenge-based face-swap artifact detection. Although excellent performance is achievable when data distribution is held constant, the strong dataset dependence and imperfect exploitation of occlusion-based cues highlight critical limitations. Addressing generalization remains paramount for developing secure and scalable forensic tools in adversarial environments. Future research should emphasize models and paradigms that incorporate explicit physical priors or anomaly detection, moving beyond end-to-end discriminative approaches that are easily confounded by dataset biases.