Beyond Semantics: Uncovering the Physics of Fakes via Universal Physical Descriptors for Cross-Modal Synthetic Detection

Abstract: The rapid advancement of AI generated content (AIGC) has blurred the boundaries between real and synthetic images, exposing the limitations of existing deepfake detectors that often overfit to specific generative models. This adaptability crisis calls for a fundamental reexamination of the intrinsic physical characteristics that distinguish natural from AI-generated images. In this paper, we address two critical research questions: (1) What physical features can stably and robustly discriminate AI generated images across diverse datasets and generative architectures? (2) Can these objective pixel-level features be integrated into multimodal models like CLIP to enhance detection performance while mitigating the unreliability of language-based information? To answer these questions, we conduct a comprehensive exploration of 15 physical features across more than 20 datasets generated by various GANs and diffusion models. We propose a novel feature selection algorithm that identifies five core physical features including Laplacian variance, Sobel statistics, and residual noise variance that exhibit consistent discriminative power across all tested datasets. These features are then converted into text encoded values and integrated with semantic captions to guide image text representation learning in CLIP. Extensive experiments demonstrate that our method achieves state-of-the-art performance on multiple Genimage benchmarks, with near-perfect accuracy (99.8%) on datasets such as Wukong and SDv1.4. By bridging pixel level authenticity with semantic understanding, this work pioneers the use of physically grounded features for trustworthy vision language modeling and opens new directions for mitigating hallucinations and textual inaccuracies in large multimodal models.

• The paper introduces a novel fusion of handcrafted physical features with vision-language models to robustly distinguish real from synthetic images.
• It proposes the FSDVA algorithm for scoring features based on stability and discriminability, enabling the selection of core descriptors like Laplacian variance and Sobel metrics.
• Empirical evaluations show systematic improvements in cross-model generalization and near state-of-the-art accuracy, reducing reliance on semantically driven detection.

Universal Physical Descriptors for Generalizable Cross-Modal Synthetic Image Detection

Introduction

The proliferation of AI-generated content (AIGC), notably through diffusion models and GAN-based synthesis, continuously erodes the perceptual and statistical boundaries between authentic and synthetic imagery, directly challenging the reliability and pertinence of existing deepfake detectors. Machine learning-based detectors typically exhibit clear generalization deficits across novel generator architectures, while semantic- and prompt-driven multimodal systems (e.g., CLIP-based methods) inherently tether their performance to LLM reliability and are consequently vulnerable to prompt engineering and hallucination.

The paper "Beyond Semantics: Uncovering the Physics of Fakes via Universal Physical Descriptors for Cross-Modal Synthetic Detection" (2604.04608) addresses this core robustness crisis by positing that universal, statistical pixel-level image descriptors—derived from handcrafted image analysis—can inject physically grounded evidence into vision-LLMs, thereby mitigating both cross-domain overfitting and linguistic unreliability.

Universal Physical Feature Selection

The authors exhaustively quantify fifteen established physical features, including Laplacian variance, Sobel edge statistics, local binary pattern (LBP) entropy, residual noise variance, DCT-based spectrum measures, channel covariances, and chroma/hue statistics, across more than twenty datasets spanning canonical GANs and cutting-edge diffusion architectures.

To isolate robust, generalizable descriptors, they propose the FSDVA (Feature Stability-Discriminability Validation Assessment) algorithm, scoring each candidate by both cross-dataset stability ($S_s$)—expressed as inverse coefficient of variation over means—and discriminative power ($S_d$), as the composite of inter/intra-class Jensen–Means distance (JMD) and AUC (Area Under ROC Curve) for real/fake separation. Classification thresholds then assign each feature as 'Core', 'Usable', 'Unstable High-Discriminability', or 'Unusable'.

Figure 1: Stability ($S_s$) and discriminability ($S_d$) scores of all physical features, categorizing feature quality for synthetic image detection.

Among all candidates, four core features emerge: Laplacian variance, Sobel magnitude mean, Sobel magnitude standard deviation, and LBP entropy, with residual noise variance as a marginal fifth, exhibiting high inter-domain stability and strong real-vs-fake separation.

Figure 2: Cross-dataset density distributions for the core features, showing broad statistical divergence for real (blue) and AI-generated (red) images across major generators.

These findings empirically refute the assumption that deep representation learning itself obviates explicit low-level statistical modeling.

Multimodal Integration: Physical Features in CLIP Training

To operationalize the utility of these descriptors within a vision-language context, the authors augment the standard image captioning workflow (ClipCap) by embedding string-encoded values of the selected physical features into the final textual prompts. For example, an image's learned caption (from ClipCap) is concatenated with a segment like "The physical features are: Laplacian variance: X, Sobel mean: Y, ...".

Figure 3: Overall cross-modal workflow—core feature extraction, feature-to-text encoding, CLIP-based training, and downstream testing.

Both image and text encoders (ViT-L/14 backbone for CLIP with LoRA fine-tuning) thus receive explicitly aligned physical evidence during contrastive and classification training. At inference, only the image encoder and its classifier head are used.

Figure 4: Example image-caption pairs illustrating the qualitative impact of adding physical feature-based text to semantic captions.

Empirical Evaluation

Training and evaluation utilize the million-scale GenImage benchmark, covering eight generator domains (Midjourney, SDv1.4, SDv1.5, ADM, GLIDE, Wukong, VQDM, BigGAN) and additional subsets from UniversalFakeDetect. For each test protocol, 6k real/6k fake images per class are randomly sampled.

The integration of core physical descriptors yields systematic, statistically significant gains in cross-model generalization and absolute accuracy relative to prompt-only baselines (C2pCLIP), and decisively outperforms strong recent generalization-focused competitors (UniFD, FreqNet, FatFormer, etc.).

Figure 5: Distributional analysis demonstrates improved image–text alignment upon inclusion of feature-based captions, as measured by CLIP cosine similarity.

Warm-started CLIP evaluations further show that incorporating feature-based descriptors consistently elevates the cosine similarity gap between real and fake images, indicating enhanced discriminability in shared latent space.

Figure 6: Caption phrasing sensitivity—text beginning with “The physical features are:” achieves measurably better image–text alignment than more generic introductions—across seven major datasets.

Additional controlled studies on prompt phrasing reveal that precise lexical framing ("The physical features are:") further boosts image–text alignment beyond numerical value inclusion alone, reinforcing the non-trivial interplay between statistical prompting and linguistic semantics.

Figure 7: Distributional shift in CLIP similarity scores for real/fake ADM images under different captioning protocols.

Limitations and Future Work

Tokenization length in the CLIP text encoder constrains feature set size and weighting expressiveness. Scalar summarization of physical characteristics may obscure spatially local statistical cues critical for complex forgeries. Interactions between prompt class labels and physical feature text remain insufficiently characterized; their disentanglement via ablation and cross-attention analysis is required. The authors acknowledge the necessity for broader, cross-modal real-world evaluations, including cross-lingual settings, multi-domain forgeries, and spatially resolved feature modeling.

Advances along these directions are foreseen as essential for interpretable, transparent, and robust synthetic artifact detection.

Conclusion

This study introduces an operational paradigm for trustworthy, generalization-robust synthetic image detection: the integration of statistically stable, dataset-agnostic physical descriptors—formally selected by stability and discriminability criteria—into both the training and test pipelines of cross-modal vision-LLMs. The resulting approach achieves state-of-the-art accuracy (up to 99.8%) and generalization across generative models. Incorporating physical feature evidence not only enhances generalization but reduces reliance on semantically brittle language modeling, suggesting a practical route for mitigating hallucination and semantic drift in multimodal reasoning.

The reported framework points to future research on the principled fusion of statistical evidence and language for generative artifact detection, with implications for robust multimodal perception, explainability, and content forensics.