CMTA: Leveraging Cross-Modal Temporal Artifacts for Generalizable AI-Generated Video Detection

Abstract: The proliferation of advanced AI video synthesis techniques poses an unprecedented challenge to digital video authenticity. Existing AI-generated video (AIGV) detection methods primarily focus on uni-modal or spatiotemporal artifacts, but they overlook the rich cues within the visual-textual cross-modal space, especially the temporal stability of semantic alignment. In this work, we identify a distinctive fingerprint in AIGVs, termed cross-modal temporal artifact (CMTA). Unlike real videos that exhibit natural temporal fluctuations in cross-modal alignment due to semantic variations, AIGVs display unnaturally stable semantic trajectories governed by given input prompts. To bridge this gap, we propose the CMTA framework, a cross-modal detection approach that captures these unique temporal artifacts through joint cross-modal embedding and multi-grained temporal modeling. Specifically, CMTA leverages BLIP to generate frame-level image captions and utilizes CLIP to extract corresponding visual-textual representations. A coarse-grained temporal modeling branch is then designed to characterize temporal fluctuations in cross-modal alignment with a GRU. In parallel, a fine-grained branch is constructed to capture intricate inter-frame variations from integrated visual-textual features with a Transformer encoder. Extensive experiments on 40 subsets across four large-scale datasets, including GenVideo, EvalCrafter, VideoPhy, and VidProM, validate that our approach sets a new state-of-the-art while exhibiting superior cross-generator generalization. Code and models of CMTA will be released at https://github.com/hwang-cs-ime/CMTA

• The paper introduces a novel CMTA framework that leverages cross-modal temporal artifacts to robustly detect AI-generated videos.
• It employs dual-branch temporal modeling via a GRU and a Transformer to integrate BLIP-generated captions with CLIP embeddings.
• Experimental results show significant AP/AUC improvements over 13 baselines, demonstrating high generalizability across diverse datasets.

Cross-Modal Temporal Artifacts for Robust AI-Generated Video Detection

Introduction

The proliferation of high-fidelity AI-generated video synthesis has escalated concerns about digital media integrity in both research and societal applications. Conventional detection pipelines targeting AI-generated videos (AIGVs) largely exploit either uni-modal visual cues or low-level spatiotemporal inconsistencies. However, these approaches under-address high-level semantic and cross-modal alignment phenomena, particularly those stemming from the distinct generative paradigms of current video synthesis models. "CMTA: Leveraging Cross-Modal Temporal Artifacts for Generalizable AI-Generated Video Detection" (2605.00630) introduces a paradigm that identifies a unique fingerprint of AIGVs: the cross-modal temporal artifact (CMTA), characterized by the abnormally stable temporal evolution of visual-textual alignment under fixed semantic prompts.

Motivation: Cross-Modal Temporal Stability as an Indicator of AI Generativity

AIGVs are typically generated from static or minimally varying prompts. While appearing visually dynamic, the underlying semantic structure remains consistent across time. In contrast, real-world videos exhibit natural fluctuations in high-level semantics due to evolving scenes and uncontrolled capture conditions. This creates pronounced, measurable differences in the temporal regularity of cross-modal alignment (e.g., image caption/CLIP embedding similarity).

Figure 1: Illustration of stochastic cross-modal similarity dynamics in real videos versus the unnaturally stable alignment observed in AIGVs.

The core insight is that frame-wise high-level visual semantics and their corresponding generated captions in AIGVs show minimal temporal variance. This "frozen semantics" effect produces distinctive temporal signals at the cross-modal level that are orthogonal to spatial or purely visual artifacts, offering a robust axis for forensic detection.

CMTA Architecture: Multi-Grained Temporal Modeling of Cross-Modal Dynamics

The CMTA framework consists of four stages: frame-wise image captioning, cross-modal feature extraction, temporal modeling at multiple granularities, and classification. The figure below summarizes this architecture:

Figure 2: Overview of the CMTA pipeline, combining BLIP-based caption generation, visual-textual encoding with CLIP, dual-branch temporal dynamics modeling, and classification.

1. Frame-wise Captioning and Feature Extraction: For sampled video frames, image captions are generated via BLIP. The captions and original frames are encoded with CLIP, yielding temporally aligned feature trajectories in a joint semantic space.
2. Coarse-Grained Temporal Modeling (GRU Branch): CMTA computes the frame-wise cosine similarity between visual and textual embeddings, producing a temporal sequence encoding cross-modal semantic alignment. This sequence is processed by a GRU to model the global evolution, with AIGVs typically resulting in highly stable trajectories and real videos exhibiting significant fluctuations.

   Figure 3: GRU cell structure for capturing temporal evolutions in cross-modal similarity.

3. Fine-Grained Temporal Modeling (Transformer Branch): Visual and textual features for each frame are concatenated and passed through a Transformer encoder with learned positional embeddings, enabling the model to capture subtle local inter-frame dependencies and variations.
4. Classification: The coarse- and fine-grained representations are concatenated and fed to an MLP for binary classification (real or AI-generated).

Experimental Results

Empirical validation is conducted across 40 subsets on four large-scale testing benchmarks: GenVideo, EvalCrafter, VideoPhy, and VidProM. CMTA decisively outperforms 13 strong baseline methods, including recent physics-driven detectors, temporal consistency approaches, and cross-modal baselines. Key highlights include:

• GenVideo: CMTA achieves mean AP/AUC of 98.74%/99.10%, a 6.58% AP improvement over the best prior method (D3). It sustains near-perfect accuracy (97.39%), with margins exceeding 9% over the previous state-of-the-art.
• EvalCrafter: CMTA secures 99.73% AP/AUC, outperforming the strongest competitor by at least 4.1%.
• VideoPhy and VidProM: High cross-benchmark generalization is demonstrated, with the framework robustly handling physically inconsistent motion and real-prompt, open-domain content.

Ablation studies verify the necessity of both visual and textual modalities and the dual-branch (coarse/fine-grained) temporal modeling. The framework attains optimal results when using BLIP-base for captioning and CLIP-ViT-base-patch32 as the shared embedding encoder. Further, t-SNE analysis of the penultimate-layer features shows linearly separable representations for real versus synthetic distributions across all generator domains:

Figure 4: t-SNE visualizations—CMTA's features cluster real and AI-generated video classes with large inter-class margins.

Theoretical and Practical Implications

CMTA's findings have several theoretical and applied ramifications. The consistent stability of high-level semantics in text-driven generative models is revealed as a systematic weakness, providing an axis for robust forensics orthogonal to pixel-level or artifact-based methods. Its performance across in-the-wild and challenging physics-violation benchmarks indicates resilience against spatial artifact suppression, compression, and evolving visual generation fidelity.

Practically, the modular use of frozen vision-LLMs enables plug-and-play adaptation to future models and data without adversarial overfitting. As generative systems trend toward prompt-conditioned, semantically stable synthesis (e.g., prompt-consistent avatar videos or cinematic content), CMTA-style temporal artifact detectors may set the standard for content authentication.

Future Directions

Several directions are emergent:

• Extending cross-modal artifact modeling to multimodal or multi-prompt video generation (e.g., LLM-guided direction, scene transitions).
• Exploiting the temporal curvature or higher-order dynamics of the semantic trajectory itself.
• Adversarially-hardening future AIGV systems to inject pseudo-random cross-modal variation and evade detection, motivating co-evolution of detector architectures.
• Combining CMTA with physics-based or causality-based violation detection for completeness.

Conclusion

CMTA establishes cross-modal temporal artifact modeling as a high-precision, high-recall paradigm for AIGV detection. By leveraging the intrinsic stability of visual-textual alignment imposed by generative prompts, it delivers robust performance across generators, content domains, and artifact suppression strategies. The framework's architecture and findings provide a foundation for next-generation forensic tools as AI-driven video synthesis continues to advance.