Auditing Generalization in AI-Generated Video Detection: A Six-Control Protocol and the VidAudit Toolkit

Abstract: AI-generated video detection benchmarks such as GenVidBench and AIGVDBench are the de facto leaderboards, yet most evaluation protocols leave uncontrolled confounds that can inflate reported generalization. As an existence proof, a three-feature clip-length classifier reaches a leave-one-generator-out (LOGO) AUC of 0.998 on GenVidBench under unaudited evaluation, while measuring nothing about motion. A 20-paper survey finds none applying all six standard controls that would catch this, so we combine them into an audited protocol and apply it to six representative feature sources (three published detectors and three repurposed signal sources), re-running it cross-dataset on AIGVDBench. The audit both debunks and certifies: the trivial classifier collapses to near chance (0.529), a CLIP baseline is caught carrying dataset identity, and the 2025 forensic detector WaveRep clears the floor at out-of-distribution LOGO AUC 0.996 with chance-level real-vs-real coherence. At a deployable FPR of 0.1%, multiple high-AUC methods fall to single-digit recall and the leaderboard order changes, so we recommend an audited tuple (AUC, above-floor margin, operating-point recall, and calibration) over a single number. As a white-box positive control, we add TemporalSpec (codec motion vectors); via cross-substrate feature fusion (XSFF), a second substrate adds genuine complementarity that survives the audit. We release VidAudit, to our knowledge the largest unified and audited detector collection for this task, providing 14 detectors behind one plugin API, a leaderboard, and Croissant metadata, available at https://github.com/KurbanIntelligenceLab/vidaudit. Together, the protocol and toolkit move evaluation from leaderboard rank toward whether a result measures what it claims.

• The paper introduces a six-control evaluation protocol that mitigates dataset confounds through techniques like canonical re-encoding and cross-dataset testing.
• The VidAudit toolkit standardizes benchmarking across forensic, motion, and multimodal detectors with reproducible, transparent reporting.
• Results reveal that high AUC scores can mask poor deployable recall, underscoring the need for multidimensional metrics and substrate fusion.

Rigorous Evaluation of AI-Generated Video Detection: Protocol Design and Toolkit Deployment

Motivation and Problem Setting

The proliferation of AI-generated video has intensified the need for robust automated detection systems, especially in domains with high stakes such as politics and finance. The paper "Auditing Generalization in AI-Generated Video Detection: A Six-Control Protocol and the VidAudit Toolkit" (2606.31004) addresses critical methodological flaws in current benchmarks for AI-generated video detection. Specifically, many leaderboards fail to control for confounding factors, leading to inflated generalization claims. The authors demonstrate with existence proofs that trivial classifiers exploiting dataset artifacts, such as clip length, can achieve near-perfect AUC scores under standard evaluations—yet measure nothing about genuine generative artifacts.

Audited Six-Control Evaluation Protocol

To systematize robust evaluation, the authors define a protocol comprising six controls:

1. Canonical codec re-encoding: eliminates codec-level fingerprints by transforming all input through a standardized pipeline.
2. Leakage audit against trivial baselines: quantifies the impact of confounds such as frame-count features, ensuring results reflect detection of generative artifacts rather than dataset structure.
3. Real-vs-real coherence probe: measures the lower bound of discrimination attributable to dataset identity, establishing a floor for genuine detection signal.
4. Matched-harness re-training: standardizes readout and data splits to normalize performance comparisons.
5. Multi-seed stability and bootstrap CIs: ensures numerical claims are robust to random seed and sample variability.
6. Cross-dataset evaluation: tests generalization across independently curated benchmarks (GenVidBench, AIGVDBench).

The survey of twenty papers finds none adopting all six controls, with most benchmarks reporting single-metric AUCs highly susceptible to unmeasured confounds.

Existence Proof of Confounds and Audit Impact

The paper establishes the practical consequences of missing controls: a three-feature L2-regularized logistic regression classifier (using raw counts of I-, P-, and B-frames per clip) reaches LOGO AUC 0.998 under unaudited evaluation. However, application of the $K{=}12$ non-I-frame filter and canonical re-encode collapses performance to near chance ($0.529$), demonstrating confound removal.

Figure 1: The audit's verdict at a glance; margins above dataset-identity floor demarcate certified detectors versus those primarily exploiting confounds.

Under this protocol, the CLIP-ViT-B/32 appearance detector achieves nominal LOGO OOD AUC of $0.852$ but sits only $+0.086$ above the real-vs-real coherence floor, indicating its performance is largely driven by dataset identity. In contrast, motion-content detectors such as TemporalSpec (codec motion-vector features) and forensic detectors like WaveRep clear the floor by $+0.189$-$+0.462$ AUC points.

VidAudit Toolkit and Feature Engineering

The VidAudit toolkit implements the protocol as a unified benchmarking suite, supporting fourteen detectors spanning forensic, appearance, motion, codec, fusion, and multimodal-LLM families behind a plugin API. Strong emphasis is placed on modularity, reproducibility, and transparent reporting. The white-box positive control, TemporalSpec, summarizes codec MVs via interpretable temporal and spectral features:

• Tier 0: overall motion magnitude, sparsity, variance, and temporal difference.
• Tier 1: spectral slope, spectral flatness, ACF decay time, first-zero lag, acceleration kurtosis, acceleration skewness, computed per-cell on a $4 \times 4$ macroblock lattice.

These features are theoretically justified via spectral bias in current generative models and empirically validated against human perception, showing robust separation across real and generated data (see supplementary concordance with VBench annotations).

Figure 2: The TemporalSpec pipeline from canonical re-encoding through feature aggregation to white-box classification.

Complementary fusion architectures (XSFF) are explored, combining codec-MV features with deep-frame trajectories (e.g., DINOv2 via ReStraV). Under conditional substrate independence, empirical lifts of $+1.4$ to $+1.6$ AUC points over single-channel baselines are observed on both GenVidBench and AIGVDBench, validating detectable complementarity.

Re-Evaluation of State-of-the-Art Detectors

A rigorous re-evaluation under the protocol reveals profound discrepancies between leaderboard results and deployable accuracy:

• At a deployable FPR of $0.1\%$, only WaveRep (TPR $0.529$0) and ReStraV (TPR $0.529$1) retain meaningful recall; high-AUC methods such as FVMD, RAFT, and TemporalSpec ($0.529$2-$0.529$3 AUC) drop to $0.529$4 recall.
• The leaderboard order changes under operating-point metrics, with CLIP surpassing FVMD and RAFT in recall, despite lower AUC.

  Figure 3: Leaderboard metric AUC versus deployable recall (TPR at FPR $0.529$5); most baselines collapse, highlighting the inadequacy of single-number reporting.

  

  Figure 4: Audited cost-vs-OOD frontier on the matched 27k GenVidBench subset; minimum deployment cost for a certified detector (e.g., TemporalSpec at 14 ms/clip, CPU-only) clearly separates modalities.

Cross-Dataset Replication and Robustness

The audit exposes substantial challenge in detecting recent ($0.529$6-era) generators—single-channel detectors drop sharply on AIGVDBench, but cross-substrate fusion preserves complementarity gains. Robustness sweeps confirm insensitivity to codec variations, spatial rescaling, and quantization, with most degradation in frame-rate drift mitigated via canonical re-encode.

Theoretical and Practical Implications

The results mandate the adoption of multidimensional audited metrics:

• Metric tuple reporting (AUC, margin above floor, deployable recall, calibration error) is essential for operational deployments, replacing the single-metric paradigm.
• Orthogonal substrate fusion is necessary to prevent protocol circumvention.
• Transparent feature engineering and open-source reproducibility are emphasized for trustworthiness and benchmarking.

Figure 5: Detection AUC: Tier~0, Tier~1, and joint, across generators, revealing motion-structure discrimination is distributed and not trivial.

Figure 6: Motion smoothness: spectral and ACF statistics track human rating axes, validating feature interpretability.

Conclusion

This paper establishes that current benchmarks and protocols in AI-generated video detection are fundamentally flawed, often measuring dataset artifacts rather than generative fingerprints. Through stringent controls, audit protocols, and open-source benchmarking, the findings advocate for transition from leaderboard-centric evaluation to empirical certification of detection claims. The VidAudit toolkit and protocol provide the implementation blueprint for such rigorous evaluation. The implications for AI safety and digital media forensics are substantial: practical deployment must ensure genuine signal detection, robust to dataset confounds and adversarial circumvention, with multidimensional, audited reporting. Ensemble strategies and substrate fusion are necessary to ensure resilience against evolving generative models and dataset shifts.