M⁴-BLIP: Face-Enhanced Forgery Detection

• The paper introduces a unified framework that fuses global image features with face-enhanced local cues and textual information to detect multi-modal manipulations.
• It employs fine-grained contrastive alignment and multi-stage fusion via a BLIP-2 backbone and Q-Former, enhancing localization of subtle forgeries.
• Experimental results show superior performance metrics (AUC, EER, text grounding) compared to state-of-the-art methods on mixed forgery benchmarks.

M⁴-BLIP (Multi-Modal Media Manipulation Detection through Face-Enhanced Local Analysis) is a unified framework developed to address the challenge of detecting subtle manipulations in image–text pairs, with a particular emphasis on forgeries localized to specific regions such as faces and localized textual edits. Unlike prior approaches that primarily focus on global modalities alignment, M⁴-BLIP fuses global and local (facial) image features as well as textual information, leveraging the BLIP-2 backbone for fine-grained feature extraction. The architecture incorporates a face-enhanced local prior, a multi-stage alignment and fusion mechanism, and integration with LLMs to improve interpretability and detection accuracy in complex, multi-modal manipulation scenarios (Wu et al., 1 Dec 2025).

1. Motivation and Problem Landscape

The prevalence of manipulated multi-modal content—particularly “fake news” comprising doctored images and misleading textual narratives—poses significant risks to information integrity. Existing detection approaches, notably those based on global feature alignment (e.g., CLIP), often fail to identify manipulations that are local and subtle, such as face swaps in images or minor text modifications.

M⁴-BLIP is predicated on three key insights:

• Local features, especially facial regions, often retain strongest manipulation cues.
• BLIP-2, pre-trained on fine-grained recognition tasks, surpasses CLIP in local feature sensitivity.
• Explicitly fusing global, local facial, and text modalities in a unified model enhances detection efficacy for both image and text forgeries.

2. Architectural Components

M⁴-BLIP comprises three principal blocks: (a) global/local feature extraction, (b) fine-grained contrastive alignment, and (c) multi-modal fusion with cross-attention.

2.1 BLIP-2-based Feature Extractor

• Global image features: Encoded via $E_v$ (ViT-g/14) yielding $e^v = E_v(I)$.
• Text features: Encoded by $E_t$ (BLIP-2 Q-Former + LLM), yielding $e^t = E_t(T)$.
• Face-enhanced local prior: A face detector extracts the main face region $I' \subset I$, which is processed using a deepfake classifier $E_d$ (EfficientNet-b4) as $e^d = E_d(I')$.

2.2 Fine-grained Contrastive Alignment (FCA)

• Aligns genuine and manipulated image/text pairs using a symmetric InfoNCE loss in both modalities.

2.3 Multi-modal Local-and-Global Fusion (MLGF) and Cross-Attention

• Q-Former fusions:
  • $f^v = Q(e^v, T)$ (global–text fusion)
  • $f^d = Q(e^d, T)$ (face-local–text fusion)
• Cross-attention: Integrates $f^v$ and $f^d$ to obtain the final fused feature $f$, which feeds subsequent classification and grounding tasks.

2.4 Block Diagram (word sequence)

• I –– $E_v$ → $e^v$ –─┐
• I –– face crop – $E_d$ → $e^d$ –┘ ├— FCA —→ Q-Former → $f^v$ ├───────────────────── Q-Former → $f^d$ └───────── cross-attention → $f$ → classification and grounding heads

Feature Mapping:

$$f_{\rm img} = E_v(I),\quad f_{\rm txt} = E_t(T),\quad f_{\rm face} = E_d(I').$$

3. Alignment, Fusion Mechanisms, and Loss Functions

3.1 Contrastive Alignment

A symmetric InfoNCE loss is used to tightly link genuine pairs and distinguish forgeries. For $K$ negative samples: $$\mathcal{L}_{v2t} = -\mathbb{E}_{(I^+,T^+)} \log \frac{\exp\bigl(S(I^+,T^+)/\tau\bigr)} {\sum_{k=1}^K \exp\bigl(S(I^+,T_k^-)/\tau\bigr)},$$

$$\mathcal{L}_{t2v} = -\mathbb{E}_{(I^+,T^+)} \log \frac{\exp\bigl(S(T^+,I^+)/\tau\bigr)} {\sum_{k=1}^K \exp\bigl(S(T^+,I_k^-)/\tau\bigr)},$$

$$\mathcal{L}_{ITC} = \tfrac12(\mathcal{L}_{v2t} + \mathcal{L}_{t2v}),\quad S(I,T) = h_v(e^v)^\top h_t(e^t).$$

3.2 Local/Global Fusion and Supervision

Fusions performed via Q-Former with separate supervision: $f^v = Q(e^v, T),\quad f^d = Q(e^d, T)$ Binary supervision on $f^d$: $$\mathcal{L}_d = \mathbb{E}_{(I,T)\sim P}\; \mathrm{H}(C_b(f^d), L_{\rm bin})$$ Multi-label type supervision on $f^v$: $$\mathcal{L}_v = \mathbb{E}_{(I,T)\sim P}\; \mathrm{H}(C_m(f^v), L_{\rm mul})$$ Cross-attended fusion: $$f = \mathrm{CrossAttention}(Q=f^d,K=f^v,V=f^v),\quad \mathrm{CrossAttention}(Q,K,V)=\mathrm{Softmax}(K^\top Q/\sqrt{D}) V$$

Class heads: $$\mathcal{L}_{BLC} = \mathbb{E}\,\mathrm{H}(C_b(f_{\rm cls}),L_{\rm bin}),\quad \mathcal{L}_{MLC} = \mathbb{E}\,\mathrm{H}(C_m(f_{\rm cls}),L_{\rm mul}),\quad \mathcal{L}_{TMG} = \mathbb{E}\,\mathrm{H}(D_t(f_{\rm tok}),L_{\rm tok})$$ Total loss: $$\mathcal{L} = \mathcal{L}_{ITC} + \mathcal{L}_d + \mathcal{L}_v + \mathcal{L}_{BLC} + \mathcal{L}_{MLC} + \mathcal{L}_{TMG}$$

4. Face-Enhanced Local Analysis

M⁴-BLIP deploys a face detector to identify the main facial region $I'$, which is then processed by a pre-trained deepfake classifier $E_d$. This embedding $e^d$ is explicitly fused via the Q-Former and supervised by the binary classification loss $\mathcal{L}_d$, emphasizing facial features likely to exhibit manipulation artifacts, such as anomalous eye blinks, mouth deformations, and edge blending artifacts. The architecture introduces no exotic layers beyond the Q-Former, with the process pipeline "face-as-prior → deepfake extractor → fusion" being structurally novel.

5. Integration with LLMs

The final fused feature $f$ is projected via a fully connected head $$h_{\text{LLM}}: \mathbb{R}^D \rightarrow \mathbb{R}^{d_{LLM}}$$. Instruction strings and image features, following a MiniGPT-4-like prompt template, are input to LLMs (e.g., Vicuna, LLaMA) for generating natural language explanations:

###Human:(Img)(ImageFeature)(/Img)(Instruction) ###Assistant:

Examples of instruction include manipulation classification or type queries, with the LLM outputting concise and human-interpretable explanations conditioned on the feature embedding.

6. Experimental Results

Datasets, Manipulations, and Metrics

Experiments are conducted on the DGM⁴ benchmark, incorporating image-based face swap (FS) and attribute change (FA), and text-based swap (TS) and attribute (TA) manipulations, annotated for binary real/fake, manipulation type, and token-level text grounding.

Evaluation metrics include:

• Binary: Area Under Curve (AUC), Equal Error Rate (EER), Accuracy (ACC)
• Multi-label: mean Average Precision (mAP), class-wise F1 (CF1), overall F1 (OF1)
• Text grounding: Precision, Recall, F1

Performance vs. SOTA

Model	Binary AUC↑	EER↓	ACC↑	mAP↑	OF1↑	Text F1↑
HAMMER	93.19	14.10	86.39	86.22	80.37	71.35
M⁴-BLIP	94.10	13.25	86.92	87.97	80.72	76.87

Single-modal ablations reveal that end-to-end M⁴-BLIP training also improves standalone image-only and text-only detection.

Ablation Studies

• Loss ablation: Removing $\mathcal{L}_{ITC}$ yields AUC drop from 94.10 to 93.81; removing local/global supervision ($\mathcal{L}_d,\mathcal{L}_v$) reduces AUC to 93.85; full loss yields best AUC (94.10).
• Feature ablation: Using only global features yields AUC 89.96; local-only yields 81.73; combination restores performance to AUC 94.10.

Qualitative Visualization

• Attention maps: On real faces, attention is distributed globally; on forgeries, the model attention intensifies over manipulated regions.
• Text grounding: Manipulated tokens are accentuated (red), non-manipulated tokens are suppressed (blue).
• LLM outputs: Fine-tuned LLM generates faithful manipulation rationales; non-finetuned LLM yields less interpretable captions.

7. Discussion and Prospects

M⁴-BLIP quantitatively and qualitatively demonstrates that incorporating a face-enhanced local prior alongside a BLIP-2 backbone confers significant robustness to localized manipulations in multi-modal settings. Notable limitations include dependence on robust face detection (susceptibility to failure on occluded or absent facial images) and increased computational overhead due to the deployment of a deepfake subnetwork and Q-Former.

Anticipated directions for future research include:

• Extending local priors to other informative objects (logos, watermarks)
• Adoption of dynamic negative mining in the contrastive alignment stage
• Deeper, potentially joint, integration with larger LLMs encompassing both manipulation detection and natural language explanation (Wu et al., 1 Dec 2025).