FKA-Owl: Advancing Multimodal Fake News Detection through Knowledge-Augmented LVLMs
Abstract: The massive generation of multimodal fake news involving both text and images exhibits substantial distribution discrepancies, prompting the need for generalized detectors. However, the insulated nature of training restricts the capability of classical detectors to obtain open-world facts. While Large Vision-LLMs (LVLMs) have encoded rich world knowledge, they are not inherently tailored for combating fake news and struggle to comprehend local forgery details. In this paper, we propose FKA-Owl, a novel framework that leverages forgery-specific knowledge to augment LVLMs, enabling them to reason about manipulations effectively. The augmented forgery-specific knowledge includes semantic correlation between text and images, and artifact trace in image manipulation. To inject these two kinds of knowledge into the LVLM, we design two specialized modules to establish their representations, respectively. The encoded knowledge embeddings are then incorporated into LVLMs. Extensive experiments on the public benchmark demonstrate that FKA-Owl achieves superior cross-domain performance compared to previous methods. Code is publicly available at https://liuxuannan.github.io/FKA_Owl.github.io/.
- Self-rag: Learning to retrieve, generate, and critique through self-reflection. In ICLR.
- Improving language models by retrieving from trillions of tokens. In ICML.
- Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.
- Exploring the role of visual content in fake news detection. Disinformation, Misinformation, and Fake News in Social Media: Emerging Research Challenges and Opportunities.
- Cross-modal ambiguity learning for multimodal fake news detection. In WWW.
- Causal intervention and counterfactual reasoning for multi-modal fake news detection. In ACL.
- Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023).
- Dina ElBoghdady. 2013. Market quavers after fake ap tweet says obama was hurt in white house explosions. The Washington Post.
- Pizzagate: From rumor, to hashtag, to gunfire in dc. The Washington Post.
- Fakeflow: Fake news detection by modeling the flow of affective information. In EACL.
- Imagebind: One embedding space to bind them all. In CVPR.
- Anomalygpt: Detecting industrial anomalies using large vision-language models. In AAAI.
- Learn over past, evolve for future: Forecasting temporal trends for fake news detection. In ACL.
- Faking fake news for real fake news detection: Propaganda-loaded training data generation. In ACL.
- Winclip: Zero-/few-shot anomaly classification and segmentation. In CVPR.
- Knowledge-augmented reasoning distillation for small language models in knowledge-intensive tasks. In NeurIPS.
- The power of scale for parameter-efficient prompt tuning. In EMNLP.
- Exploring disentangled content information for face forgery detection. In ECCV.
- Detect rumors in microblog posts for low-resource domains via adversarial contrastive learning. In Findings of NAACL.
- Focal loss for dense object detection. In ICCV.
- Ra-dit: Retrieval-augmented dual instruction tuning. In ICLR.
- Visual news: Benchmark and challenges in news image captioning. In EMNLP.
- Visual instruction tuning. In NeurIPS.
- Spatial-phase shallow learning: rethinking face forgery detection in frequency domain. In CVPR.
- V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV.
- Divide-and-conquer: Post-user interaction network for fake news detection on social media. In WWW.
- Salman Bin Naeem and Rubina Bhatti. 2020. The covid-19 ‘infodemic’: a new front for information professionals. Health Information & Libraries Journal.
- Mdfend: Multi-domain fake news detection. In CIKM.
- Improving fake news detection of influential domain via domain-and instance-level transfer. In COLING.
- Improving fake news detection by using an entity-enhanced framework to fuse diverse multimodal clues. In ACM MM.
- Hierarchical multi-modal contextual attention network for fake news detection. In SIGIR.
- Thinking in frequency: Face forgery detection by mining frequency-aware clues. In ECCV.
- Generalized intersection over union: A metric and a loss for bounding box regression. In CVPR.
- Joshua Robinson and David Wingate. 2022. Leveraging large language models for multiple choice question answering. In ICLR.
- Detecting and grounding multi-modal media manipulation. In CVPR.
- Prompting large language models with answer heuristics for knowledge-based visual question answering. In CVPR.
- Zoom out and observe: News environment perception for fake news detection. In ACL.
- Article reranking by memory-enhanced key sentence matching for detecting previously fact-checked claims. In ACL-IJCNLP.
- Kaede Shiohara and Toshihiko Yamasaki. 2022. Detecting deepfakes with self-blended images. In CVPR.
- Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355.
- Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
- Attention is all you need. In NeurIPS.
- Fake news detection via knowledge-driven multimodal graph convolutional networks. In ICMR.
- Multimodal fusion with co-attention networks for fake news detection. In Findings of ACL-IJCNLP.
- Bootstrapping multi-view representations for fake news detection. In AAAI.
- Learning self-consistency for deepfake detection. In ICCV.
- A survey of large language models. arXiv preprint arXiv:2303.18223.
- Face forensics in the wild. In CVPR.
- Knowledge-augmented methods for natural language processing. In ACL.
- Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592.
- Generalizing to the future: Mitigating entity bias in fake news detection. In SIGIR.
- Memory-guided multi-view multi-domain fake news detection. TKDE.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.