2000 character limit reached
Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs
Published 23 Apr 2024 in cs.CV, cs.AI, cs.CL, and cs.MM | (2404.15406v2)
Abstract: Multimodal LLMs are the natural evolution of LLMs, and enlarge their capabilities so as to work beyond the pure textual modality. As research is being carried out to design novel architectures and vision-and-language adapters, in this paper we concentrate on endowing such models with the capability of answering questions that require external knowledge. Our approach, termed Wiki-LLaVA, aims at integrating an external knowledge source of multimodal documents, which is accessed through a hierarchical retrieval pipeline. Relevant passages, using this approach, are retrieved from the external knowledge source and employed as additional context for the LLM, augmenting the effectiveness and precision of generated dialogues. We conduct extensive experiments on datasets tailored for visual question answering with external data and demonstrate the appropriateness of our approach.
- Flamingo: a Visual Language Model for Few-Shot Learning. In NeurIPS, 2022.
- With a Little Help from your own Past: Prototypical Memory Networks for Image Captioning. In ICCV, 2023.
- The (R)Evolution of Multimodal Large Language Models: A Survey. arXiv preprint arXiv:2402.12451, 2024.
- VisualGPT: Data-Efficient Adaptation of Pretrained Language Models for Image Captioning. In CVPR, 2022.
- Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions? In EMNLP, 2023.
- Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality, 2023.
- Scaling Instruction-Finetuned Language Models. arXiv preprint arXiv:2210.11416, 2022.
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv preprint arXiv:2305.06500, 2023.
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394, 2023.
- Retrieval Augmented Language Model Pre-Training. In ICML, 2020.
- LoRA: Low-Rank Adaptation of Large Language Models. arXiv preprint arXiv:2106.09685, 2021.
- Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities. In ICCV, 2023a.
- REVEAL: Retrieval-Augmented Visual-Language Pre-Training With Multi-Source Multimodal Knowledge Memory. In CVPR, 2023b.
- Language Is Not All You Need: Aligning Perception with Language Models. arXiv preprint arXiv:2302.14045, 2023.
- Unsupervised Dense Information Retrieval with Contrastive Learning. arXiv preprint arXiv:2112.09118, 2021.
- Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answering. In SIGIR, 2021.
- Active Retrieval Augmented Generation. arXiv preprint arXiv:2305.06983, 2023.
- Billion-Scale Similarity Search with GPUs. IEEE Trans. on Big Data, 7(3):535–547, 2019.
- Grounding Language Models to Images for Multimodal Inputs and Outputs. In ICML, 2023.
- Cross-modal Retrieval for Knowledge-based Visual Question Answering. arXiv preprint arXiv:2401.05736, 2024.
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv preprint arXiv:2301.12597, 2023a.
- Evaluating Object Hallucination in Large Vision-Language Models. arXiv preprint arXiv:2305.10355, 2023b.
- Improved Baselines with Visual Instruction Tuning. arXiv preprint arXiv:2310.03744, 2023a.
- Visual Instruction Tuning. In NeurIPS, 2023b.
- LLaVA-NeXT: Improved reasoning, OCR, and world knowledge, 2024.
- MMBench: Is Your Multi-modal Model an All-around Player? arXiv preprint arXiv:2307.06281, 2023c.
- Ok-VQA: A Visual Question Answering Benchmark Requiring External Knowledge. In CVPR, 2019.
- Encyclopedic VQA: Visual Questions About Detailed Properties of Fine-Grained Categories. In ICCV, 2023.
- WebGPT: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.
- Training Language Models to Follow Instructions with Human Feedback. In NeurIPS, 2022.
- Safe-CLIP: Removing NSFW Concepts from Vision-and-Language Models. arXiv preprint arXiv:2311.16254, 2024.
- SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM. arXiv preprint arXiv:2403.04735, 2024.
- Learning Transferable Visual Models from Natural Language Supervision. In ICML, 2021.
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR, 21(140):1–67, 2020.
- SmallCap: Lightweight Image Captioning Prompted With Retrieval Augmentation. In CVPR, 2023.
- Retrieval-Augmented Transformer for Image Captioning. In CBMI, 2022.
- Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation. In CVPR, 2023.
- A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge. In ECCV, 2022.
- Learning to Summarize with Human Feedback. In NeurIPS, 2020.
- Stanford Alpaca: An Instruction-Following LLaMA Model, 2023.
- LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971, 2023.
- Multimodal Few-Shot Learning with Frozen Language Models. In NeurIPS, 2021.
- Benchmarking Representation Learning for Natural World Image Collections. In CVPR, 2021.
- Uniir: Training and Benchmarking Universal Multimodal Information Retrievers. arXiv preprint arXiv:2311.17136, 2023.
- Google Landmarks Dataset v2 - A Large-Scale Benchmark for Instance-Level Recognition and Retrieval. In CVPR, 2020.
- Grounding Language Models for Visual Entity Recognition. arXiv preprint arXiv:2402.18695, 2024.
- MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. arXiv preprint arXiv:2311.16502, 2023.
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arXiv preprint arXiv:2304.10592, 2023.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.