Visual Question Answering

• Visual Question Answering is a multimodal task that fuses image features and text to generate accurate, natural-language answers.
• Modern VQA models employ attention mechanisms, compositional reasoning, and transformer architectures to enhance performance.
• Benchmarks and datasets chart VQA’s evolution from CNN-LSTM baselines to advanced vision–language pre-training approaches.

Visual Question Answering (VQA) is the task of answering a natural-language question about an image by jointly processing visual content and linguistic input. In its canonical form, the input consists of an image $I$ and a question $Q$, and the system must produce an answer $A$, usually a short natural-language response; many formulations treat this as learning a mapping $f:(I,Q)\mapsto A$ or, in classification settings, a distribution $p_\theta(a\mid I,Q)$ over a fixed answer vocabulary (Agrawal et al., 2015). Since its formalization as a large-scale benchmark in 2015, VQA has become a central multimodal problem linking computer vision and natural language processing, and later surveys describe its development from early CNN–LSTM systems through attention mechanisms, compositional reasoning models, vision–language pre-training, and large multimodal LLMs (Pandey et al., 13 Jan 2025).

1. Task formulation and conceptual scope

VQA was introduced as a free-form and open-ended task in which an image and a natural-language question are mapped to an accurate natural-language answer (Agrawal et al., 2015). Two evaluation modalities were established early: open-ended VQA, in which the model outputs any text answer, and multiple-choice VQA, in which the model selects from a provided candidate set. In practice, many systems reformulate VQA as classification over the top $K$ most frequent answers; early baselines in the original VQA benchmark used $K=1000$, covering 82.67% of train+val answers (Agrawal et al., 2015).

A central distinction in the literature is between questions answerable from visual evidence alone and questions that require additional reasoning or knowledge. The original VQA paper already emphasized that questions may require fine-grained recognition, counting, activity recognition, spatial reasoning, commonsense, and domain knowledge (Agrawal et al., 2015). Later work makes this distinction explicit. Knowledge-augmented VQA focuses on questions requiring commonsense knowledge, world knowledge, and reasoning about entities or concepts not explicitly visible in the image (Ghosal et al., 2023). Other formulations treat VQA as a multi-task problem because different question types—such as colour, count, position, and size—behave like distinct but related tasks sharing a common visual domain (Pollard et al., 2020).

The field has also broadened beyond the standard “image plus question” pipeline. Some approaches cast multiple-choice VQA as binary classification over image–question–answer triplets rather than softmax over an answer vocabulary (Jabri et al., 2016). Others reformulate VQA as reading comprehension by converting image descriptions and knowledge into text (Li et al., 2018), as symbolic reasoning over first-order predicate logic (Sethuraman et al., 2021), or as a meta-learning task in which a support set of labeled examples is provided at test time (Teney et al., 2017). These variants do not change the fundamental objective—answering questions about visual content—but they do alter what counts as the primary representational substrate for reasoning.

2. Benchmarks, supervision, and evaluation

The original VQA dataset contains approximately 0.25M images, ~0.76M questions, and ~10M answers, with 204,721 real images from MS COCO and 50,000 abstract scenes made from clipart (Agrawal et al., 2015). Questions were collected with three questions per image or scene from distinct workers, and each question has 10 answers from different workers. Answers are overwhelmingly short: for real images, 89.32% are 1-word, 6.91% are 2-word, and 2.74% are 3-word answers (Agrawal et al., 2015). Human performance with question and image is 83.30% on real images, while human performance with question and caption only is 57.47%, which shows that caption-level information is insufficient for many VQA questions (Agrawal et al., 2015).

The standard open-ended evaluation metric is: $$\text{accuracy}=\min\left(\frac{\#\text{ humans that provided that answer}}{3},1\right),$$ after normalization such as lowercasing, number conversion, and removal of punctuation and articles (Agrawal et al., 2015). This metric is also used in multiple-choice VQA, where each question is paired with 18 answer options (Agrawal et al., 2015). For datasets emphasizing semantic similarity rather than exact match, other metrics appear; for example, DAQUAR introduced modified WUPS, while some specialized domains also report BLEU or Word-Based Semantic Similarity (Zou et al., 2021, Barra et al., 2021).

Subsequent datasets stress different capabilities. Visual7W Telling provides 4 answer choices with human-generated distractors and reports simple accuracy (Jabri et al., 2016). Knowledge-oriented benchmarks include A-OKVQA, a 25K-instance multiple-choice dataset focused on world and commonsense knowledge, and ScienceQA, with approximately 10K science questions with images (Ghosal et al., 2023). Diagnostic reasoning datasets include CLEVR and GQA, both of which are central in logic-based and compositional-reasoning work (Sethuraman et al., 2021). Surveys further organize the dataset landscape into authentic-image datasets, synthetic datasets, diagnostic datasets, and knowledge-based datasets (Pandey et al., 13 Jan 2025, Zou et al., 2021).

3. Architectural paradigms and multimodal learning

Early VQA systems typically combine an image encoder and a question encoder, fuse the resulting representations, and predict an answer class (Agrawal et al., 2015). The original benchmark paper compared bag-of-words, LSTM, and deeper LSTM encoders for questions, together with VGG image features and an MLP classifier; the best reported model on test-dev was deeper LSTM Q + norm I with 57.75% open-ended accuracy and 62.70% multiple-choice accuracy on real images (Agrawal et al., 2015). This family of models established the now-standard decomposition into visual feature extraction, textual feature extraction, multimodal fusion, and answer prediction.

A major line of critique questioned whether the prevalent softmax-over-answers formulation was even the right problem statement for multiple-choice VQA. "Revisiting Visual Question Answering Baselines" reformulated multiple-choice VQA as binary classification over image–question–answer triplets, with scoring functions such as

$y=\sigma(Wx_{iqa}+b)$

for a linear model or

$y=\sigma\!\left(W_2\max(0,W_1x_{iqa})+b\right)$

for an MLP, trained with binary logistic loss (Jabri et al., 2016). On Visual7W, this paper reported 67.1% for MLP(A,Q,I), surpassing several more complex attention-based baselines, and showed that the binary formulation was substantially better than softmax when the architecture was otherwise held fixed (Jabri et al., 2016).

Attention-based architectures then became a dominant design principle. "Task-driven Visual Saliency and Attention-based Visual Question Answering" added a saliency-like pre-selection over overlapping regions using BiLSTM, and replaced scalar spatial attention with an element-wise multiplication–based attention mechanism (Lin et al., 2017). On COCO-VQA validation, the full SalAtt model achieved 60.62 overall accuracy with ResNet features, outperforming holistic, traditional attention, and convolution-based pre-selection variants reported in the same paper (Lin et al., 2017). Another attempt to inject additional structure into the question side is "VQABQ: Visual Question Answering by Basic Questions", which decomposes a main question into basic questions using a LASSO formulation,

$Q$0

then concatenates selected basic questions to the original question before co-attention-based answering (Huang et al., 2017).

An orthogonal strand treats VQA as an explicitly multi-task problem. By reformatting COCO-QA and DAQUAR into grouped samples containing one question per type and training shared encoders with task-specific heads, "Visual Question Answering as a Multi-Task Problem" reported increases of 5–9% over single-task formatting and substantially faster convergence, with multi-task models converging in < 50 epochs versus ~200 epochs for the single-task counterparts (Pollard et al., 2020). This suggests that question types can be operationalized not only as evaluation categories but also as supervision structure.

4. Reasoning, knowledge, and alternative formalizations

As benchmarks increasingly emphasized questions requiring more than direct visual recognition, external knowledge and structured reasoning became central. "Language Guided Visual Question Answering" introduced LG-VQA, in which a multimodal model is augmented with guides such as rationales, captions, scene graphs, and object summaries (Ghosal et al., 2023). In A-OKVQA validation, guidance raised CLIP from 68.30% to 75.98% and BLIP-2 from 75.02% to 79.83%, with especially large gains on hard questions and “Why” questions (Ghosal et al., 2023). A different strategy, "Knowledge Detection by Relevant Question and Image Attributes in Visual Question Answering", retrieves ConceptNet knowledge only for image objects that are relevant to the question, rather than using image-only or question-only retrieval; on OK-VQA validation it reported 0.5963 for the image–question co-attention variant versus approximately 0.516 for image-only or question-only attention baselines (Ahir et al., 2023).

Alternative representations challenge the assumption that VQA must always be solved by direct multimodal fusion. "Visual Question Answering as Reading Comprehension" converts image descriptions and optional external facts into natural language and then applies a reading-comprehension model, specifically QANet, to answer the question (Li et al., 2018). On Visual Genome QA with ground-truth descriptions, the open-ended model achieved 44.8% overall, outperforming the cited prior best of 39.3%; on Visual7W Telling with ground-truth descriptions, the multiple-choice model achieved 69.8% overall (Li et al., 2018). The key premise is that once images and knowledge are expressed as text, VQA becomes a textual inference problem rather than a heterogeneous feature-fusion problem.

Logic-based formulations push this rethinking further. "Visual Question Answering based on Formal Logic" converts scene graphs into predicates such as attribute/3 and relation/3, translates questions into first-order logic clauses with a BART-based semantic parser, and answers via Prolog satisfiability checks (Sethuraman et al., 2021). The method achieved 99.6% on CLEVR, 99.1% on CLEVR when trained on only 10% of the training data, 99.5% on both splits of CLEVR-CoGenT, and 93.1% on the consistent subset of GQA validation (Sethuraman et al., 2021). Meta-learning offers another nonstandard formulation: "Visual Question Answering as a Meta Learning Task" introduces support sets available at test time and combines dynamic parameters with dynamic prototypes so that the system can produce novel answers not seen during training (Teney et al., 2017). This paper reports improved answer recall, especially for rare answers, and qualitatively distinct behavior relative to static-vocabulary VQA baselines (Teney et al., 2017).

5. Bias, grounding, and the limits of benchmark accuracy

A persistent controversy in VQA concerns the extent to which benchmark performance reflects genuine grounding and reasoning rather than exploitation of dataset biases. The most direct empirical critique is "Revisiting Visual Question Answering Baselines", which showed on Visual7W that MLP(A)—a model using answers only—reached 52.9%, close to an earlier LSTM(Q,I) baseline at 52.1%, and that MLP(A,Q) reached 58.5%, outperforming many vision-based systems (Jabri et al., 2016). On VQA Real Multiple Choice, the same simple binary MLP obtained 65.2% overall, comparable to several sophisticated attention and co-attention models (Jabri et al., 2016). The paper’s error analysis further found weak grounding for color, shape, counting, and spatial reasoning, and only modest gains from stronger CNN backbones, concluding that current VQA systems were not significantly better than systems designed to exploit dataset biases (Jabri et al., 2016).

The original VQA benchmark also documented that language priors are strong. Humans answering questions without seeing the image achieved 40.81% overall on real images, and 67.60% on yes/no questions; with question plus caption, they achieved 57.47% overall, still far below the 83.30% obtained with question plus image (Agrawal et al., 2015). This establishes both that question priors are informative and that captions do not contain enough targeted detail for many VQA queries. Balanced datasets and challenge splits were introduced precisely because models could otherwise answer many questions by exploiting priors alone (Zou et al., 2021, Faria et al., 2023).

This literature also cautions against equating architectural complexity with reasoning. Attention, memory, and module mechanisms can improve performance, but the surveys repeatedly note that bias reduction, out-of-distribution generalization, grounding quality, and interpretability remain unresolved (Faria et al., 2023, Pandey et al., 13 Jan 2025). Synthetic and diagnostic datasets such as CLEVR, GQA, VQA-CP, and GQA-OOD emerged to isolate these issues more sharply (Zou et al., 2021, Huynh et al., 7 Jan 2025). A plausible implication is that VQA accuracy is best interpreted alongside diagnostics for grounding, compositionality, robustness, and knowledge use rather than as a single scalar measure of multimodal intelligence.

6. Specialized domains, nonstandard sensing, and evolving directions

VQA has expanded into domains where the visual input, the knowledge requirements, or both differ substantially from the original COCO-based setting. "CS-VQA: Visual Question Answering with Compressively Sensed Images" asks whether VQA remains possible when the model receives compressive measurements $Q$1 satisfying

$Q$2

rather than a conventional image $Q$3 (Huang et al., 2018). The paper concludes that VQA is indeed solvable in the compressed domain, with nominal degradation relative to standard images, and that accuracy can be recovered when state-of-the-art deep neural networks for compressive-sensing reconstruction are inserted before the VQA pipeline (Huang et al., 2018). This directly connects VQA to sub-Nyquist and resource-constrained sensing.

Application-focused surveys show that the task has also moved into medical imaging, assistive technology for visually impaired users, surveillance, cultural heritage, advertising, and education (Barra et al., 2021). The most mature application area is medical VQA, with datasets such as ImageCLEF 2018 VQA-Med, ImageCLEF 2019 VQA-Med, VQA-RAD, and PathVQA; the survey reports, for example, that a co-attention-based CNN plus word-embedding model on VQA-RAD achieved approximately 74.1% accuracy (Barra et al., 2021). For visually impaired users, VizWiz provides 31,000 visual questions from blind people, reflecting real-world conditions such as blur, poor framing, and privacy-sensitive content (Barra et al., 2021). These specialized settings force the field to confront issues—domain shift, annotation scarcity, latency, privacy, interpretability—that are less visible in general-purpose benchmarks.

Recent surveys describe the present phase of VQA as being shaped by transformer architectures, multimodal pre-training, and large multimodal LLMs (Pandey et al., 13 Jan 2025, Huynh et al., 7 Jan 2025). They also describe a broadening of the evaluation landscape to include OCR-heavy VQA, knowledge-based reasoning, video understanding, multilingual and code-mixed VQA, scientific diagrams, charts, and mathematics (Huynh et al., 7 Jan 2025, Faria et al., 2023). At the same time, the field continues to revisit foundational questions: whether answers should be extracted or generated, whether reasoning should be implicit or explicit, whether external knowledge should be retrieved or baked into model parameters, and how much of current performance reflects true visual understanding. The cumulative literature suggests that VQA is no longer a single benchmarked task but a family of multimodal reasoning problems whose difficulty is determined as much by dataset design and supervision structure as by model architecture.