DomainCQA: Crafting Expert-Level QA from Domain-Specific Charts

Abstract: Chart Question Answering (CQA) benchmarks are essential for evaluating the capability of Multimodal LLMs (MLLMs) to interpret visual data. However, current benchmarks focus primarily on the evaluation of general-purpose CQA but fail to adequately capture domain-specific challenges. We introduce DomainCQA, a systematic methodology for constructing domain-specific CQA benchmarks, and demonstrate its effectiveness by developing AstroChart, a CQA benchmark in the field of astronomy. Our evaluation shows that current MLLMs face fundamental challenges in vision-language alignment and domain adaptation, highlighting a critical gap in current benchmarks. By providing a scalable and rigorous framework, DomainCQA enables more precise assessment and improvement of MLLMs for domain-specific applications.