MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models

Abstract: Evaluation benchmarks are essential for assessing vision-LLMs (VLMs), but most multimodal benchmarks are static, making them vulnerable to temporal staleness, data contamination, and costly maintenance. We present MMBench-Live, a continuously evolving multimodal benchmark built by a multi-agent-driven automated pipeline. Our framework treats benchmark evolution as task-guided dataset construction, integrating structured benchmark specification, feedback-controlled real-time data acquisition, and verifiable QA generation with executable reasoning. To maintain cross-version comparability, we introduce a distribution-consistent update strategy that extracts task-related visual patterns from the original benchmark to guide data collection and filtering. Instantiated from MMBench, MMBench-Live contains 5.9K newly generated evaluation instances with a high answer correctness rate, while each update costs about USD 30 and takes 1-2 hours. Extensive evaluations show that MMBench-Live preserves stable model rankings, maintains semantic alignment with the original benchmark, and exhibits weaker contamination-related memorization signals, suggesting a practical and scalable paradigm for sustainable multimodal benchmark evolution. The project is available at https://github.com/PRIS-CV/MMBench-Live.

• The paper introduces an automated, continuously evolving benchmark that mitigates data leakage, semantic drift, and contamination in multimodal evaluations.
• It employs a multi-agent workflow with task-aware data acquisition and executable QA generation, achieving a 96.06% manual answer correctness rate at minimal cost.
• Empirical results show stable cross-version model rankings and reduced memorization signals, validating the framework’s robustness and scalability.

MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models

Motivation and Background

Static benchmarks for vision–LLMs (VLMs) present acute risks of temporal staleness and data contamination as both multimodal foundation models and pretraining corpora are scaled at unprecedented rates. Existing evaluation corpora—often static—are compromised by unintentional data leakage, semantic drift upon repetitive use, and prohibitively high costs and turnaround times required to update or replace them. Current approaches to benchmark update—including visual perturbation, linguistic rewrite, synthetic generation, or limited source updates—fall short in balancing temporal freshness, semantic alignment, fine-grained task coverage, and stable cross-version comparability.

MMBench-Live (2607.01813) addresses these critical challenges with an automated pipeline for sustainable evaluation in multimodal learning. Starting from the original MMBench, MMBench-Live implements a system for continuous evolution of evaluation data, mitigating contamination issues while covering the compositional and granular capability spectrum required by state-of-the-art VLMs.

Framework and Pipeline Overview

MMBench-Live's pipeline comprises benchmark structuring, task-aware data acquisition, and robust QA instance generation with executable reasoning verification. The system encodes the original benchmark into a multi-level, structured representation to ensure distributional fidelity and semantic consistency through update cycles. Benchmark evolution is recast as a task-driven data construction problem via a multi-agent workflow.

Figure 1: Overview of the MMBench-Live framework, showing structured benchmark conversion, task-aware data acquisition, and executable QA generation with automated verification.

Key technical features include:

• Structured Benchmark Description: Target capabilities, task hierarchy, and atomic evaluation units are precisely specified, grounding all data acquisition and QA synthesis activities. Semantic anchor points—task-related visual patterns—are identified and distilled from MMBench using VLM/LLM-driven sampling, consolidation, and redundancy removal across multiple visual and semantic axes (e.g., content, style, spatial relations).
• Task-aware Data Acquisition: Execution is organized by an acquisition planner, executor, and feedback controller. Enumerated retrieval strategies (source-specific crawling, keyword-guided search, open-domain sampling) are enhanced with temporal filtering (e.g., restrict to past year via Flickr/Google APIs) and metadata retention.

The feedback controller leverages a prompt-programmed multimodal judge to filter retrieved instances, enforce consistency with task purpose and visual pattern constraints, and iteratively refine queries on failure. This feedback loop—guided by real-time accept/reject statistics—enables the pipeline to efficiently exclude diluted or off-distribution samples while requiring minimal human intervention.

• Executable QA Generation and Validation: Automatically curated image data, together with atomic task descriptions, drive the generation of structured tuples ($q$, $a$, $\pi$), where QA content is validated using process-level executable reasoning chains over intermediate results from external perception modules (object detection, OCR, attribute recognition). Acceptance is contingent on deterministic equivalence between model-inferred answer and tool-derived output, disallowing hallucinated or non-verifiable QA pairs.

Empirical Evaluation and Analysis

Scale, Efficiency, and Data Correctness

MMBench-Live achieves fully automated benchmark construction for 5.9K QA pairs, attaining a manual answer correctness rate of 96.06%. Each benchmark update incurs a cost of approximately \$30 and completes within 1–2 hours—orders of magnitude more efficient than corresponding manual annotation pipelines, which are cited to incur costs and turnaround times at the scale of months and thousands of dollars.

Cross-Version Ranking Consistency

Evaluated across multiple families of open-source VLMs on MMBench-Live versus the original MMBench, the induced ranking order of models remains highly stable. Pearson $r$ and Spearman $\rho$ correlations at both model-capability and model-aggregated levels are strong, confirming that MMBench-Live preserves semantic granularity and capability coverage while enabling robust, temporally relevant benchmarking.

Figure 2: Rank change heatmap between MMBench and MMBench-Live showing that ranking variations are localized and do not destroy global stability across model–task pairs.

Dynamic update does not induce global ranking reversals, and the most significant deltas are found in a small subset of highly discriminative sub-tasks. Importantly, performance divergence between near-frontier models (e.g., Qwen2.5-VL vs Qwen3-VL) makes explicit the capacity of MMBench-Live to reveal subtle generalization gaps in VLMs when static benchmarks mask memorization or overfitting.

Distributional Alignment and Task Semantics

FID and PCA-KL analyses confirm that enabling task-related visual patterns and semantic feedback in the update pipeline substantially reduces semantic drift relative to the original. Ablation studies removing these mechanisms yield sharp increases in distributional divergence, underscoring the necessity of structured guidance for sustainable benchmark evolution.

Task-level semantic structure, as measured by label vocabularies and frequency distributions (e.g., for spatial-relation tasks), is maintained and in some cases expanded in MMBench-Live. The benchmark thereby remains suited for compositional, fine-grained model evaluations while surfacing emergent or newly relevant capability axes.

Mitigation of Memorization and Contamination

Adopting the PaCoST framework, MMBench-Live demonstrates lower mean confidence shifts and smaller effect sizes for memorization proxies compared to MMBench and to benchmark variants that rely on image editing, such as addition or elimination. While no evaluation pipeline can guarantee total leakage immunity given the scale and opacity of web corpora, these results quantitatively indicate MMBench-Live’s effectiveness in blunting text- or image-side contamination signals.

Failure Case Analysis

Despite robust tool-verified construction, certain error modes persist—primarily in high-level relation reasoning, especially for tasks involving complex geometric transformation or non-canonical scene layouts. Tool-based perception outputs are typically reliable; reasoning failures result from misinterpretation of the context or ambiguous reference frames.

Figure 3: A representative failure case of executable reasoning verification: correct visual grounding but mistaken diagonal spatial relation due to reference frame ambiguity.

Practical and Theoretical Implications

The methodology instantiated by MMBench-Live yields several implications:

• Long-Term Sustainability: Automated, low-cost, frequent benchmark refreshes are enabled for any VLM evaluation protocol with explicit task structuring, addressing the escalating risk of staleness and contamination.
• Theoretical Robustness: Structured preservation of benchmark semantics, coupled with explicit, agent-driven QA verification, establishes a template for rigorous, reproducible, and discriminative evaluation in the context of rapidly evolving foundation models.
• Extensibility: The paradigm can generalize to other (multimodal) evaluation domains. Further research is warranted into supplementing task-grounded evolution with task-expansive protocols to address shifting or emergent capability landscapes.

Future advances may seek to integrate geometric normalization for spatial tasks, improve executable reasoning reliability via more sophisticated reasoning engines, and explore continual semantic expansion beyond version-consistent preservation.

Conclusion

MMBench-Live (2607.01813) delivers a principled, multi-agent, automation-driven solution for dynamic multimodal benchmark construction. Quantitative assessment establishes its efficacy in maintaining benchmark semantic integrity, reducing contamination risks, ensuring stable model ranking, and supporting efficient, scalable, and high-fidelity evaluation for the next generation of vision–LLMs. Its design provides a foundational blueprint for sustainable, adaptive benchmarking across the AI community.