VIFBENCH: Multimodal Evaluation Benchmarks

• VIFBENCH is a collection of benchmarks spanning visible–infrared image fusion, image-grounded video perception, and LLM instruction verification.
• It provides comprehensive datasets, standardized algorithms, and diverse evaluation metrics that drive methodological innovation and robust comparison.
• The suite supports practical applications by addressing real-time performance, integration challenges, and fine-grained, constraint-level analysis across modalities.

VIFBENCH is a term encountered in three distinct research domains: (1) visible-infrared image fusion assessment, (2) image-grounded video perception and reasoning with multimodal LLMs, and (3) instruction-following verification for LLMs. Each use case provides a dataset or benchmark relevant to its respective field, supporting algorithmic evaluation, methodological advancement, and robust comparison.

1. Visible–Infrared Image Fusion Benchmark (VIFBENCH)

Dataset Construction and Content

VIFBENCH for image fusion consists of 21 geometrically registered visible–infrared image pairs, sourced from public tracking datasets (e.g., OSU Color-Thermal, TNO, VLIRVDIF), the INO video analytics dataset, established fusion-tracking benchmarks, and supplementary internet downloads. Scenarios span indoor/outdoor, day/night, under low light or over-exposure, and include thermal targets and challenging conditions such as shadows and glare (Zhang et al., 2020).

All images undergo subpixel-precision registration. For color–infrared fusion, each visible RGB channel is fused independently with the infrared modality and then recombined.

Algorithm Library

VIFBENCH provides a unified codebase for 20 fusion algorithms:

Category	Methods
Multi-Scale Decomposition	MSVD, GFF, ADF, CBF, GFCE, HMSD_GF, Hybrid_MSD, MGFF
Hybrid/Sparse-Representation	MST_SR, RP_SR, NSCT_SR
Subspace-Based	FPDE
Saliency-Based	TIF, VSM_WLS, LatLRR, IFEVIP
Deep Learning	CNN, DLF, ResNet
Other	GTF

Algorithms include classical decomposition and saliency strategies, as well as recent deep networks implemented in MATLAB/Python with a standardized interface.

Evaluation Metrics

Thirteen metrics span information-theoretical, structural, image-feature, and human-perception-inspired measures:

Metric Type	Examples	Metric Direction
Information-Theory based	Entropy (EN), MI, CE	EN, MI: higher is better; CE: lower is better
Structural Similarity	SSIM, RMSE, PSNR	SSIM, PSNR: higher; RMSE: lower
Feature-based	AG, EI, SF, SD	All: higher is generally better
Perceptual Quality	Q_CB, Q_CV	Q_CB: higher; Q_CV: lower

Explicit formulae are provided for each, e.g., $EN(F) = -\sum_k p_F(k) \log p_F(k)$.

Experimental Results and Analysis

• No method achieves dominance across all metrics; NSCT_SR, LatLRR, and DLF each attain three first-places on distinct criteria.
• Deep learning (CNN, DLF, ResNet) marginally underperforms comparably against classical multi-scale or saliency-based techniques in quantitative and qualitative assessments.
• Fast, multi-scale decomposition methods (GFF, TIF, IFEVIP) best balance efficiency and competitive quality, supporting real-time application requirements.
• Under visibility extremes (glare, hidden targets), methods with guided filtering or saliency weighting (GFF, MGFF, TIF, VSM_WLS) show robust performance.
• Artifacts and blurring are prevalent in some methods (CBF, NSCT_SR, MST_SR, CNN).
• For runtime, deep learning models are slowest (e.g., CNN: 31.8 s/pair) while GFF and TIF are near real-time (< 0.5 s/pair).

Key implications include the importance of multi-metric evaluation and continued relevance of non-deep methods in practical settings (Zhang et al., 2020).

Future Prospects

Development areas include dataset expansion (greater variety, dynamic sequences), new metrics (including learned or no-reference perceptual scores), lighter neural methods, unsupervised and GAN-based approaches, and robustness to misregistration and noise.

2. Image-Grounded Video Perception and Reasoning with Multimodal LLMs (“IV-Bench” / VIFBENCH)

Dataset Construction

IV-Bench (also cited as "VIFBENCH" in some literature) comprises 967 long-form videos (≥5 min each) and 2,585 human-annotated queries. Videos are distributed over five thematic categories:

• Knowledge (lectures, documentaries)
• Film & Television
• Sports Competitions
• Artistic Performances
• Life Records

Each query is associated with one of 13 task types, a manually retrieved external image (never a video frame), a textual question, and 10 answer options (1 correct, up to 9 effective distractors). A stringent two-stage quality control procedure eliminates questions solvable by video or world knowledge alone and ensures at least two distractors are dependent on the image context (Ma et al., 21 Apr 2025).

Task Suite

Tasks span 7 perception and 6 reasoning categories:

Perception Tasks	Reasoning Tasks
Existence, Reverse Existence, NLI, Spatial Relationship, Keyframe Extraction, Constrained OCR, Detailed Events	Counting, Space-Time Computing, Summary, Instruction Understanding, Attribute Change, Temporal Reasoning

Input for all tasks comprises a set of sampled frames, a reference image, and a question. All tasks are multiple-choice (10 options; random accuracy = 10%).

Evaluation Protocol

• Accuracy is the principal metric: $$\mathrm{Acc} = \frac{\#\text{correct}}{\#\text{total}}$$
• Supplemental open-ended metrics (precision, recall, F1) are defined for ablation.
• Frame sampling (default 32 frames) and multiple tested resolutions (72p–240p).
• Three inference patterns are analyzed: video-first, image-first, text-only.

Benchmark Results

Selected results (top accuracy):

Model	Accuracy (%)
Qwen2.5-VL-72B	28.9
InternVL2.5-78B	28.6
Gemini-2 Pro	27.7
Gemini-2 Flash	27.4
GPT-4o	20.7

Perception tasks reach ~35% best average; reasoning ~22%. Temporal Reasoning is most challenging (≤17%). Model scaling yields modest improvement; performance plateaus below 30% overall.

Analysis and Insights

• Small models may degrade with image inclusion, indicating weak grounding mechanisms.
• Large models improve when the image is processed after video frames, evidencing temporal information retention issues.
• Frame count and resolution increases enhance accuracy, but only up to a threshold.
• Under constrained visual-token budgets, small models benefit from additional frames; large models flexibly utilize spatial/temporal input.
• Data synthesis for format alignment (LLAVA-video178K) produces marginal improvement over direct fine-tuning.

Recommendations and Open Challenges

• Incorporate authentic image-grounded video samples, emphasize challenging distractor construction.
• Develop temporal attention and memory modules explicitly conditioned on static image cues.
• Pursue dynamic allocation of spatial/temporal resolution by task.
• Exploit memory-augmented architectures to persist static image context throughout temporal reasoning.
• Future work should support multi-step and hierarchical reasoning pipelines.

IV-Bench exposes major limitations in multimodal LLMs’ capacity for integrating external images within temporal video context, and provides a rigorous foundation for advancing complex multimodal reasoning (Ma et al., 21 Apr 2025).

3. Instruction-Following Verification Benchmark for LLMs (VifBench)

Motivation and Objectives

Existing verifiers for LLM instruction-following are constrained either by narrow schemas or binary global decision making. VifBench addresses these gaps by enabling fine-grained, constraint-level evaluation for arbitrary natural-language instructions, supporting composite logical and semantic requirements (Su et al., 25 Jan 2026).

Dataset Composition

• 820 entries, each with an instruction, LLM output, and annotated result.
• Balanced set: 391 (47.7%) fully satisfying all constraints ("sat"), 429 (52.3%) violating at least one ("unsat").
• Instructions are synthesized with 2–10 randomly selected constraints from a taxonomy of 10 types:
  • Semantic: writing topic, tone
  • Logic: keyword inclusion/exclusion, title, subtitle, sentence word count, even/odd length, required global/subsection start/end words
• Each violation is tracked at the constraint index level; e.g., violation set = {“Total word count”}.

Methodology

• Constraint templates informed by instruction-tuning corpora (OASST, lmsys-Chat-1M, WildChat) and prior benchmarks (ComplexBench, InFoBench, FollowBench).
• For each complexity value $C$, $C$ unique constraint types are composed into a single instruction.
• “Sat” outputs use error-corrected GPT-4.1 completions; “unsat” outputs are mutated for targeted violations.
• Author inspection and logic/semantic checking ensure labeling fidelity.

Evaluation Metrics and Protocols

Metrics—per verifier—include:

• Precision, Recall, F₁ (harmonic mean), Pass@1 (first-guess accuracy)
• Aggregate compliance and violation rate are computed as:

$$\text{Compliance} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}(\text{output}_i \models f(C_i))$$

$$\text{ViolationScore} = \frac{1}{\sum_i k_i}\sum_{i=1}^N \sum_{j=1}^{k_i} \mathbf{1}(\text{constraint}_{i,j}\text{ violated})$$

Cross-verifier protocols encourage both global and per-constraint analysis.

Comparative Results

Method	F₁ (%)
Baseline (LLM-Judge)	≈ 69.1
GEPA-CoT	67.1
Conv-CoT	78.9
Nsvif-Neu	84.0
Nsvif (full)	94.8

Nsvif achieves markedly higher precision and recall (F₁ = 94.8%), with reduced false positive rate (2.8%) over other LLM-based approaches.

Interpretability and Feedback

Fine-grained annotation enables constraint-localized feedback, e.g., flagging under-length output or confirming tonal correctness. Such feedback loops enable rapid correction: in case studies, 5/9 errors were fixed within three iterative LLM responses.

Limitations and Open Questions

Limiting factors include task and language coverage (currently English free-form writing), reliance on hard-constraint CSP (soft constraint/optimization is a natural extension), and dependence on subjective ground-truth for semantic constraints. Dataset size (820) is moderate; further expansion across domains and modalities is a prospective direction.

4. Context Among Related Benchmarks

VIFBENCH's various instantiations should not be conflated: the visible/infrared image fusion suite (Zhang et al., 2020), the image-grounded video LLM benchmark (“IV-Bench”) (Ma et al., 21 Apr 2025), and the LLM instruction-following verifier dataset (Su et al., 25 Jan 2026) all target distinct modalities, methodologies, and evaluation regimes.

Unlike VBench-2.0 (Zheng et al., 27 Mar 2025), which assesses intrinsic video generation faithfulness, VIFBENCH does not encompass physical realism, causal or compositional consistency, or multi-modal generative assessments as central metrics.

5. Significance and Prospective Developments

VIFBENCH datasets across modalities signal a maturing landscape in evaluation best practices, with common emphases on:

• Granular, interpretable evaluation (constraint-level for language, modality-level for fusion/grounding)
• Comprehensive algorithm comparison using diverse, orthogonal metrics
• Addressing practical deployment demands (speed, robustness, generalization)
• Benchmark-driven methodological innovation (in instruction-following, cross-modal grounding, video reasoning)

A plausible implication is that the VIFBENCH paradigm—dataset curation, broad task coverage, and fine-grained labeling—serves as a template for future benchmarks in multi-modal and multi-criteria machine perception and reasoning.

• Visible-Infrared Image Fusion Benchmark (VIFB): (Zhang et al., 2020)
• IV-Bench: Image-Grounded Video Perception and Reasoning: (Ma et al., 21 Apr 2025)
• VifBench for Instruction-Following Verification: (Su et al., 25 Jan 2026)
• VBench-2.0 for Intrinsic Faithfulness in Video Generation: (Zheng et al., 27 Mar 2025)