VIS-Shepherd: Multimodal Visualization Critic

• VIS-Shepherd is a specialized multimodal language model critic that evaluates data visualizations using expert annotations to identify defects and suggest improvements.
• It fine-tunes a compact 7B-parameter model on an expert-curated dataset, achieving a mean Likert score increase from 2.9 to 3.9 and outperforming larger models in defect detection.
• The system is applied as a post-processor in NL2VIS pipelines and educational platforms, offering actionable feedback and guiding iterative improvements in chart design.

VIS-Shepherd is a specialized multimodal LLM (MLLM)-based critic system for analyzing, identifying defects in, and providing actionable feedback about data visualizations generated by LLMs. It integrates domain-specific critique capabilities by ingesting rendered chart images along with textual instructions and data context, producing expert-like critiques. VIS-Shepherd addresses the inherent challenge in evaluating visualization outputs, which are non-textual and require both domain expertise and multimodal understanding for robust assessment. The system achieves performance comparable to or exceeding substantially larger or proprietary models when trained on a carefully curated critique dataset (Pan et al., 16 Jun 2025).

1. Origin, Motivation, and Objectives

VIS-Shepherd was conceived to address three principal challenges in LLM-based visualization generation:

• Visualization outputs are rendered graphics, making assessment of spatial layout, encoding, and visual readability infeasible for text-only critics.
• Expert-level diagnosis of visualization defects—such as missing facets, legend truncation, or overcrowding—requires specialized training.
• State-of-the-art LLMs frequently fail to reliably follow user instructions or adhere to visualization best practices.

VIS-Shepherd’s primary goals are to construct a multimodal critic that (a) comprehends the instruction and dataset context along with the LLM-generated visualization, (b) identifies concrete defects, and (c) offers actionable improvement feedback. This is enabled by curating an expert-annotated dataset where each entry encapsulates the instruction, the dataset, a human reference chart, the LLM-generated chart, and a structured critique. The approach aims to demonstrate that even a compact, 7B-parameter MLLM—when fine-tuned on domain-specific critique data—can deliver critique performance rivalling large or proprietary models (Pan et al., 16 Jun 2025).

2. Dataset Construction and Annotation Pipeline

The VIS-Shepherd dataset construction proceeds through four rigorously defined stages, resulting in quintuplets $(I, D, V_h, V_{LLM}, C)$:

Stage 1: Human-created Instance Curation

• Source: 180,000 static D3.js charts (with code) crawled from Observable.
• Automated filtering (via Gemini-2.0-flash) eliminates blanks, low-quality, and interactive charts, retaining 67,000 candidates.
• Expert annotators select 6,900 high-quality exemplars; final deduplication (Simhash plus manual review) produces 1,700 charts $V_h$.

Stage 2: Instruction Synthesis & Data Exportation

• Each human chart receives a synthesized natural language instruction $I$ generated by an LLM (e.g., Claude 3.5/Sonnet) after simulating a plausible user profile, expertise level, and scenario.
• LLM produces data-export scripts, saving the original dataset in tidy CSV/GeoJSON; correctness is verified by re-running chart code.
• Output triplets: $(I, D, V_h)$.

Stage 3: LLM-based Visualization Generation

• Contemporary LLMs (e.g., GPT-4o, Claude-Sonnet) generate D3.js code from $(I, D)$, resulting in a synthetic chart $V_{LLM}$.
• Multi-turn refinement is simulated via annotator feedback, with only compilable results kept.
• Quadruples constructed: $(I, D, V_h, V_{LLM})$.

Stage 4: High-quality Critique Collection

• Ten visualization experts annotate defects guided by a taxonomy distilled from 200 pilot examples. Categories include: Instruction Compliance, Visual Clarity, Semantic Readability, or (if no defects are present) preference/aesthetic suggestions.
• Each LLM-generated chart receives exactly one critique; if “No Defect,” annotators provide a design tip, optionally assisted by GPT-4o comparison with the human reference.
• The finalized dataset comprises 2,700 critiques—2,500 for training and 160 held out for testing.

The resulting empirical data distribution is formalized as: $$p(I, D, V_h, V_{LLM}, C) = p(I, D)\, p(V_h \mid I, D)\, p(V_{LLM} \mid I, D)\, p(C \mid I, D, V_h, V_{LLM})$$

This schema forms a unique, expert-driven resource coupling multi-source context, reference ground-truths, and actionable critiques.

3. Model Architecture and Loss Formulation

VIS-Shepherd utilizes Qwen-2.5-VL-7B, an open-source 7B-parameter multimodal LLM pretrained on chart/image/text pairs. Input is represented as follows:

• The LLM-generated chart $V_{LLM}$ is tokenized into visual patch embeddings.
• Instruction $I$ and dataset preview $D$ are serialized as text token sequences.
• Vision and text tokens are concatenated and processed jointly by the transformer backbone.
• The model outputs free-form critique text $C$.

The model is trained by minimizing the standard token-wise cross-entropy loss on ground-truth critique annotations. Given model parameters $\theta$, with $f_\theta(C|V_{LLM}, I, D)$ denoting the critic’s output distribution, the loss is: $$\mathcal{L}_{\text{critique}} = -\mathbb{E}_{(I, D, V_{LLM}, C)\sim \text{train}} \left[ \log f_\theta(C\mid V_{LLM}, I, D) \right ]$$

Notably, the model is trained exclusively on the curated critique dataset, without additional augmentation or explicit regularization beyond dropout inherited from the base MLLM.

4. Training Regimen and Infrastructure

The training configuration follows standard fine-tuning procedures:

• Optimizer: AdamW, weight decay 0.01.
• Learning rate: $1 \times 10^{-5}$ (cosine schedule, 10-step warmup).
• Batch size: 8; gradient accumulation: 1; gradient norm clipping: 1.0.
• Floating-point format: bf16.
• Epochs: 1 (specifically to prevent overfitting).
• Hardware: 8 NVIDIA A800 GPUs (80 GB each), total runtime approx. 0.5 hours per run.
• No supplemental data augmentation or regularization beyond base dropout.

This configuration enables efficient adaptation of the 7B-parameter model to the expert critique domain.

5. Evaluation Methodology

Evaluation is conducted via two complementary protocols:

Automatic (Model-based) Evaluation:

• GPT-4o is prompted as an LLM “judge,” using a 5-point Likert rubric (see Fig. 4 in (Pan et al., 16 Jun 2025)).
• Two main metrics:
  • Mean Likert score ($\in [1,5]$)
  • Percentage of high-quality responses (score ≥ 4)

Human Preference Study:

• Annotators conduct pairwise head-to-head comparisons, each reviewing two model-generated critiques for the same visualization (see Fig. 5 in (Pan et al., 16 Jun 2025)).
• Outcomes are reported as win/tie/loss percentages, gauging which model provides more accurate defect identification or constructive suggestions.

These evaluations jointly quantify both the objectivity (Likert) and subjective utility (pairwise preference) of the model’s feedback on generated visualizations.

6. Principal Results and Analytical Insights

Key findings from the VIS-Shepherd evaluation include:

• Automatic Metrics: Fine-tuned Qwen-7B (VIS-Shepherd) increases mean Likert score from approximately 2.9 (base Qwen-7B) to 3.9, and high-quality critique percentage from ~28% to ~62%.
• VIS-Shepherd surpasses Qwen-2.5-VL-72B (over 10x parameter count) and Llama-4-maverick, though performance remains slightly below GPT-4o.
• Human Preference: VIS-Shepherd achieves ~60% win-rates versus both Qwen-72B and Llama-4, with infrequent ties/losses. Against GPT-4o, VIS-Shepherd secures a ~45% win-rate and ~35% tie-rate.
• Case study analyses show VIS-Shepherd alone correctly detecting nontrivial defects (e.g., legend truncation) missed by both base models and GPT-4o.
• Ablation Studies: There is a monotonic relation between training data volume and model critique quality; at ~2.5K critiques, VIS-Shepherd overtakes Qwen-72B, indicating that domain-specific data curation can outweigh mere scale in MLLM critic performance.

These results substantiate that a modestly-sized, domain-aligned MLLM—when provided with rich, expert-labeled critique data—can rival or exceed much larger or proprietary models in automatic visualization assessment.

7. Applications, Implications, and Prospective Advances

VIS-Shepherd’s critic paradigm supports several practical and research-oriented applications:

• Integration as a post-processor in NL2VIS (natural language to visualization) pipelines to automate chart quality control prior to end-user delivery.
• Deployment as a “visualization coach” within educational platforms to improve the design skills of novice analysts.
• Use in iterative co-generation loops, feeding critique-driven feedback from VIS-Shepherd back to generative LLMs for self-improvement.

Planned extensions and open research directions include:

• Enlarging the critique dataset to encompass a broader range of visualization types and domains (including medical and geospatial visualizations).
• Fine-tuning larger multimodal backbone models to approach or surpass GPT-4o-level performance.
• Expansion beyond static charts to interactive or animated visualizations, necessitating new annotation frameworks.
• Development of benchmarks that assess not only defect detection but also the impact of automated corrections/refinements on visualization quality.

A plausible implication is that “critic-augmented” visualization generation may represent a new paradigm, wherein expert feedback mechanisms are embedded into the LLM development pipeline for closed-loop self-improvement (Pan et al., 16 Jun 2025).