Grounded Visual Chat Dataset

• Grounded Visual Chat datasets are multimodal benchmarks that combine visual grounding with natural language dialogue to train Large Multimodal Models.
• They utilize region-level annotations (bounding boxes/masks) and structured tokens, integrating human and LLM-aided protocols to align dialogue with specific image or video regions.
• The datasets drive improvements in both conversational fluency and precise object localization, while highlighting challenges in annotation scope and dialogue complexity.

Grounded Visual Chat (GVC) datasets define a class of multimodal benchmarks aimed at unifying region-level visual grounding with natural language dialogue. These resources are engineered to measure and train Large Multimodal Models (LMMs) to reason jointly over images (or videos), object localization (via segmentation or bounding boxes), and conversational language. This field spans static image chat datasets such as the LLaVA-Grounding GVC (Zhang et al., 2023), video-grounded dialogues like the Twitch-FIFA GVC (Pasunuru et al., 2018), and large-scale referential video chat corpora such as SAMA-239K (Sun et al., 24 May 2025). GVC datasets are crucial for progress toward grounded multimodal assistants, enabling models to respond with fluent, contextually appropriate chat while explicitly resolving referring expressions to image or video regions.

1. Foundational Definitions and Motivation

Grounded Visual Chat datasets are constructed to address the limitations of prior multimodal resources that typically decouple conversation (image chat, visual dialogue) from fine-grained region-level grounding (object localization, segmentation). Their core goal is to provide supervision and evaluation for systems that, given an input visual context and language prompt, output not only coherent conversational text but also region-level references—linking language phrases to explicit spatial regions (masks/boxes).

The primary motivation is the observed failure cases in LMMs: chat models may “hallucinate” or be vague about visual region references, while classic grounding/grounding+caption datasets supply only terse descriptions and lack dialogue structure. GVC benchmarks enforce simultaneous grounding and dialogic coherency, incentivizing models that do not degrade chat fluency or localization precision when both are required (Zhang et al., 2023).

2. Data Collection and Annotation Protocols

The construction of GVC datasets involves complex pipelines combining automatic generation, human annotation, and LLM-aided alignment:

• Static Image GVC (LLaVA-Grounding): Each sample is a tuple $(I_i, C_i, T_i, A_i, G_i)$, where $I_i$ is a COCO image, $C_i$ the conversational prompt, $T_i$ the answer text (annotated with grounding markers), $A_i$ auxiliary information, and $G_i$ the mapping of text spans (noun phrases) to grounded instances (boxes/masks). The pipeline overlays COCO human-annotated object instances atop GPT-4-generated chat turns, then enlists GPT-4 to match phrases to object regions. Special tokens—〈g_s〉/〈g_e〉 for grounded phrases, 〈seg〉 for segmentation points, 〈obj〉 for referring expression slots—are used to wrap the output, and each 〈seg〉 aligns to an explicit segmentation mask (Zhang et al., 2023).
• Video-context GVC (Twitch-FIFA): Data consists of triples $(V_i, C_i, R_i)$, with $V_i$ a 20 s context of video frames (downsampled, feature-extracted), $C_i$ the multi-speaker preceding chat, and $R_i$ the target next chat message (filtered by consensus). There are no explicit region-level groundings, but models must infer relevant spatial/temporal cues from the video and chat to predict the next reply (Pasunuru et al., 2018).
• Spatiotemporally Grounded Video Chat (SAMA-239K): The dataset includes 15,143 videos with 67,005 object descriptions and 172,296 multi-turn referential QA pairs, each paired with spatiotemporal masks or bounding boxes. The annotation protocol leverages LLMs (e.g., Gemini-1.5 Pro) prompted with color-coded regions, proceeding from object-level captions through multi-turn questions demanding explicit linguistic references to specific video regions and attributes (Sun et al., 24 May 2025). Quality is enforced via automated and human filtering for relevance and correctness.

Dataset	Visual Modality	Region Grounding	Dialogue Structure
LLaVA-GVC	Image	Mask/box	Single-turn QA
Twitch-FIFA GVC	Video	—	Many-speaker, next-turn
SAMA-239K	Video	Spatiotemporal	Multi-turn, referential QA

This table distinguishes GVC dataset types by visual modality, presence and form of region grounding, and dialogue granularity.

3. Grounded Visual Chat Data Structures and Representation

Formally, GVC datasets encode samples as higher-order tuples linking visual context, conversational context, and region-level supervision:

• Static image GVC:

$$D_\text{GVC} = \left\{ (I_i, C_i, T_i, A_i, G_i) \right\}_{i=1}^{N}$$

Where for each $i$, $G_i$ specifies the mapping: phrase $\to$ region (mask or bounding box).

• Video-context GVC:

$D = \left\{ (V_i, C_i, R_i) \right\}_{i=1}^{N}$

With $V_i$ a set of video frame features, $C_i$ the chat context, $R_i$ the next best (consensus) reply.

• Spatiotemporal GVC (SAMA-239K):

Each sample aligns textual dialogue turns to one or more pixel-accurate masks or COCO-format bounding boxes over sampled video frames. Spatial and temporal completeness is enforced by systematic frame sampling (e.g., every 4th, 16 uniformly spaced, or dataset-specific).

This explicit alignment facilitates supervised training of multimodal models to emit both high-quality conversation and granular object/region selection sequences.

4. Benchmarks and Evaluation Metrics

Evaluation of GVC performance mandates end-to-end metrics for both language and grounding:

• Grounding-Bench (LLaVA-Grounding): Pairs 1,000 held-out COCO images (∼7,000 entities) with description or follow-up chat prompts. Models output text and a set of boxes/masks, scored in two stages:
  1. Chat quality via standard LLM metrics (after removing grounding tags).
  2. Grounding via phrase-to-region F1: true positives require semantic correctness (GPT-4-verified) and IoU $> 0.5$ overlap with GT regions. Precision, recall, and F1 are computed as

  $$\text{IoU}(B_\text{pred}, B_\text{gt}) = \frac{|B_\text{pred} \cap B_\text{gt}|}{|B_\text{pred} \cup B_\text{gt}|}$$

  and

  $\text{F1} = \frac{2P R}{P + R}$

Baseline F1s: Shikra ≈ 27.6%, miniGPT-v2 ≈ 25.6%, CogVLM-Grounding ≈ 32.0%, LLaVA-Grounding = 37.1% (Zhang et al., 2023).

• Video Chat Retrieval (Twitch-FIFA): Recall@k for ranking the gold next message among 10 candidates; no region-level evaluations due to absence of grounding annotations (Pasunuru et al., 2018).
• SAMA-Bench: 5,067 questions over 522 videos, with region grounding scored by mean IoU, recall@IoU$>$0.5; text scored by METEOR, CIDEr, and the CLAIR reference-sensitivity metric (Sun et al., 24 May 2025).

5. Model Architectures and Training Regimes

GVC benchmarks have driven the development of new LMM architectures with unified grounding–dialogue modules:

• LLaVA-Grounding System: Augments LLaVA by attaching
  • a prompt encoder $h_{\phi_p}$ (typically Semantic-SAM) mapping user prompts to dense vectors $V_p$, which are projected into the LLM input, and
  • a grounding model $s(\cdot)$ (OpenSeeD), producing masks/boxes at each 〈seg〉 step in decoding. The models are trained in three stages: pretrain alignment on vision–language datasets (frozen LLM), instruction-tune on 150K GVC samples (fine-tune all but the CV encoder), and prompt extension for referring expression usage (Zhang et al., 2023).
• SAMA System: Incorporates a spatio-temporal context aggregator combining spatial cross-attention, temporal token aggregation, and fusion with question embeddings. The grounding component (SAM2) is triggered whenever the LLM emits a special token representing a referential query, yielding pixel-accurate masks in video (Sun et al., 24 May 2025).

6. Comparative Performance and Limitations

Empirical studies indicate GVC datasets uniquely support the co-training of chat and grounding:

• Performance: LLaVA-Grounding achieves F1 = 37.1% on Grounding-Bench with chat all-task ≈ 79.3%. On classic benchmarks (RefCOCO/+/g, Flickr30K Entities), it matches or surpasses prior open-source and specialized models in referring expression comprehension ([email protected] up to 89.16%) and segmentation (mIoU up to 79.68%) (Zhang et al., 2023). SAMA sets new state-of-the-art on video referential grounding benchmarks (Sun et al., 24 May 2025).
• Limitations:
  • Annotation scope: LLaVA-Grounding's semantic coverage is bounded by COCO's vocabulary and the typically sparse instance-per-image regime—a challenge for instance disambiguation and open-vocabulary recognition.
  • Dialogue complexity: Many GVC samples consist of only single-turn exchanges; richer multi-turn phenomena remain underrepresented in some benchmarks.
  • Long-tail diversity: Both static and video GVC datasets may lack coverage for rare categories or complex events, suggesting future extensions toward Object365, LVIS, and more varied domains (Zhang et al., 2023).

A plausible implication is that as GVC benchmarks mature to cover broader domains and more dialogic diversity, model architecture and evaluation protocols will need further refinement.

7. Position Within the Broader Landscape

Grounded Visual Chat datasets have defined a new frontier at the intersection of referential grounding, vision–language understanding, and conversational AI. They contrast with:

• Captioning/grounding datasets: e.g., COCO Captions, RefCOCO, which lack dialogue.
• Vision-language dialogue datasets: e.g., Visual Dialog, Image-Chat, which typically offer only whole-image-level references.
• Video chat datasets: e.g., Twitch-FIFA (Pasunuru et al., 2018), SAMA-239K (Sun et al., 24 May 2025), which push goals toward spatiotemporal referential understanding and true conversational flow.

GVC benchmarks are now central to the assessment and advancement of LMMs capable of both conversational fluency and fine-grained, phrase-conditioned region localization. Continued development in open-domain annotation protocols, robust evaluation, and multi-turn conversational structure is expected to consolidate their role in the landscape of multimodal AI research.