VIST-GPT: Multimodal Visual Storytelling

• VIST-GPT is a multimodal model family that integrates computer vision and natural language processing to generate coherent, image-grounded narratives and dialogues.
• It employs architectures like the encoder–adapter–decoder pipeline and cross-modal fusion strategies, tailored for both narrative storytelling and podcast-style dialogue.
• Robust training protocols using synthetic-to-real datasets and advanced evaluation metrics ensure enhanced visual grounding, narrative coherence, and conversational naturalness.

VIST-GPT denotes a family of multimodal LLMs for visual storytelling: the task of generating coherent, image-grounded narratives, or long-form dialogues, from sequences of photographs. These systems integrate computer vision and natural language processing to bridge the visual and textual modalities, producing outputs ranging from single-sentence-per-image stories to multi-speaker podcast dialogues. VIST-GPT models are characterized by the use of advanced vision encoders, instruction-tuned LLMs, explicit cross-modal token fusion, and comprehensive evaluation protocols moving beyond legacy n-gram metrics to address narrative coherence, visual grounding, and conversational naturalness (Zeng, 3 Jan 2026, Gado et al., 27 Apr 2025).

1. System Architectures and Modalities

VIST-GPT implementations exhibit variation according to task framing—narrative generation versus podcast-style dialogue—but share core architectural principles.

The archetypal narrative VIST-GPT employs an encoder–adapter–decoder pipeline. Two frozen vision encoders, CLIP ViT-L/14 for independent per-image spatial features and InternVideo-v2 to encode temporal dynamics through an "8-frame video" abstraction, extract feature tensors. Trainable MLP adapters (V-L adapters) project these vision features into the LLM’s embedding space, concatenated with prompt tokens before feeding to a causal-transformer LLM (often Phi-3-mini-4k-instruct) fine-tuned solely via LoRA adapters. Visual tokens are pooled to avoid excessive sequence length, and early fusion is adopted to maximize cross-modal integration (Gado et al., 27 Apr 2025).

In podcast-style models, such as SPoRC-VIST, a Qwen3-VL-32B architecture is used: a frozen cross-modal vision encoder with gated cross-attention fuses image patch embeddings with a 32B-parameter transformer backbone, further adapted via LoRA in all self-attention and MLP modules. The system prompt specifies conversational constraint (“Speaker 1:,” “Speaker 2:,” “~800 words,” etc.), and the setup prescribes image input followed by prompt concatenation before autoregressive decoding (Zeng, 3 Jan 2026).

2. Training Data and Synthetic-to-Real Regimens

VIST-GPT models require carefully curated parallel corpora of image sequences and aligned narrative text. Two principal data paradigms are documented:

• VIST-based Narrative Training: The Visual Storytelling (VIST) dataset pairs quintets of real photographs with five-sentence human-authored stories. Images are resized/normalized for encoder compatibility, and stories are concatenated into tokens with enforced sentence-image alignment (Gado et al., 27 Apr 2025).
• SPoRC-derived Dialogue Training: For podcast-oriented VIST-GPT, curated multi-turn, two-speaker podcast excerpts from the Structured Podcast Research Corpus (SPoRC) are selected based on richness of visual description, with LLM-driven prompt engineering to generate five synthetic images per excerpt using Stable Diffusion 3.5. Average excerpt: 1,000 words, mean turn length 67.3, speaker switch rate 14.8/1k words (Zeng, 3 Jan 2026). Crucially, all training images are synthetic, while evaluation is performed on real-world VIST photo sequences, enforcing a synthetic-to-real generalization challenge.

3. Fine-Tuning Protocols and Optimization

Low-rank adaptation (LoRA) is central to the adaptation of LLMs within VIST-GPT. Only the LLM layers are tuned; all visual encoders and adapters remain frozen, reducing computational cost and risking less overfitting.

• Optimization details (SPoRC-VIST): LoRA rank $r=16$, scaling $\alpha=32$, dropout 0.05. Learning rate $\eta=4\times10^{-6}$ with cosine decay, 10% warmup, weight decay 0.1, gradient clipping 0.3, NEFTune noise $\alpha=5.0$, batch size per GPU 1, gradient accumulation 4 (effective 32), bf16 precision, one epoch on 8×A100 GPUs with ZeRO-3 and gradient checkpointing. Training loss converges to 2.56 after ~88 A100-GPU-hours (Zeng, 3 Jan 2026).
• Optimization details (VIST narrative): AdamW optimizer, weight decay 0.01. Linear warm-up (500 steps) to peak $2 \times 10^{-5}$, then linear decay. Batch size 4, 5 epochs on DGX A100 (80GB GPU). No data augmentation except image-to-video padding for InternVideo-v2 (Gado et al., 27 Apr 2025).

4. Evaluation Metrics and Comparative Results

Traditional automatic text metrics (BLEU, ROUGE, METEOR, CIDEr) are insufficient for visual storytelling—penalizing valid paraphrases and miss ground-truth visual/image-text grounding. VIST-GPT techniques advance alternative methodologies:

Metric	Description	Citation
CLIPScore	Cosine similarity between CLIP-encoded image set and transcript, weighted for sensitivity to grounding	(Zeng, 3 Jan 2026)
RoViST	Comprises Visual Grounding (VG), Coherence (C), Non-Redundancy (NR); uses GloVe and ViT embeddings for noun-region alignments, ALBERT for coherence, Jaccard for redundancy	(Gado et al., 27 Apr 2025)
GROOVIST	CLIP-based, noun-to-image cosine similarity for object presence and alignment	(Gado et al., 27 Apr 2025)
Conversational Style Metrics	Average turn length, speaker switch rate	(Zeng, 3 Jan 2026)
Distinct-2	Bigram diversity (number of unique bigrams/total)	(Zeng, 3 Jan 2026)
UniEval	Unified QA-based evaluation of coherence, understandability, fluency	(Gado et al., 27 Apr 2025)
AI-as-a-Judge	Triplet LLM comparison of naturalness in conversation	(Zeng, 3 Jan 2026)

Results on VIST-GPT variants demonstrate:

• VIST-GPT v2 achieves VG: 0.9962, Coherence: 0.7837, Non-Redundancy: 0.9301, Human–Machine distance: 0.0459 (lower is better), UniEval fluency: 0.950, outperforming previous approaches (AREL, GLACNET, KG Story, MCSM+BART) by large margins (Gado et al., 27 Apr 2025).
• SPoRC-VIST’s Qwen3-VL-32B achieves CLIPScore: 20.39 (identical to much larger 235B baseline), average turn length 57.5 (+51% over baseline), speaker switch rate 16.0 (–41%), number of turns 15.8 (–35%), Distinct-2: 0.82. AI Judge win rate exceeds 80% versus 235B baseline; inference speed is 1.8× faster with longer, more richly styled outputs (Zeng, 3 Jan 2026).

5. Qualitative Observations and Analysis

Qualitative evaluations highlight VIST-GPT’s capacity to capture image semantics and maintain narrative or conversational coherence.

• Narrative grounding: VIST-GPT v2 accurately references scene objects and actions (e.g., correctly describing “ollie” at a skate park or “multi-generational bonding” on a family boating trip) and orders events according to photo sequencing. It avoids irrelevant hallucinations present in previous models (e.g., referencing “skiing” in a non-winter scene) (Gado et al., 27 Apr 2025).
• Podcast dialogue: The fine-tuned 32B model in SPoRC-VIST produces dialogue richer in personality and narrative flow. The style metrics, including reduced speaker switch rate and increased turn length, align with natural spoken conversation (Zeng, 3 Jan 2026).
• Lexical and structural diversity: Slight reduction in bigram diversity compared to baseline, attributed to human-like discourse markers and fillers (Zeng, 3 Jan 2026).

6. Limitations and Future Directions

VIST-GPT systems demonstrate state-of-the-art performance but retain several limitations:

• Omission of fine details: Models may ignore subtle visual cues or background details, focusing on salient foreground objects or primary actions (Gado et al., 27 Apr 2025).
• Controlled hallucination risk: While the models hallucinate less than prior baselines, plausible but ungrounded narrative embellishments may still appear in ambiguous visual contexts.
• LLM capacity constraints: The narrative nuance and emotional depth may be constrained by the parameter count and instruction fine-tuning scope; scaling to larger LLMs (e.g., LLaMA 3) is a suggested path for deeper narrative sophistication (Gado et al., 27 Apr 2025).
• Domain generalization: Synthetic-to-real transfer may not capture the full distributional diversity of real-world images, suggesting further augmentation (exemplar mixing, urban/historical domain extension, color jitter) could increase robustness (Gado et al., 27 Apr 2025, Zeng, 3 Jan 2026).
• Potential biases: Podcast-trained models risk encoding demographic or stylistic biases from the SPoRC corpus; bias mitigation layers are proposed for future extensions (Zeng, 3 Jan 2026).

Proposed future VIST-GPT capabilities include streaming/variable-length inputs (for video or live feeds), SSML/emotion-token output for prosody in TTS, reinforcement learning with listener engagement signals, watermarking for revenue and provenance tracking, and broader dataset augmentation (Zeng, 3 Jan 2026).

7. Significance for Multimodal Language Modeling

VIST-GPT demonstrates that compact, instruction-tuned LLMs, when precisely aligned with powerful vision encoders and guided by robust reference-free metrics, can approach or surpass human-level visual storytelling quality. The explicit focus on narrative—rather than merely descriptive—output, and movement beyond n-gram overlap as a quality criterion, establishes new benchmarks for evaluability, coherence, and utility in generative vision-language research. The synthetic-to-real paradigm and conversational podcast generation setting suggest extensibility to new modalities (e.g., video, streaming, TTS) and further applications in media, education, and accessibility (Gado et al., 27 Apr 2025, Zeng, 3 Jan 2026).