Video-LMMs: Multimodal Video Reasoning

• Video-LMMs are neural architectures that fuse video frames, audio, and text to enable advanced temporal reasoning and comprehensive multimodal understanding.
• They employ pretrained visual encoders, temporal modeling, and adaptive fusion techniques like cross-attention to efficiently integrate diverse data modalities.
• Despite notable advances, challenges remain in long-form video processing, precise temporal contrast, and robustness against misleading inputs.

Video Large Multi-modal Models (Video-LMMs) are neural architectures that combine video frame sequence analysis with textual and, in some cases, audio and other modality understanding via LLMs. They enable high-level comprehension, reasoning, and interaction with dynamic audio-visual content—substantially expanding the interface of large models beyond static images to encompass temporal, multimodal, and context-rich video data. Video-LMMs draw on foundational vision encoders, explicit temporal modeling, multimodal fusion, instruction tuning, and are evaluated via an increasingly sophisticated suite of benchmarks targeting comprehension, reasoning, grounding, and culture-aware or domain-specific tasks.

1. Canonical Architectures and Feature Fusion

Video-LMMs typically comprise a frozen or pre-trained visual encoder (often a vision transformer such as ViT), an optional audio encoder (e.g. ImageBind or Whisper), lightweight modality adapters (Q-Formers, MLPs, or prefix-tuning heads), and a decoder-style LLM such as LLaMA or Vicuna. Video inputs are sampled as sequences of frames $F_v \;\in\;\mathbb{R}^{T\times H\times W\times 3}$, with each frame independently encoded and further endowed with explicit temporal position embeddings $p_i$ for temporal modeling. Adapters, such as Q-Formers, cross-attend to sequences or chunks of encoded frame (and audio) tokens, aggregate them, and project into the LLM’s embedding space via learnable linear mappings.

The fusion of modalities is achieved by concatenating video-derived tokens $\hat Q_v$, optionally audio tokens $\hat Q_a$, and embedded textual prompts, prepending the aggregate sequence to the LLM for autoregressive generation. Some models, such as LLaVA-NeXT-Interleave (Li et al., 2024), treat video frames as a simple extension of multi-image input, interleaving patch tokens in chronological order and relying on self-attention for implicit temporal inference.

Recent models introduce refined mechanisms for efficiency and expressiveness. Quicksviewer compresses videos by partitioning them into variable-density cubes using a Gumbel-Softmax cubing network, dramatically reducing spatio-temporal redundancy (up to $45\times$) (Qi et al., 21 Apr 2025). The Slow–Fast architecture introduces dual token streams (compressed “fast” preview tokens and uncompressed “slow” tokens cross-attended by text embeddings), efficiently balancing fine spatial detail with long-range temporal coverage (Shi et al., 2 Apr 2025). CrossLMM interleaves extreme spatial pooling with dual cross-attention (visual-to-visual and text-to-visual) within the LLM to recover fine details at dramatically reduced token cost (Yan et al., 22 May 2025).

2. Temporal and Multimodal Reasoning

A central challenge for Video-LMMs is explicit modeling of temporal changes, order, and causality. The Video Q-Former proposed in Video-LLaMA injects learnable temporal position embeddings and cross-attends to sequences of region tokens, enabling the model to learn video-language correspondence through video-to-text generation tasks under a cross-entropy objective (Zhang et al., 2023). However, benchmarks such as RTime-QA (Liu et al., 25 May 2025) demonstrate that standard architectures, which concatenate frame encodings with positional tags, still struggle with atomic temporal contrasts. Strict accuracy on temporally negative video pairs is barely above random (34.6%) for top models, far below human performance; instruction tuning with dedicated datasets (RTime-IT) can nearly double temporal accuracy (65.9%).

Multimodal integration is a strength of models leveraging universal audio-visual embedding spaces (e.g. ImageBind). This design unlocks capabilities such as simultaneous auditory and visual QA in open-ended conversational settings. Audio features are particularly effective in downstream tasks such as engagement prediction, where models using full audio tracks notably outperform those with visual and textual inputs alone (Sun et al., 4 Aug 2025).

Benchmarks targeting robustness (CVRR-ES (Khattak et al., 2024)) expose weaknesses in complex reasoning (multi-action, OOD social/emotional scenes) and robustness to misleading prompts (over-affirmation bias, temporal confusion). Training-free prompting protocols, such as Dual-Step Contextual Prompting (DSCP), can significantly lift performance without retraining, especially in adversarial or negative reasoning scenarios.

3. Instruction Tuning and Post-Training Pipelines

Instruction tuning with high-quality visual and video datasets (e.g., MiniGPT-4, LLaVA, Video-Chat) is critical for aligning Video-LMMs to human-centric conversational and reasoning tasks. The typical workflow is a two-stage process: large-scale pre-training on captioned web videos and images, followed by fine-tuning on curated instruction-following datasets. Overall optimization combines video-language cross-entropy objectives and instruction-following likelihoods, e.g.,

$L \;=\;\alpha\,L_{\mathrm{vid2txt}\;+\;\beta\,L_{\mathrm{instr}}$

with staged weighting.

Post-training advances include supervised fine-tuning with chain-of-thought (CoT) supervision to bootstrap multi-step reasoning formats, reinforcement learning from verifiable objectives (e.g., GRPO), and test-time scaling via beam search, chain-of-thought prompting, self-consistency voting, and tool-augmented inference. Sophisticated reward designs accommodate answer correctness, temporal localization (tIoU), region IoU, and budget awareness (Tang et al., 6 Oct 2025).

Efficient scalable tuning is exemplified by RED-VILLM (Huang et al., 2024), which "upgrades" an existing image-LLM backbone via a plug-and-play temporal module, requiring only parameter-efficient adaptation (a few million trainable weights) and modest instruction data (100k video QA pairs).

4. Benchmarks: Comprehension, Reasoning, Grounding, and Cultural Diversity

A new generation of benchmarks assesses Video-LMMs across comprehension, advanced reasoning, spatial-temporal grounding, anomaly detection, and multicultural understanding.

• Comprehensive evaluation suites: Video-MME (Fu et al., 2024) covers 900 videos, 2,700 questions, and multimodal inputs (frames, subtitles, audio) across 6 visual domains and 30 categories; Gemini 1.5 Pro achieves 81.6% accuracy (with subtitles), far exceeding open-source models.
• Complex reasoning and robustness: CVRR-ES (Khattak et al., 2024) probes 11 dimensions, including social/emotional context, partial and non-existent actions. Human performance (96.7%) is unmatched; open-source models rarely exceed 33%, but dual-step contextual prompting can boost them to >40%.
• Atomic temporal event understanding: RTime-QA (Liu et al., 25 May 2025) rigorously tests models on temporally negative pairs; state-of-the-art models lag at ~34%.
• Pixel-level grounding and multi-turn referential chat: SAMA establishes new benchmarks for multi-turn, referentially grounded video dialogues, achieving state-of-the-art performance in both grounding (mIoU=0.70) and chat metrics (Sun et al., 24 May 2025). PG-Video-LLaVA (Munasinghe et al., 2023) leverages off-the-shelf trackers for plug-and-play pixel localization guided by LLM outputs.
• Cultural and multilingual understanding: ViMUL-Bench (Shafique et al., 8 Jun 2025) tests models across 14 languages and 15 categories (including festivals, food, media, and public figures), providing a baseline for future inclusive LMM development.
• Video anomaly detection: VANE-Bench (Bharadwaj et al., 2024) exposes broad deficiencies in open-source Video-LMMs (<10% accuracy on subtle anomalies), while closed-source models (GPT-4o, Gemini) reach >50–80%.

5. Efficiency, Scalability, and Long-Form Video Analytics

Handling long video sequences within the quadratic context constraints of decoder-style LLMs is a central scalability challenge. Efficient designs exploit adaptive compression, chunked aggregation, late fusion, cross-attention dynamics, and memory hierarchies.

• Adaptive dynamic partitioning: Quicksviewer partitions videos into nonuniform cubes based on moment-to-moment temporal information density, compressing sequences by up to $45\times$ without loss of performance and outperforming uniform baselines by up to +8.72% accuracy (Qi et al., 21 Apr 2025).
• Late fusion pipelines: QMAVIS performs long-form analytics by chunking input, running video and audio through separate pretrained models, and fusing chunkwise captions and transcriptions in a text-only LLM. It achieves +38.75% relative improvement over VideoLLaMA2 on long-video tasks (Lin et al., 10 Jan 2026).
• Plug-and-play efficient strategies: The Slow–Fast architecture (Shi et al., 2 Apr 2025) extends input capacity from 16 to 128 frames at only ~3% extra compute by decoupling fast/slow token streams and introducing cross-attended hybrid layers at select points in the LLM decoder.
• Pooling and cross-attention: CrossLMM combines aggressive spatial pooling (729→9 tokens/frame) with visual-to-visual and text-to-visual cross-attention to achieve competitive accuracy with 4–40× fewer tokens/frame (Yan et al., 22 May 2025).
• Recursive summarization and chunk hierarchies: QMAVIS generalizes to hour-long videos through recursive text summarization and hierarchical chunk aggregation.

6. Empirical Performance and Weaknesses

Despite rapid improvements, Video-LMMs remain far from human-level generalization and robust reasoning. Top proprietary models reach 55–81% on challenging multi-modal benchmarks with audio and subtitles, while open-source solutions lag at 33–52% (Fu et al., 2024, Zhang et al., 2024). Primary weaknesses include:

• Sharp performance drop on long videos;
• Poor robustness to adversarial and misleading prompts;
• Inadequate reasoning over partial, non-existent, physically anomalous, or OOD events;
• Insufficient temporal granularity for atomic event contrast;
• Weak fusion of multimodal evidence for adaptation and context inference.

Progress is being made via elaborate instruction tuning, reward-aware RL, dynamic and recursive aggregation, and expanded multimodal and multilingual benchmarks.

7. Prospective Directions and Recommendations

Future advances are likely to stem from:

• Scalable, memory-augmented architectures for long-horizon video reasoning;
• Cross-modal, culturally aware instruction tuning and reward schemes;
• Unified multimodal benchmarks spanning perception, reasoning, grounding, anomaly localization, and interaction;
• Expanded, high-quality datasets with explicit temporal negatives, context labels, and domain diversity;
• Agentic modules for key-frame selection, streaming perception, and anytime reasoning;
• Tight coupling of chain-of-thought, reward, and confidence estimation in post-training and test-time scaling pipelines.

In sum, Video-LMMs have evolved from basic captioners to emerging reasoners, grounding agents, and cultural interpreters via a proliferation of architectural innovations and increasingly rigorous evaluation. Significant challenges remain in multi-turn interaction, long-context scalability, robust event understanding, and cross-cultural adaptation. The field is actively addressing these through modular fusion, adaptive compression, post-training techniques, and inclusive benchmarks (Zhang et al., 2023, Khattak et al., 2024, Lin et al., 10 Jan 2026, Hu et al., 23 Jan 2025, Shafique et al., 8 Jun 2025, Shi et al., 2 Apr 2025).