LLaVA-1.5/NeXT: Unified Multimodal Fusion

• LLaVA-1.5/NeXT is a unified multimodal system that fuses visual and textual tokens through minimal cross-modal mixing, enabling versatile single-image, multi-image, and video comprehension.
• It leverages a frozen vision encoder, a shallow trainable projector, and a large pretrained LLM to process diverse scenarios without needing specialized adapters.
• A multi-stage training curriculum with adaptable token allocation drives state-of-the-art 0-shot benchmark performance across a range of visual-language tasks.

LLaVA-1.5/NeXT refers to a series of large multimodal models (LMMs) designed to unify visual and linguistic reasoning with minimalistic, extensible architectures. Originating from the LLaVA-1.5 baseline, these systems evolved through LLaVA-NeXT and culminated in LLaVA-OneVision. The core innovations involve streamlined cross-modal fusion, adaptable token allocations, and scaling principles that enable a single model to excel across single-image, multi-image, and video understanding tasks. The platform serves as a foundation for research in transfer learning, emergent multimodal reasoning, and scenario-agnostic visual-language modeling.

1. Architectural Principles and Model Evolution

LLaVA-1.5 preserves a three-module design: a frozen vision encoder (typically CLIP or SigLIP), a shallow trainable projector, and a large pretrained LLM. The system encodes visual inputs through the fixed encoder, maps them linearly via the projector, and injects these tokens into the LLM input sequence, processed autoregressively in tandem with text tokens.

LLaVA-NeXT introduced architectural adjustments to support “AnyRes” input, removing the restriction of a fixed-size input by allowing arbitrary-resolution crops and pooling strategies. This led directly to LLaVA-OneVision, which generalizes the approach further by:

• Utilizing SigLIP as the vision encoder and Qwen-2 as the LLM backbone.
• Introducing “Higher-AnyRes” fusion, in which cropping strategies and token pools dynamically adapt to resolution and input scenario.
• Aligning token budgets across single-image, multi-image, and video sequences to ensure similar context sizes for the LLM.
• Maintaining minimal fusion: all cross-modal mixing is handled by the LLM’s own self-attention layers, with no additional cross-attention adapters or scenario-specific blocks.

Formally, visual representations $v'$ are projected from encoder outputs $g(x)$ by $p(v)$, yielding $v' = p(g(x)) \in \mathbb{R}^{L \times d_{model}}$. These tokens are concatenated with text (following a special <image> delimiter) and processed together:

$p(x) = \prod_{i=1}^{|x|} p(x_i | x_{<i})$

where $x$ denotes the multimodal sequence.

2. Training Protocol and Optimizations

The LLaVA-1.5/NeXT sequence uses a multi-phase curriculum:

• Stage 1: Language–Image Alignment 558K image–caption pairs are used to optimize only the projector by mean-squared alignment of projected visual tokens to LLM embeddings:

$$\mathcal{L}_{\text{align}} = \|p(g(x)) - e_{img}(x)\|^2_2$$

• Stage 1.5: High-Quality Knowledge Learning 3.5M self-captioned images, 1.1M OCR/text, and 235K language-SFT samples; cross-entropy loss for synthetic captions.
• Stage 2: Visual Instruction Tuning Divided into (2.1) single-image and (2.2) mixed-scenario (single + multi-image + video) phases, training on 5.2 million instructions (3.2M single-image, 1.6M multi-image, 0.35M video). Both LLM and projector are updated; some ViT layers may be fine-tuned at a lower rate.

All modules are trained for one epoch per stage, with increasing image resolution and token count throughout the process.

3. Unified Cross-Scenario Processing

The defining attribute of LLaVA-NeXT and OneVision is scenario-unified modeling:

• All modalities (images, image groups, video frames) are converted to token sequences.
• Videos are treated as concatenated multi-image sequences.
• The LLM’s token budget is statically balanced:

$$\text{max\_tokens\_single} \simeq \text{max\_tokens\_multi} \simeq \text{max\_tokens\_video} \simeq 7290$$

• Fusion occurs implicitly through self-attention; no scenario-specific mechanisms such as temporal modeling or cross-modal adapters are introduced.
• The curriculum starts from supervised single-image alignment, followed by broad transfer to multi-modal and sequential data in later stages.

This architecture enables strong generalization from high-quality single-image pretraining to multi-image and video understanding, with no specialized architectures required for each scenario (Li et al., 2024).

4. Benchmark Performance and Quantitative Analysis

LLaVA-OneVision (with Qwen-2, 72B parameters) has been benchmarked on a diverse suite of single-image, multi-image, and video datasets. Notable 0-shot results include:

Benchmark	OneVision (%)	Notable Baselines (%)
AI2D	85.6	GPT-4V 78.2 / GPT-4o 94.2
MMBench	85.9	GPT-4V 75.0
MathVista	67.5	GPT-4V 49.9 / 63.8
NLVR2 (multi)	93.8	GPT-4V 88.8 / 91.1
ActivityNet-QA	62.3	GPT-4V 57.0
VideoMME	66.2	59.9 / 71.9

Across the board, OneVision-72B matches or surpasses GPT-4V, and in some single-image and video tasks, it narrows the margin to GPT-4o. The absence of scenario-specific modules makes these results particularly notable, exemplifying successful transfer via unified representation (Li et al., 2024).

5. Emergent Abilities and Qualitative Insights

OneVision, without scenario-specific adaptation, demonstrates emergent compositional reasoning:

• Correct integration of chart and diagram data in financial reasoning.
• Describing multi-step GUI actions from sequences of screenshots.
• Referring to object sets within images without explicit per-object training.
• Responding appropriately to visual prompts and highlighting, including identification in both still and sequential (video) data.
• Robust video difference and event understanding, outperforming scenario-specific open LMMs on qualitative examples.

This suggests the curriculum, minimal fusion design, and token balancing confer real cross-modal transfer, yielding abilities not directly represented in the explicit training objectives (Li et al., 2024).

6. Contextual Significance and Broader Impact

The LLaVA-1.5/NeXT lineage represents a shift away from task-specialized multimodal pipelines towards scenario-agnostic, LLM-centered fusion. The technical design—a vision encoder (often SigLIP), a shallow projector, and a single LLM processing all modalities—proves sufficient for state-of-the-art results across a wide range of visual-linguistic benchmarks. It enables open models to compete directly with proprietary LMMs such as GPT-4V, Gemini-1.5, and Claude-3.5.

A plausible implication is that future visual-language research may increasingly focus on curriculum design and unified objective functions, rather than on scenario- or modality-specific architectural changes. This architectural minimalism, combined with extensive high-quality data and staged training, points toward robust, transferable, and scalable multimodal intelligence (Li et al., 2024).