Autoregressive Vision-Language Models

• Autoregressive Vision-Language Models are large multimodal architectures that predict joint sequences of visual and textual tokens using a causal Transformer framework.
• They integrate a vision encoder, a text decoder, and cross-modal fusion modules to address tasks such as image captioning, VQA, and conditional image generation.
• Incorporating refinement modules and multimodal pretraining, these models reduce error accumulation and enhance spatial consistency and semantic alignment.

Autoregressive Vision-LLMs (VLMs) are a class of large multimodal models that integrate vision and language processing within a shared next-token prediction framework. These models formulate multimodal reasoning, generation, and understanding tasks as a sequence modeling problem—typically predicting a joint sequence of visual and/or textual tokens in a strictly causal (autoregressive) manner. This paradigm supports powerful unified architectures for tasks spanning captioning, visual question answering, conditional image generation, visual-semantic understanding, and action sequencing.

1. Core Architectural Principles and Objectives

Autoregressive VLMs predict multimodal outputs by factorizing the joint distribution over sequences of image- and text-derived tokens using the chain rule: $$P_\theta(y \mid x, D) = \prod_{t=1}^T P_\theta(y_t \mid y_{<t}, x, D)$$. The dominant implementation backbone is a causal Transformer or LLM, typically augmented with a vision encoder that provides image-derived features or tokens projected into the LLM’s embedding space (Awadalla et al., 2023, &&&1&&&).

Model architectures commonly exhibit the following components:

• Vision Encoder: Often a ViT or CLIP variant; outputs patch-level or pooled embeddings.
• Text Encoder or LLM Decoder: Large, pre-trained causal Transformer, such as Mistral or Vicuna, handling token sequences with self-attention.
• Cross-Modal Fusion: Techniques including Perceiver Resamplers or cross-attention modules insert image-derived tokens or representations into the text sequence, allowing interleaved multimodal context (Awadalla et al., 2023).
• Autoregressive Objective: Models predict the next token (visual or text), conditioned on previous tokens—enabling both language generation conditioned on vision and, reciprocally, visual generation or reconstruction conditioned on language.

In bridge-style VLMs such as OpenFlamingo and LLaVA-1.6, the architecture is typically “frozen vision encoder + projectors + LLM decoder,” with minimal adaptation layers to ensure compatibility (Awadalla et al., 2023, Wang et al., 19 Jan 2026).

2. Autoregressive Supervision in Multimodal Tasks

The autoregressive regime extends to several task settings beyond classic text-only LLMs:

a) Vision-to-Language Generation

E.g., captioning and VQA, where image features are prepended or interleaved before target token prediction. The loss function is a standard cross-entropy over the correct next token:

$$L_{CE}(\theta) = -\frac{1}{T} \sum_{t=1}^T \log P_\theta(y_t^* \mid y_{<t}^*, x, D)$$

(Wang et al., 1 Oct 2025, Awadalla et al., 2023).

b) Visual Sequence Generation

In inpainting, colorization, or edge-detection, a VQGAN tokenization produces discrete image tokens. The model autoregressively reconstructs the entire image token grid, predicting a token at each spatial position in raster order (Wang et al., 1 Oct 2025).

c) Semantic Visual Token Reconstruction

Recent extensions such as Autoregressive Semantic Visual Reconstruction (ASVR) introduce sequence prediction over high-level, semantic tokens produced by a separate quantizer, in addition to textual autoregression. The joint loss sums log-likelihoods of both modalities:

$$L(\theta) = L_{AR}^{text}(\theta) + \lambda \cdot L_{AR}^{vision}(\theta)$$

(Wang et al., 10 Jun 2025).

d) Action Sequence Modeling

In vision-language-action models for robotics, autoregressive decoders generate end-effector (x, y, z, θ) vectors as discrete token sequences, matching the structure of language generation:

$$P(a_1,\ldots,a_k \mid I,T) = \prod_{i=1}^k P(a_i\mid I, T, a_{<i})$$

(Budzianowski et al., 18 Jul 2025).

3. Refinement and Enhancement of Autoregressive Decoding

The sequential nature of autoregressive decoding can induce cumulative errors, particularly in spatially structured visual outputs. Visual Self-Refinement for Autoregressive Models introduces a plug-and-play refinement module, augmenting the base AR backbone post-pretraining:

• The refinement module is a lightweight self-attention block (∼16M parameters), operating on the sequence of predicted token embeddings. This module is trained to minimize the mean cosine distance between refined and ground-truth token embeddings, leveraging global context to jointly revise all tokens:

$$L_{refine}(\phi) = \frac{1}{T} \sum_{t=1}^T [1 - \cos(e_t', e_t^*)]$$

• Only the refiner’s parameters are updated; the AR backbone is frozen during this phase (Wang et al., 1 Oct 2025).

This approach yields measurable improvements in perplexity, FID, PSNR, SSIM, and recall across tasks such as colorization, inpainting, and edge detection. Ablation studies demonstrate the superiority of self-attention versus MLP/CNN refiners, and the efficacy of a cosine similarity loss over L2 distance (Wang et al., 1 Oct 2025).

4. Multimodal Pretraining and Alignment

Multimodal autoregressive pretraining constructs strong cross-modal representations by forcing the model to reconstruct both image and text tokens in a causal order. In models such as AIMv2 (used by AutoRad-Lung):

• Images are patch-tokenized and concatenated with text tokens, forming a joint [vision, text] input sequence.
• An autoregressive transformer decoder reconstructs each element in order—first image patches, then text (or vice versa).
• The loss penalizes negative log-likelihood over both modalities:

$$\mathcal{L}_{pre} = - \sum_{i=1}^I \log p(v_i\mid v_{<i}) - \sum_{t=1}^L \log p(w_t\mid v_{1:I}, w_{<t})$$

(Khademi et al., 26 Mar 2025).

This approach results in encoders capturing fine-grained pixel-level or semantic variations, supporting applications where subtle visual distinctions are critical.

Recent systematic evaluations have demonstrated that autoregressive VLMs (LLaVA-1.6, Qwen2.5-VL) achieve higher cross-modal alignment than diffusion-based VLMs, as measured by cosine similarity of positive image-text pairs and t-SNE clustering in embedding space:

• LLaVA-1.6/Qwen2.5-VL: mean cosine ≈ 0.82–0.85
• LaViDa (diffusion): ≈ 0.75
• MMaDA (diffusion): <0.60 Resulting in stronger performance on classification, VQA, and retrieval tasks (Precision@1 59.7–67.6% vs. 56.2–63.2% for LaViDa, and much larger gap to MMaDA) (Wang et al., 19 Jan 2026).

5. Applications and Task-Specific Adaptations

General Multimodal Reasoning

OpenFlamingo, LLaVA, and Qwen2.5-VL architectures achieve robust performance on captioning, visual QA, and multimodal classification benchmarks, with in-context learning abilities and few-shot generalization (Awadalla et al., 2023, Wang et al., 19 Jan 2026).

Joint Semantic–Textual Understanding

ASVR shows that predicting semantic visual token sequences improves average multimodal benchmark scores by 5 absolute points over text-only AR supervision on large-scale evaluation (e.g., LLaVA-1.5: +5%) (Wang et al., 10 Jun 2025). Pure appearance-based AR supervision degrades multimodal understanding.

Visual Control and Robotics

In vision-language-action settings (OpenVLA), the autoregressive decoder produces robot control tokens sequentially. EVLA demonstrates that removing the AR sequence and directly regressing all outputs yields a 7× speedup with <1% loss in action-accuracy, but AR decoding is essential for tasks with true sequential structure (Budzianowski et al., 18 Jul 2025).

Medical Imaging

AutoRad-Lung leverages an AIMv2 AR vision encoder to capture subtle CT nodule textures. When fused with radiomic-guided prompts, it surpasses CLIP-based counterparts on lung nodule malignancy prediction, especially for ambiguous ("unsure") cases (Khademi et al., 26 Mar 2025).

6. Empirical Evaluation and Comparative Analysis

Empirical studies across AR VLMs demonstrate:

Model	Classification (avg)	VQA (avg)	Retrieval (avg)
LLaVA-1.6	59.7%	57.8%	67.6%
Qwen2.5-VL	58.4%	59.0%	67.5%
LaViDa (diffusion)	56.2%	57.5%	63.2%
MMaDA (diffusion)	33.9%	25.9%	40.4%

Autoregressive VLMs outperform even strong diffusion counterparts in cross-modal alignment and downstream retrieval/classification/VQA performance (Wang et al., 19 Jan 2026).

Refinement modules can further reduce error accumulation: e.g., a 25% reduction in area-under-curve for token cosine distance, and consistent quantitative improvements (e.g., colorization Perplexity ↓20.06→19.01, FID↓59.70→59.24) (Wang et al., 1 Oct 2025).

7. Limitations, Trade-offs, and Future Directions

While AR VLMs exhibit strong cross-modal understanding and sequence modeling, their strict causality may induce error accumulation when modeling spatially structured outputs. Sequential next-token prediction can disrupt the modeling of global spatial dependencies in imagery (Wang et al., 1 Oct 2025).

Augmentation strategies—such as plug-and-play self-attention refinement, semantic token supervision, and radiomic-guided prompt adaptation—mitigate these deficits, particularly for spatial consistency and nuanced medical imaging classification. AR’s representational power is highest when the model is tasked with predicting semantically meaningful, rather than purely low-level, tokens (Wang et al., 10 Jun 2025, Khademi et al., 26 Mar 2025).

Scaling studies show diminishing returns at extremely large data/model sizes, and semantic token vocabularies still risk missing subtle details in high-fidelity applications (Wang et al., 10 Jun 2025). Biomedical and robotic settings require further adaptation and efficiency improvements, as illustrated by the move from AR to non-AR decoders for latency-sensitive edge deployment (Budzianowski et al., 18 Jul 2025).

A plausible implication is that, while AR VLMs remain central to state-of-the-art multimodal modeling, continued research into globally consistent generation (via refinement, semantic supervision, or hybrid bidirectional models) is vital for resolving spatial modeling and efficiency challenges at scale.