InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action for Compositional Generalization

Published 6 Jul 2026 in cs.RO | (2607.04988v1)

Abstract: Unified models for robot manipulation aim to equip one policy with both the semantic priors of pretrained VLMs and the physical dynamics learned through future prediction. In practice, existing designs tend to erode the semantics of the pretrained backbone, suffer interference among heterogeneous objectives, and learn future prediction from scratch in pixel space, leaving the dynamics priors of pretrained video generators unexploited. We present InternVLA-A1.5, which builds the policy on a native VLM backbone that keeps training on VQA and subtask prediction, and attaches a lightweight unified expert for continuous action generation. Future prediction is recast as a latent-querying problem, where a small set of learnable foresight tokens condenses the task-relevant future into a compact latent code under the supervision of a frozen pretrained video generation model, so the policy inherits world-model dynamics priors without ever learning pixel-level generation. The video branch is discarded at inference, keeping real-time control. Pretrained on 1.2M robot episodes and 3M multimodal samples, InternVLA-A1.5 achieves the best overall results on all six simulation benchmarks. In the real world, the preserved semantics deliver the strongest compositional generalization on held-out instruction bindings, and the two designs together sustain long-horizon execution.

Abstract PDF Upgrade to Chat

Authors (29)

First 10 authors:

Summary

The paper introduces a novel framework that unifies vision-language understanding, latent foresight, and continuous action generation for robotic manipulation.
The architecture employs a Mixture-of-Transformers with a pretrained VLM, learnable foresight tokens, and flow-matching action decoding to optimize simulation and real-world performance.
Empirical results demonstrate enhanced compositional generalization, faster convergence, and robust long-horizon planning compared to previous unified models.

Unifying Vision-Language Understanding, Latent Foresight, and Action: An Expert Analysis of InternVLA-A1.5

InternVLA-A1.5 introduces a novel architecture and training paradigm for generalist robotic manipulation, aiming to unify vision-language understanding, spatiotemporal future-prediction (foresight), and continuous action generation by leveraging advances in large-scale pretrained models. The work systematically addresses major limitations in prior unified frameworks and empirically demonstrates substantial gains in both simulation and real-world manipulation tasks requiring compositional generalization and long-horizon planning.

Figure 1: InternVLA-A1.5 unifies understanding, latent foresight, and action using a pretrained VLM, a lightweight expert, and foresight tokens under supervision by a frozen video generation model.

Model Architecture: Mixture-of-Transformers with Latent Foresight

InternVLA-A1.5 builds on a Mixture-of-Transformers (MoT) design. The architecture consists of:

A large-scale pretrained vision-LLM (VLM), specifically Qwen3.5, responsible for multimodal perception and semantic reasoning.
A lightweight “unified expert”, sharing full-attention layers with the VLM but utilizing reduced hidden dimension and separate linear attention pathways for efficient action decoding.
Learnable foresight tokens that act as latent, context-aware queries: these tokens attend to encoded multimodal (image, language, proprioception) context, are projected to conditioning vectors for a frozen video generative model, and thereby acquire spatiotemporal dynamics priors.
Action query tokens subsequently decoded via a flow-matching objective for high-frequency continuous control, supporting efficient closed-loop execution.

Crucially, foresight tokens and continuous action heads are disentangled: foresight is trained exclusively via backpropagation through the conditioning path of a strong video generator (WAN2.2), while action heads receive gradients from flow-matching objectives only. The video generator is used solely during training and is omitted during inference, maintaining real-time performance.

Figure 2: MoT architecture with a pretrained VLM, unified expert with foresight and action tokens, and flow-matching action prediction.

This architectural separation, combined with careful prompt and data design, enables robust transfer of both semantic and world-dynamics priors—addressing knowledge erosion and objective interference observed in prior unified models.

Training Paradigm and Data Strategy

InternVLA-A1.5 employs a two-stage pretraining and a multi-objective loss function. Key components:

Stage 1: Joint supervision of the VLM on vision-language question-answering (VQA), subtask description, and discretized action token prediction (FAST tokenization), using a unified chat-style prompt.
Stage 2: Introduction of the unified expert; implementation of foresight using learnable tokens optimized via video prediction loss against a frozen video generator; and flow-matching based continuous action regression.

The training loss combines VLM next-token cross-entropy, foresight supervision (video prediction loss), and action flow-matching, with empirical weighting to avoid interference.

Figure 3: Unified chat-prompt structure for both robot action and VQA samples.

Figure 4: Foresight tokens act as context-aware latent queries, with supervision from the video world model.

Figure 5: Hybrid attention masking supports distinct training/inference workflows and prevents leakage between action heads.

This multi-stage recipe is supported by an extensive heterogenous data pipeline:

1.2M robot episodes (861M frames) aggregated from synthetic and real-world sources, with domain- and embodiment-aware sampling strategies to ensure diversity and sim-to-real coverage.
~3M multimodal samples, including general VQA and spatial/trajectory grounding tasks, ensuring maintenance of semantic ability in the VLM backbone.
Figure 6: Large-scale robot (a) and multimodal (b) training data composition and sampling strategy.

Experimental Results: Simulation and Real-World

Real-World Manipulation and Compositional Generalization

Evaluation was performed on four real-world long-horizon and instruction-following tasks. A notable protocol element is the held-out instruction binding setting, which tests compositional generalization by withholding certain binding pairs (tube color, arm, position) during training:

Figure 7: Example real-world instruction-following and long-horizon tasks.

InternVLA-A1.5 outperforms strong baselines ( $\pi_{0.5}$ , Motus) in most real-world and compositional settings:

Figure 8: Success rates across three instruction-following and one long-horizon task (MOF).

Figure 9: Generalization to in-domain vs. out-of-distribution instruction bindings reveals a particular advantage in unseen (compositional) settings.

Significant gains are observed in tasks requiring robust grounding of previously unseen instruction compositions and for long-horizon sequential procedures, attributed to preserved semantic priors and effective temporal/world-model understanding.

Simulation Benchmarks

InternVLA-A1.5 achieves state-of-the-art or highly competitive results on six challenging simulation benchmarks, including LIBERO, RoboTwin 2.0, SimplerEnv, EBench, LIBERO-Plus (robustness under distribution shift), and DOMINO (dynamic manipulation):

LIBERO: Achieves 98.9% average success, highest across all methods.
LIBERO-Plus: 84.8% total in zero-shot robustness, exceeding all baselines.
RoboTwin 2.0: 93.2% average, nearly invariant to domain randomization.
DOMINO: Outperforms all baselines without fine-tuning (zero-shot).

Ablation studies demonstrate that both video foresight loss and explicit foresight tokens are causally important—removal of either reduces generalization, especially on zero-shot, perturbed, or long-horizon tasks.

Analysis: Training Efficiency and World Model Integration

InternVLA-A1.5 converges faster and achieves lower final supervised loss during downstream adaptation relative to prior unified models. This suggests that the architecture and pretraining regime yield a more favorable optimization landscape and extract richer transferable representations.

Figure 10: SFT training loss curves highlight improved convergence and stability.

Visualization of future rollout predictions, generated via the frozen WAN model and foresight embeddings, qualitatively confirms plausible and consistent multi-step physical reasoning for both manipulation and environment-level consequences—covering kinematic and non-kinematic dynamics across seen and zero-shot cases.

Figure 11: Foresight-conditioned multi-frame rollouts demonstrate accurate, causally-consistent scene evolution.

A key interpretive takeaway is that effective world-model knowledge can be distilled into compact latent foresight codes without incurring the cost of explicit pixel-level video prediction at inference.

Implications and Future Directions

Theoretical and Practical Impact

The results establish that (i) preserving semantic priors of VLMs via unified prompt/label/objective design, and (ii) injecting world-dynamics priors using latent foresight tokens and pretrained video models, are both necessary and sufficient for high-level generalization and long-horizon handling in robotic agents. The method overcomes prevalent degradations in semantics/dynamics transfer observed in prior VLM+control integrations.

In practice, by discarding video branches during inference, InternVLA-A1.5 delivers real-time closed-loop control, a critical requirement for embodied applications. The approach sets a blueprint for integrating advanced world models into action policies without runtime costs.

Limitations and Future Work

Foresight horizon: Currently, foresight supervision spans only short horizons (one action chunk); future work must address explicit long-range imagination or planning.
World model limitations: The video generator is kept frozen and generic—thus, the transferred priors are bounded by its coverage of embodied scenes. Tailoring or extending video world models for specific robotic domains is a promising avenue.
Planning and memory: While subtask prediction supports long-horizon progression, purely reactive policies may still face limitations on tasks requiring explicit memory or receding-horizon planning.

Conclusion

InternVLA-A1.5 provides compelling evidence that a judicious combination of maintained semantic VLM priors, latent foresight distillation from strong pretrained world models, and efficient continuous action modeling delivers improved generalization, robustness, and practical usability for robotics. The architectural and algorithmic innovations directly address known failure modes in unified models, setting a precedent for future research on long-horizon, robustly generalizing policy architectures.