CLAP: Direct VLM-to-VLA Adaptation via Language-Action Grounding

Abstract: Vision-language-action models (VLAs) inherit semantic capabilities from pretrained VLMs, yet large-scale post-training on robot data and architectural modifications can reshape the backbone so extensively that it becomes difficult to isolate what the VLM contributes to control. Directly converting pretrained VLMs into VLAs with minimal architectural change offers a more transparent path to understanding how VLM capabilities transfer across model scales. The core obstacle is output-distribution mismatch: predicting actions as bare numeric token sequences moves generation away from the VLM's pretrained language distribution, degrading the capabilities we seek to preserve. To address this, we propose CLAP (Causal Language-Action Prediction), which prepends each numeric action sequence with a natural-language action description, causally conditioning precise action-token prediction on a language-action plan without modifying the backbone architecture. With single-epoch fine-tuning alone, 2B CLAP achieves 90.8% on LIBERO (+14.9 pt over VLA-0) and improves robustness on LIBERO-PRO under language, object, and spatial perturbations. We will release CLAP at 0.8B, 2B, and 4B as an open-weight, multi-scale compact VLA family from a single VLM lineage, enabling controlled analysis of VLM-to-VLA capability transfer.

• The paper introduces a novel language-action grounding approach that prepends deterministic natural-language action descriptions to numeric tokens.
• It leverages pretrained VLMs without architectural changes to significantly boost control performance, achieving up to a 90.8% success rate on LIBERO benchmarks.
• The method demonstrates robust real-world deployment on a UR5e arm and opens new avenues for scalable, democratized robotic learning research.

CLAP: Direct VLM-to-VLA Adaptation via Language-Action Grounding

Introduction

CLAP (Causal Language-Action Prediction) (2607.08974) introduces a minimal recipe for converting pretrained Vision-LLMs (VLMs) into Vision-Language-Action (VLA) models without architectural modification or auxiliary action experts. The work addresses the critical challenge of output-distribution mismatch between language-pretrained sequence models and the bare numeric action token sequences commonly used in VLA fine-tuning. By prepending a deterministic, natural-language summary of the predicted actions before the tokenized robot actions, CLAP maintains alignment with the VLM's pretraining distribution, resulting in improved control performance, robustness to distribution shift, and a controlled mechanism for evaluating capability transfer across scaling regimes.

Figure 1: CLAP converts a pretrained VLM directly into a deployable VLA by prepending a natural-language action description to numeric action tokens, keeping every prediction step closer to the VLM pretraining distribution. CLAP requires no action expert or architectural change.

Method: Causal Language-Action Prediction

The core innovation in CLAP is the restructuring of the output sequence for VLA fine-tuning. Rather than directly predicting a flattened integer sequence of robot actions, CLAP modifies the target such that a concise, templated language description—generated deterministically from the ground-truth action chunk—precedes the action tokens. This composite sequence is generated autoregressively; thus, the numerically coded actions are conditioned on the language description at every step, leveraging the pretrained token dependencies of the VLM.

Figure 2: Overview of CLAP. CLAP changes only the target output sequence for VLA fine-tuning. During training, the language-action description is generated from the action chunk by a fixed template; at inference, both language description and numeric action are generated autoregressively.

This approach achieves three objectives:

1. Semantic retention: By preserving sequential generation within the model's pretrained domain (i.e., structured natural language), CLAP avoids catastrophic forgetting of semantic and spatial reasoning abilities.
2. Direct executability: Despite the language prefix, the output still contains fully-discretized numeric tokens for direct robot execution.
3. No auxiliary modules: There is no action expert, output head change, or vocabulary extension, keeping the backbone, tokenizer, and cross-entropy objective unchanged.

Experimental Evaluation

Learning Efficiency and Robustness

CLAP is instantiated using Qwen3.5 VLMs at scales of 0.8B, 2B, and 4B, and evaluated under single-epoch fine-tuning on the LIBERO benchmark, comparing directly against VLA-0 under identical protocols. Across all sizes, CLAP delivers substantial performance improvements over VLA-0:

• 2B CLAP: Achieves a 90.8% average success rate on LIBERO, a +14.9 pt improvement over VLA-0.

Success is broad-based, with notable gains on spatial, object, goal-conditioned, and long-horizon tasks. These improvements stem solely from the output representation change—no added data, model capacity, or action-specific modules.

Figure 3: LIBERO Success Rate of [email protected] across training epochs on the four LIBERO suites. Success exceeds 90% on Spatial, Object, and Goal already at epoch~1, while Long benefits from one additional epoch before saturating.

On the LIBERO-PRO OOD generalization suite, CLAP models consistently outperform VLA-0, especially with larger backbones and under spatial and semantic perturbations. The gains are largest in the 4B model with action masking, providing further evidence that conditioning numeric actions on language descriptions yields more robust, generalizable policies.

Capability Transfer and Scaling Analysis

Qwen3.5 backbones are benchmarked pre-finetuning on VLABench to evaluate multi-dimensional capabilities (e.g., spatial reasoning, mesh and texture understanding, semantic comprehension, physics law inference, and complex task composition).

Figure 4: VLABench capability profiles before robot fine-tuning. Comparing non-CoT (left) and CoT (right) prompting shows 2B as consistently balanced, while CoT primarily improves the 4B model on Spatial and Mesh/Texture tasks. Trends motivate CLAP's language-action design for compact fine-tuning.

Figure 5: VLABench capability profiles under 0-shot prompting. Across model scales, Physical Law accuracy collapses without in-context example, emphasizing the value of demonstration.

Despite the expectation of monotonic scaling, the 2B model generally outperforms the 4B model on both VLABench and downstream VLA tasks, demonstrating that parameter count alone does not guarantee better control transfer. Furthermore, the experiments show that "reasoning" style prompting (CoT) particularly benefits high-capacity VLMs and certain task dimensions, but can introduce output instability in smaller backbones. CLAP’s deterministic language-action prefix provides a robust middle ground, minimally relying on unconstrained reasoning traces.

Real-World Robot Deployment

CLAP is directly deployed on a UR5e arm in challenging, cluttered pick-and-place scenarios using demonstration data from only 120 episodes with three target objects and multiple distractors.

Figure 6: Real-Robot Setup. A UR5e arm with a Robotiq 2F-85 gripper, equipped with wrist and fixed RGB cameras for observation.

Figure 7: Pick-and-Place Task in Cluttered Scene. Left: Several randomized object layouts; Right: An OOD scenario with unseen objects.

Results align with simulation findings: the 2B CLAP model achieves 60% success in both in-distribution and out-of-distribution settings, while 0.8B performance drops dramatically under OOD perturbations. This highlights the practical value of backbone capacity for robustness in physically diverse environments and the potential for scaling up real robot datasets in future work.

Practical and Theoretical Implications

CLAP's principal contribution is demonstrating that output-target reformulation—rather than architectural augmentation, expert networks, or multitask pretraining—is a highly effective and efficient route to capability-preserving VLM-to-VLA adaptation. The method's transparency enables systematic investigation of capability transfer as a function of backbone, prompting regime, and data regime. The fact that single-epoch, single-node training is sufficient to approach and sometimes match robot-pretrained systems (SmolVLA, $\pi_{0.5}$, OpenVLA) (Shukor et al., 2 Jun 2025, Intelligence et al., 22 Apr 2025) [kim2024openvlaopensourcevisionlanguageactionmodel], on both in-distribution and OOD tasks, underlines its relevance to scalable, democratized robot learning research.

Practically, CLAP opens several future research avenues:

• Scaling to diverse backbones and multi-modality (beyond Qwen).
• Joint study of language-action and alternative reasoning intermediates.
• Systematic exploration of efficiency and latency via compression, quantization, and accelerated decoding.

Conclusion

CLAP establishes a simple, effective, and transparent baseline for VLM-to-VLA conversion. By causally conditioning action prediction on a language-action prefix, it achieves strong closed-loop performance and robustness across simulation and real-world robotic manipulation, without any architectural change or additional modules. This output-representation-centric approach enables principled studies of capability transfer and exposes new opportunities for lightweight, accessible VLA development, accelerating progress in generalist robot learning.