$M^2$-VLA: Boosting Vision-Language Models for Generalizable Manipulation via Layer Mixture and Meta-Skills

Abstract: Current Vision-Language-Action (VLA) models predominantly rely on end-to-end fine-tuning. While effective, this paradigm compromises the inherent generalization capabilities of Vision-LLMs (VLMs) and incurs catastrophic forgetting. To address these limitations, we propose $M^2$-VLA, which demonstrates that a generalized VLM is able to serve as a powerful backbone for robotic manipulation directly. However, it remains a key challenge to bridge the gap between the high-level semantic understanding of VLMs and the precise requirements of robotic control. To overcome this, we introduce the Mixture of Layers (MoL) strategy that selectively extracts task-critical information from dense semantic features. Furthermore, to facilitate efficient trajectory learning under constrained model capacity, we propose a Meta Skill Module (MSM) that integrates strong inductive biases. Extensive experiments in both simulated and real-world environments demonstrate the effectiveness of our approach. Furthermore, generalization and ablation studies validate the architecture's zero-shot capabilities and confirm the contribution of each key component. Our code and pre-trained models will be made publicly available.

• The paper introduces a framework that preserves semantic integrity by decoupling visual-language perception from robotic control using a frozen VLM.
• It employs a Mixture-of-Layers mechanism to selectively aggregate multi-layer VLM representations, filtering noise while focusing on spatially relevant features.
• The Meta Skill Module retrieves task-relevant meta-skills from an external memory to refine action trajectories, achieving high success in both simulation and real-world tests.

$M^2$-VLA: Advancing Vision-Language-Action Generalization via Layer Mixture and Meta-Skills

Motivation and Problem Analysis

Generalizable manipulation in Embodied AI requires robust semantic comprehension and actionable spatial features. Traditional Vision-Language-Action (VLA) methodologies depend on fine-tuning pre-trained Vision-LLMs (VLMs), substantially impairing their semantic generalization via catastrophic forgetting and degrading visual question answering (VQA) capabilities in out-of-distribution instruction or object settings. The authors empirically demonstrate this phenomenon: state-of-the-art VLA approaches excel on in-domain tasks but suffer pronounced performance collapses when confronted with tasks or instructions outside the training distribution, losing their VQA performance (Figure 1).

Figure 1: Catastrophic forgetting in VLM backbones; SOTA VLA models lose VQA capability when fine-tuned for manipulation.

Preserving the semantic integrity of VLMs is thus essential for manipulation policy generalization. The challenge is to exploit the semantic abstractions of frozen VLMs for low-level robotic action generation without direct fine-tuning, requiring both selective feature extraction and augmented action head capacity.

$M^2$-VLA Framework Overview

The $M^2$-VLA framework leverages a frozen, generalized VLM as the perceptual backbone, decoupling perception from control. Visual and language tokens, including learnable queries, are concatenated and processed by the VLM for hierarchical feature extraction. Action generation is realized via a denoising transformer head. Two principal innovations are introduced:

1. Mixture of Layers (MoL): This mechanism selectively aggregates multi-layer VLM representations through decoupled cross-modal attention, filtering out irrelevant semantic information for manipulation refinement.
2. Meta Skill Module (MSM): MSM augments trajectory learning by retrieving task-relevant meta-skills—previously stored perceptual-action pairs—enabling inductive bias injection and efficient action refinement.

Figure 2 details the architectural flow and interaction of these modules.

Figure 2: $M^2$-VLA framework. The model concatenates vision, language, and learnable queries, extracts hierarchical VLM features via MoL, and refines actions using MSM.

Mixture-of-Layers (MoL): Hierarchical Feature Selection

MoL is constructed as three parallel attention branches: visual, textual-proprioceptive, and action-self attention. Each VLM layer's representations are interpreted as 'experts' in the Mixture-of-Experts perspective. Independent, layer-wise attention gates dynamically aggregate task-critical information, discarding noisy or redundant features.

MoL is applied at each action head transformer layer, with modality-decoupled attentions to counteract cross-modal scale inconsistency. Empirical visualization confirms that middle-layer VLM features receive higher MoL attention, and their heatmaps concentrate on spatially relevant regions (gripper and target) (Figure 3, Figure 4).

Figure 3: MoL architecture. Three decoupled attention branches integrate layer-wise VLM features for downstream action generation.

Figure 4: Layer-wise instruction attention heatmaps; middle layers focus on manipulation-relevant regions.

Meta Skill Module (MSM): Trajectory-Induced Knowledge Reuse

MSM maintains an external memory buffer of perceptual keys (concatenated hierarchical VLM features) and action values (trajectory chunks). During inference and training, MSM retrieves nearest-neighbor meta-skills via $L_1$ distance between current observations and buffer keys, projecting retrieved actions into latent space and integrating them through cross-attention and residual connections.

Correlation analysis verifies that the retrieval strategy establishes a strong association between perceptual context keys and action values, facilitating effective skill transfer (Figure 5, Figure 6).

Figure 5: MSM architecture. Maintains perceptual-skill repository and integrates retrieved meta-skills for action refinement.

Figure 6: Strong positive correlation between Keys and Values in MSM, indicating effective meta-skill retrieval.

Empirical Validation: Generalization and Robustness

Generalization Tasks: $M^2$-VLA achieves high success rates on instruction-following (66.2%, 29.4% drop) and novel-object tasks (34.4%, 30.4% drop) compared to SOTA fine-tuned baselines, which experience much larger performance degradation.

Simulation Benchmarks: Across all four LIBERO suites, $M^2$-VLA outperforms both fine-tuned and frozen VLM approaches, achieving average success of 95.3% with minimal model parameters (0.3B).

Real-world Manipulation: On four robust manipulation tasks (instruction variation and novel objects), $M^2$-VLA achieves success rates of 85–90% on fundamental tasks and 75–80% on generalization tasks, surpassing baselines by a substantial margin. Sequential rollout images demonstrate reliable execution across diverse scenarios (Figure 7).

Figure 7: Robotic trajectories in real-world tasks; $M^2$-VLA reliably executes manipulation across object and instruction variations.

Ablation and Visualization Studies

Ablations confirm that both MoL and MSM are integral: removal of either module results in a significant drop in success rate. Visualization evidences MoL's proficiency in dynamically filtering spatial manipulation cues and MSM's effective skill retrieval based on hierarchical context similarity.

Implications and Future Directions

The $M^2$-VLA paradigm substantiates that robust VLMs can be harnessed as frozen backbones for manipulation policy generalization without sacrificing semantic capabilities. Practical implications include efficient adaptation of large-scale VLMs to robotic manipulation under constrained compute, with strong zero-shot abilities for unseen instructions and objects.

Theoretically, the decoupled MoL and MSM modules instantiate new forms of modular policy structures capable of complex compositionality and cross-modal integration, potentially generalizable to other Embodied AI domains.

Future developments may explore multi-action head architectures for supporting disparate tasks and robot embodiments within a universal VLA framework, leveraging shared VLMs and specialized controllers.

Conclusion

$M^2$0-VLA demonstrates that frozen, generalized Vision-LLMs, augmented with Mixture-of-Layers and Meta Skill Module strategies, serve as effective perceptual backbones for task-agnostic manipulation. The architecture achieves strong generalization across instruction and object variations, validated in simulation and real-world, with efficient training and robust policy performance. Upcoming works may extend this blueprint toward universal, multi-embodiment VLA agents, reinforcing the modular backbone-control decoupling for scalable embodied intelligence.