CoRE-VLA: Towards Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Experts

Abstract: Vision-language-action (VLA) models have advanced generalist robotic manipulation, yet real-world deployment reveals a fundamental challenge: robots are equipped with diverse and heterogeneous sensor configurations, auxiliary sensors can fail unexpectedly during operation, and different robot embodiments often lack certain sensors by design. A unified policy that can exploit auxiliary perceptual inputs when available while remaining reliable under sensor absence, whether incidental or by design, is therefore essential for practical deployment. However, existing VLA policies couple action generation to a fixed sensor set through shared dense computation, making them brittle when sensors are missing and limiting their ability to specialize across diverse tasks and long-horizon behaviors. We propose CoRE-VLA, a scalable and robust VLA framework that formulates action generation as context-conditioned sparse computation. Sensor availability gates modality-specialized experts, enabling graceful degradation under missing sensors without retraining. Task intent further routes action-side representations to task-relevant experts, improving specialization across diverse tasks and long-horizon subgoals. While the framework is designed to accommodate different auxiliary sensors, we focus on depth as a representative and practically important auxiliary modality in our experiments. Experiments on LIBERO, RoboCasa GR1 Tabletop, and real-world dual-arm manipulation show that CoRE-VLA achieves strong results on long-horizon and multi-task benchmarks, and outperforms both a dense-action-generator ablation and a strong pretrained VLA baseline, including in zero-shot generalization to unseen scenarios. Modality analysis shows that CoRE-VLA can exploit auxiliary depth when available while remaining robust when depth is unavailable during deployment.

• The paper introduces a conditional routing mechanism that enables VLA models to robustly handle missing or unreliable auxiliary sensors in robotic manipulation tasks.
• It integrates vision-language models with modality-specific experts and employs sparse, task-intent conditioned routing to optimize long-horizon and multi-task performance.
• Empirical evaluations demonstrate near-perfect simulation success and graceful degradation in real-world tasks when auxiliary sensors are absent.

CoRE-VLA: Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Experts

Introduction and Motivation

CoRE-VLA addresses persistent challenges in vision-language-action (VLA) policy learning for robotic manipulation in real-world scenarios, particularly sensor heterogeneity and robustness to missing or unreliable auxiliary perceptual inputs. Existing VLA models typically assume a fixed set of sensors, leading to policy brittleness when auxiliary modalities (such as depth, force, or tactile sensors) are absent or fail during deployment. Furthermore, the standard shared dense action generators limit adaptability across diverse tasks and hinder scalability for long-horizon or multi-task settings due to negative transfer and conflicting gradients.

CoRE-VLA proposes a modular and context-aware architectural and training approach that enables policies to (1) adaptively exploit auxiliary perceptual cues when present, (2) gracefully degrade when such cues are unavailable, and (3) specialize computation for different tasks and temporal subgoals, thereby advancing both robustness and task-specific performance in real-world deployments.

CoRE-VLA Architecture

Framework Overview

CoRE-VLA is designed as an extension of the action-diffusion transformer (Action DiT) framework and incorporates pretrained vision-LLMs (VLMs) for encoding multimodal input (RGB observations, language instructions). Auxiliary modalities (e.g., depth) are processed by dedicated modality encoders. The key innovation lies in conditioning action generation on both sensor availability and high-level task intent, which is achieved by gating specialized expert modules and routing informative internal representations accordingly.

Figure 1: The CoRE-VLA architecture encodes multimodal input and routes action representations to general and modality-specialized experts conditioned on sensor availability and task intent.

Conditional Routing of Experts (CoRE) Block

Within the Action DiT, selected transformer layers are augmented with CoRE blocks, which implement two primary forms of sparse, context-conditioned computation:

1. Task-Intent Conditioned Representation Selection: The architecture sparsely selects a subset of action-side representations based on pooling VLM-encoded language tokens, extracting a task intent embedding that guides which representations receive expensive conditional processing.
2. Modality-aware Expert Routing:

For selected representations, CoRE dynamically assigns processing to general experts or modality-specialized experts. Modality-specialized experts are only active if the corresponding auxiliary modality is available, controlled by a binary indicator and masked during both training (via modality dropout) and inference as needed.

Figure 2: The CoRE block selectively routes representations to experts based on task intent and modality availability.

This mechanism enables the overall action generator to support extensibility to new sensor inputs without retraining from scratch and, critically, prevents over-reliance on sensors that may be missing at deployment.

Training Objectives and Inference

CoRE-VLA leverages a flow-matching objective over action chunks, using supervised trajectories and Gaussian noise to learn a velocity field mapping that can be integrated to produce smooth, multimodal actions. Training further incorporates (1) a selection regularizer to enforce sparse computation according to target capacity, and (2) a Mixture-of-Experts (MoE) load-balancing loss to encourage equitable utilization across available experts, preventing routing collapse.

At deployment, the model adapts dynamically to the available sensory context by reconfiguring the computation graph: modality-specialized experts are masked when the associated sensors are unavailable, and the routing of computation is determined by the current instruction and sensory state.

Benchmark Results

Simulation: LIBERO and RoboCasa GR1 Tabletop

CoRE-VLA is evaluated on two major simulation benchmarks, demonstrating strong results in both multi-task and long-horizon manipulation scenarios. The framework incorporates noisy depth as a representative auxiliary modality and explicitly evaluates both extension and degradation under missing modalities.

Notably, in the LIBERO benchmark, CoRE-VLA achieves the highest recorded average success rates across all major suites:

• LIBERO-Spatial: 99.0%
• LIBERO-Object: 99.2%
• LIBERO-Goal: 98.8%
• LIBERO-Long (Long-horizon): 97.6%
• Overall Average: 98.7% (higher than GR00T-N1.7, $\pi_{0.5}$, and OpenVLA baselines)

These numbers indicate not only strong task-level capability but also pronounced robustness and specialization in long-horizon and sensor-variable environments. Critically, ablation studies show that CoRE-VLA degrades gracefully when depth is unavailable—only a 0.6 point drop in performance for LIBERO-Long—contrasting with sharp drops in naive depth-augmented policies.

In RoboCasa GR1 Tabletop, CoRE-VLA achieves an average success rate of 56.5%, outperforming both the best dense model baselines and prior mixture-of-experts architectures.

Real-World Robotic Manipulation

CoRE-VLA is validated on an AgileX ALOHA dual-arm robot in three complex manipulation tasks: Vegetables-Picking, Clothes-Folding, and Fabric-Folding. The Clothes-Folding and Fabric-Folding tasks, in particular, require coordinated bimanual, long-horizon, and contact-rich manipulation capabilities, as well as zero-shot generalization to novel deformable objects.

Key numerical results:

• On Fabric-Folding (zero-shot to new materials/colors): achieves 87.5 task score and 80% success with physical depth camera and 75/70 when depth is absent.
• Outperforms dense Action DiT and strong $\pi_{0.5}$ pretrained baselines across all tasks and success metrics.
• Delivers consistent gains both with and without auxiliary depth at inference.

  Figure 3: Real-world rollout on Vegetables-Picking, demonstrating spatial reasoning and long-horizon object transfer.

  

  Figure 4: Real-world rollout on Clothes-Folding, showcasing complex bimanual manipulation of deformable garments.

  

  Figure 5: Zero-shot transfer to diverse fabrics in Fabric-Folding, despite training only on gray cloth demonstrations.

Modality Robustness and Ablation Insights

A systematic ablation on LIBERO-Long elucidates several high-level claims and empirical effects:

• Task-intent routing alone (w/o modalities) improves long-horizon generalization due to specialized computation for sequential subgoals.
• Naive depth augmentation increases performance when present, but exhibits significant drops (−4.4%) when unavailable, unless matched with explicit modality dropout and specialized experts.
• Combining modality dropout and modality-specialized experts delivers both extensibility (higher with depth) and robustness (minimal performance drop without depth).
• Routing conditioned on instruction tokens is more effective than generic pooled multimodal representations, highlighting the importance of instruction-focused adaptation.

Theoretical and Practical Implications

Theoretical Implications

CoRE-VLA's context-conditioned sparse computation offers a principled approach to modularizing robot policy learning, decoupling shared manipulation knowledge from task- and modality-specific procedures. The architecture elegantly integrates advances in sparse routing (MoE, conditional computation) with flow-matching generative action policies, providing a formal path toward scalable, generalist robot policy optimization under changing embodied contexts.

Practical Implications

The framework enables real-world deployment of VLA policies in heterogeneous fleets, gracefully accommodating differing and unreliable sensor layouts. It provides a scalable path for unifying vision, language, proprioception, and arbitrary auxiliary sensors under a single action policy, reducing the engineering cost of supporting additional modalities or robot hardware. The architecture's robustness to missing modalities critically reduces operational risk and maintenance costs in practical embodied deployments.

Future Directions

Prospective developments include scaling the CoRE-VLA architecture to larger, more heterogeneous robotics datasets spanning a wider array of physical embodiments and extending the same design paradigm to richer sensing suites (e.g., force/torque, tactile, haptics). There is significant opportunity in further automating the routing policy using reinforcement learning or hierarchical planning, and in applying similar conditional computation frameworks to other domains of embodied decision-making.

Conclusion

CoRE-VLA introduces a context-conditional, multi-expert action generation paradigm for vision-language-action robotics, providing robust, extensible, and scalable capabilities across variable sensor configurations and manipulation tasks. Its empirical superiority in both simulation and real-world settings, coupled with minimal degradation under missing modalities, supports the claim that effective, real-world generalist robot agents require action backbones explicitly adaptive to both task and sensory context.

Reference:

"CoRE-VLA: Towards Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Experts" (2607.03693)