RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalization

Abstract: Vision-Language-Action (VLA) models hold promise for generalist robotics but currently struggle with data scarcity, architectural inefficiencies, and the inability to generalize across different hardware platforms. We introduce RDT2, a robotic foundation model built upon a 7B parameter VLM designed to enable zero-shot deployment on novel embodiments for open-vocabulary tasks. To achieve this, we collected one of the largest open-source robotic datasets--over 10,000 hours of demonstrations in diverse families--using an enhanced, embodiment-agnostic Universal Manipulation Interface (UMI). Our approach employs a novel three-stage training recipe that aligns discrete linguistic knowledge with continuous control via Residual Vector Quantization (RVQ), flow-matching, and distillation for real-time inference. Consequently, RDT2 becomes one of the first models that simultaneously zero-shot generalizes to unseen objects, scenes, instructions, and even robotic platforms. Besides, it outperforms state-of-the-art baselines in dexterous, long-horizon, and dynamic downstream tasks like playing table tennis. See https://rdt-robotics.github.io/rdt2/ for more information.

• The paper presents a three-stage pipeline that integrates discrete action alignment, continuous diffusion synthesis, and distilled one-step inference to enhance scalability.
• The paper leverages a diverse UMI dataset from over 100 real-world environments, achieving up to ~47% success rates in unseen robotic setups.
• The paper shows that scaling data volume and model size predictably improves performance, enabling rapid zero-shot adaptation across dynamic tasks.

RDT2: Scaling Universal Manipulation Data for Zero-Shot Cross-Embodiment Generalization

Motivation and Challenges

The RDT2 framework responds to critical limitations facing Vision-Language-Action (VLA) models for robotics: data scarcity, architectural inefficiency, and the inability to generalize across hardware embodiments. The complexity of compositional generalization—across objects, scenes, instructions, and robot morphologies—imposes major challenges on real-world deployment. Existing VLA models demonstrate strong performance only within narrow domains and embodiments, requiring extensive retraining for new robots, which severely restricts scalability. Furthermore, slow inference and inadequate alignment between symbolic (VLM) and continuous (control) modalities limit the practical utility of large-scale multimodal models.

UMI Dataset and Hardware Advances

The cornerstone of RDT2’s generalization capabilities is its large-scale, embodiment-agnostic dataset, collected via a redesigned Universal Manipulation Interface (UMI). The authors enhanced the mechanical rigidity, tracking precision, and dexterity of the original UMI, deploying ~100 units across >100 real-world environments. This effort yielded 10,000+ hours of manipulation data covering a broad spectrum of manipulation primitives, scenes, and objects—emphasizing diversity, real-world ecological validity, and robustness to hardware heterogeneity. The engineering design explicitly facilitates transferability by minimizing physical gaps in vision and grasping between the handheld data collection and robotic deployments.

Model Architecture and Training Pipeline

RDT2 builds on a 7B-parameter vision-LLM (Qwen2.5-VL), integrating vision, language, and action representations into a unified latent space. The training pipeline comprises a three-stage process:

1. Stage 1: Discrete Action Alignment via RVQ. Residual Vector Quantization (RVQ) converts continuous action sequences into discrete tokens. The VLM backbone is trained to predict these tokens using cross-entropy loss, preserving the model's pretrained semantic knowledge and accelerating convergence relative to pure diffusion-based methods.
2. Stage 2: Continuous Action Synthesis with Flow-Matching Diffusion. The VLM backbone is frozen and a dedicated action expert (400M parameters) is trained to generate continuous actions via flow-matching loss, leveraging efficient Grouped Query Attention (GQA) and cross-attending to the backbone's representations. This hybrid framework combines the learning speed of discrete autoregressive models with the expressivity and multimodality of continuous diffusion.
3. Stage 3: Ultra-Fast Inference via Diffusion Distillation. For real-time applications, multi-step diffusion policies are distilled into single-step generators via on-the-fly regression against diffused trajectories, maintaining performance while reducing model latency substantially.

Experimental Results and Scaling Laws

Zero-Shot Generalization

RDT2 is explicitly evaluated under the “4U” protocol: unseen embodiment, scene, object, and instruction. Without any fine-tuning, models trained solely on human data and vision-language pairs achieve nontrivial success rates (up to ~47% in various tasks such as pick-and-place, wiping, shaking, and button pressing) when deployed on previously unencountered hardware platforms. The combinatorial generalization across all four factors is not achieved by prior VLA approaches (2602.03310).

Scaling Analysis

Systematic scaling studies demonstrate predictable improvements in model performance as both parameter count and data volume increase, confirming scaling laws for embodied models. Training loss decreases monotonically with more data and larger models, indicative of increased generalization capacity, consistent with findings in NLP and vision [Kaplan et al., 2020; Hoffmann et al., 2022].

Fine-Tuning Benchmarks

On dexterous, long-horizon, and dynamic tasks—including deformable object manipulation and table tennis—RDT2 fine-tuned variants outperform state-of-the-art models (To-FAST, To.5). For example, in cloth folding, unseen object success rates are ~4x baseline; in dynamic button pressing, reaction times are significantly reduced; and in table tennis, hit rates and robustness at varying speeds demonstrate effective real-time control. Notably, RDT2-UltraFast achieves both the fastest inference and highest accuracy among large models.

Ablation Studies

Ablation results validate core design decisions. RVQ enables aggressive token compression with minimal discretization error, outperforming alternative tokenization schemes. Hybrid AR-diffusion training provides faster convergence and better preservation of VLM knowledge compared to training diffusion policies from scratch. Latency benchmarks confirm that distilled one-step generators are required for highly dynamic control scenarios.

Practical and Theoretical Implications

RDT2’s embodiment-agnostic, data-scalable design marks a step toward universally generalizable, open-vocabulary robotic agents. Substantial reliance on unstructured, diverse human demonstration data in naturalistic environments bridges the gulf between laboratory benchmarks and real-world settings. Methodologically, the three-stage pipeline provides a template for integrating discrete and continuous modalities in large-scale embodied models, balancing semantic grounding and motor precision.

Practically, RDT2 enables rapid zero-shot deployment of robotic assistants in unstructured domains and across hardware, drastically reducing the adaptation cost and data acquisition burden. The demonstration of scaling laws in the embodied domain reinforces the imperative for massive, heterogeneous data collection efforts. However, the expansion of such datasets, especially from private homes, raises acute privacy and safety challenges regarding annotation, usage, and unforeseen real-world behaviors.

Theoretically, RDT2 suggests a path to compositional generalization in open-vocabulary embodied tasks, leveraging both pre-trained vision-language backbones and scalable, embodiment-agnostic data. The approach paves the way for large, general foundation models in robotics analogous to those in NLP, but with additional multimodal and control requirements.

Future Directions

Future work will likely focus on:

• Extending the scaling limit with even larger datasets and models;
• Automated data annotation and active learning to further reduce labeling costs;
• Safety-critical guardrails for real-world robotic deployment;
• Improved cross-modal fusion strategies for even tighter alignment between semantic reasoning and continuous action generation;
• Transfer learning and modularity to rapidly adapt foundation models to novel environments and hardware;
• Exploration of simulator-based synthetic data augmentation for rare or hazardous manipulation scenarios.

Conclusion

RDT2 systematically addresses major obstacles in generalist robotics by combining large-scale, embodiment-agnostic human demonstration data with an efficient pipeline for aligning vision-language semantics and continuous action. The results verify strong zero-shot transfer and state-of-the-art performance on challenging downstream tasks, substantiated by robust scaling laws. RDT2 establishes a new point of reference for foundation models in robotics and highlights the critical role of data, architecture, and inference efficiency in achieving compositional generalization across objects, instructions, scenes, and robot embodiments (2602.03310).