Duet: Dual-Robot Understanding via Efficient Teaching

Abstract: Dual-robot collaboration enables tasks that exceed the reach and payload of a single robot, such as collaboratively transporting objects across environments and executing coordinated handovers. Data acquisition is the primary bottleneck for training these systems. To this end, we introduce DUET, a dual-robot learning framework for mobile manipulation. For efficient data collection, we create a unified dual-embodiment synchronized VR-based teleoperation system for in-domain heterogeneous robot data collection. We further develop a complementary tracking pipeline that records human-human coordination and collaborative mobile manipulation priors. To allow efficient learning, we introduce an Action Chunking Transformer based architecture that first pretrains collaborative policies on efficient human-human demonstrations, before finetuning them on a minimal set of real-robot teleoperation trajectories. We develop a benchmark of four collaborative tasks to evaluate our framework using a Unitree G1 humanoid and a Dexmate Vega1 mobile manipulator. The results demonstrate that harnessing human priors not only yields superior task performance compared to baselines trained only on robot data, but also reduces the total human effort required for data collection. Our human data collection pipeline achieves 5.4x acceleration on average from teleoperation, but we perform equally or better than robot-only data trained policies across all tasks. Our project page is available at https://zhaoy37.github.io/Duet/.

• The paper introduces DUET, which leverages synchronized dual-operator teleoperation and scalable human demonstrations to rapidly learn dual-robot manipulation policies.
• It employs an Action Chunking Transformer (ACT) to fuse visual and proprioceptive signals, achieving up to 70% success in complex, contact-rich tasks.
• The approach enables hardware-agnostic, scalable coordination with robust zero-shot generalization under diverse visual and dynamic perturbations.

Efficient Dual-Robot Policy Learning via Human-Driven Collective Training

Framework Overview

The paper presents DUET, a unified policy learning system for collaborative mobile manipulation with heterogeneous dual-robot teams. The framework capitalizes on efficient human demonstration pipelines as priors for rapid and robust dual-robot policy learning. DUET integrates a synchronized dual-operator VR teleoperation system for robot-domain data, a scalable human-human data collection process, and an Action Chunking Transformer (ACT) backbone to aggregate cross-domain visual and proprioceptive signals into temporally coherent joint actions.

Figure 1: DUET introduces a dual-robot policy learning paradigm using human demonstrations and dual-robot mimicry.

The methodological contributions include hardware-agnostic real-time teleoperation, scalable mesh-based human demonstration extraction, and a collective training regimen grounded in cross-embodiment pose projection. The policy backbone receives unified egocentric RGB streams and calibrated pose trajectories aligned into a shared coordinate space, with the ACT architecture imposing strong spatial-temporal coupling necessary for contact-rich mobile manipulation.

Dual-Robot Teleoperation and Human Demonstration Pipelines

DUET employs a synchronized dual-operator teleoperation interface using PICO VR, streaming retargeted motion and real-time egocentric video to both a Unitree G1 humanoid and a Dexmate Vega1 manipulator. Operators leverage platform-specific retargeting—SONIC for G1 and GMR for Vega1—paired with EMA smoothing to ensure physical reliability and stability at high actuation rates.

Figure 2: Example of real-time teleoperation translating human actions into coordinated dual-robot execution.

Complementing robot-side teleoperation, the human demonstration pipeline uses multi-view RGB-D capture and mesh extraction via SAM 3D Body. YOLO-based detection, ByteTrack identification, and depth-sensor calibration yield metric-aligned 3D anatomical pose trajectories closely mirroring robot embodiments. This pipeline achieves data acquisition acceleration by a factor of 5.4× on average relative to teleoperation, drastically reducing temporal and cognitive overhead.

Figure 3: Human data collection process demonstrating mesh extraction and pose calibration.

Collective Training Architecture

The ACT policy is pretrained on human data and subsequently finetuned on small-volume robot teleoperation data. The architecture partitions its output into shared pose and robot-specific joint-space predictions, applying modality-sensitive projection and regularization. Pose histories are root-centered and rotated for cross-modality consistency; images are normalized and ingested via a ResNet-18 encoder. Training follows a CVAE regularization regime for pose-only chunks, facilitating robust latent structure transfer from human to robot domain.

Figure 4: DUET's two-stage training leverages human demonstrations for pretraining and robot trajectories for minimal finetuning.

Benchmarking and Empirical Results

DUET is evaluated on four collaborative manipulation tasks: Trash Collection, Box Organization, Board Tilting, and Doll Passing. Scenarios require spatial-temporal synchronization, vision-driven handovers, and force-sensitive manipulation across dual robots. Human pretraining achieves either equal or superior performance to robot-only baselines, with success rates of up to 70% for complex tasks and significant reductions in collection effort.

Figure 5: Human demonstration offers substantial data collection efficiency gains across all benchmarked tasks.

Figure 6: Snapshots depict the physical setup and salient phases for each benchmark task.

Figure 7: Comparison of DUET policy performance to robot-only baselines highlights the benefits of human data pretraining.

The policy exhibits strong zero-shot generalization under visual and physical distribution shifts, such as altered background textures and changes to manipulated object types. Robustness is maintained despite drastic visual and dynamic variation.

Figure 8: DUET retains task efficacy under out-of-distribution visual and physical perturbations.

Theoretical and Practical Implications

DUET validates that direct multi-human priors substantially improve dual-robot coordination while minimizing robot data requirements. Scaling collaborative robot learning via efficient human demonstrations bypasses hardware restrictions, accelerates dataset generation, and produces policies with higher smoothness and task fidelity. The hardware-agnostic nature and modality-bridging pose space allow extension to arbitrary robot teams and embodied configurations.

The collective training regime for dual-robot learning directly addresses long-standing challenges in multi-agent policy scaling and coordinated manipulation. Incorporating large-scale human priors paves the way for pretraining universal visuomotor backbones that support rapid adaptation to diverse multi-robot systems, effectively enabling scalable mobile manipulation and contact-rich coordination in real-world environments.

Future Directions

The study identifies promising avenues for expanding DUET: adaptive exploration of backbone architectures beyond ACT, integration with larger robot teams, and generalizing to more complex tasks involving higher-dimensional coordination. Optimizing the representation and regularization of human priors could enable even more efficient transfer learning and scaling to broader classes of embodied agents. Probing policy generalization in broader OOD distributions and in-the-wild robot-free demonstration scenarios remains an open area for further investigation.

Conclusion

DUET introduces a dual-robot policy learning system unified by efficient human demonstration pretraining and rapid robot-domain finetuning. Empirical benchmarking confirms that human-driven priors are sufficient to match or surpass robot-only-trained baselines, with substantial reductions in data collection effort and direct improvements to motion smoothness and task reliability. The paradigm fundamentally advances scalable multi-robot learning and lays a foundation for leveraging universal human priors for collaborative policy design (2606.20990).