RwoR Approach: Human-to-Robot Data Generation

• RwoR Approach is a scalable pipeline that transforms wrist-mounted human hand videos into robot gripper demonstrations without physical robots.
• It employs a diffusion-based generative model and temporal cycle-consistency for precise alignment of hand and gripper data.
• Experimental evaluations demonstrate high visual fidelity and robust policy performance across diverse manipulation tasks, despite some instance-level generalization limits.

The RwoR approach ("Robot without Robot") is a data-generation and policy-learning pipeline in robotic manipulation that enables the collection and translation of natural human hand demonstrations into robot gripper demonstrations, entirely without deploying a physical robot during data acquisition. This method addresses the major scalability and distributional gap issues inherent in both teleoperated robot data collection and direct use of human hand demonstrations for imitation learning. The fundamental innovation of RwoR is a generative model that bridges the observation gap by converting wrist-mounted third-person hand videos into visually and kinematically aligned robot demonstration data suitable for end-to-end visuomotor policy training (Heng et al., 5 Jul 2025).

1. System Architecture and Data Collection

The RwoR pipeline is predicated on efficient, scalable data collection using a single GoPro Hero9 camera with a fisheye "Max" lens, rigidly attached to the human demonstrator's wrist. The specific design ensures consistent, wide-field imagery of hand-object interactions over a tabletop workspace.

Two types of demonstrations are recorded for model training:

• Human-Hand Runs: The demonstrator performs manipulation tasks naturally, yielding RGB wrist-camera video $\{h_1, \ldots, h_{T_1}\}$.
• Paired UMI Gripper Runs: Using the same scene and matched camera pose, the human controls a lightweight "UMI" parallel-jaw gripper, generating $\{r_1,\ldots,r_{T_2}\}$ and 6-DoF wrist camera poses derived from GoPro IMU and ORB-SLAM3.

Rigorous pose-matching between sessions is enforced to minimize scene and viewpoint discrepancies, facilitating reliable cross-domain alignment.

2. Data Alignment: Temporal Synchronization and Observational Matching

The practice of learning from hand-to-robot demonstration pairs imposes strict requirements on both temporal and observational alignment:

• Timestamp Synchronization: Self-supervised embeddings for human-hand and gripper video frames are learned via Temporal Cycle-Consistency (TCC). For each $h_t$, the corresponding $r_s$ is found as

$$s(t) = \arg\min_{s} \| \phi_H(h_t) - \phi_R(r_s) \|_2,$$

where $\phi_H$ and $\phi_R$ are the learned TCC embeddings. The paired, synchronized sequences are $\{(h_t, r_{s(t)})\}$ of length $T$.

• Observational Alignment: Despite synchronization, residual differences exist (lighting, trajectory mismatch, perspective). RwoR utilizes SAM2 for precise background segmentation ($B_t$ from $h_t$), and extracts gripper-plus-object foregrounds ($F_t$ from $r_{s(t)}$). "Inpaint Anything" fills background holes ($\hat{B}_t$), and compositing ($F_t \cup \hat{B}_t$) yields an aligned gripper image $\hat{r}_t$ over the human video's original background.

This composite forms the "ground-truth" output for generative training, ensuring domain-localized translation.

3. Hand-to-Gripper Generative Model

RwoR employs an image-to-image latent diffusion network, specifically an adaptation of InstructPix2Pix built on Stable Diffusion, to learn the transformation from hand images to robot gripper images:

• Model Components:
  • Encoder $E$ produces latents $z_0 = E(h_t)$
  • U-Net denoiser $\epsilon_\theta$ predicts noise given $(z_t, t, E(h_t), l)$ where $l$ is a prompt
  • Decoder $D$ reconstructs RGB images from denoised latents
• Text Prompts: Short instructions such as "Turn the hand into a gripper holding a ⟨obj⟩" are concatenated with image embeddings to focus generation on task-relevant object interactions.
• Loss Function:

$$\mathcal{L}_{\text{diff}} = \mathbb{E}_{h_t, \hat{r}_t, \epsilon \sim \mathcal{N}(0, 1), t} \| \epsilon - \epsilon_\theta(z_t, t, E(h_t), l) \|_2^2$$

With noise scheduling parameters $\alpha_t, \sigma_t$, $z_t = \alpha_t E(h_t) + \sigma_t \epsilon$.

Training minimizes $\mathcal{L}_{\text{diff}}$, mapping wrist-view hand frames to visually coherent gripper interaction images.

4. SE(3) Action Extraction and Policy Representation

Each hand video frame $h_t$ produces a GoPro-SLAM 6-DoF camera pose $C_t \in SE(3)$, which, via a fixed hand-to-fingertip transform $G$, yields the fingertip (proxy end-effector) pose $F_t = C_t \cdot G$. For policy learning, actions are parameterized as relative transformations $\Delta F_t = F_{t+1} \cdot F_t^{-1}$, given as translation $\Delta p_t \in \mathbb{R}^3$ and rotation $\Delta R_t \in SO(3)$. Gripper open/close status $g_t$ is inferred by thresholding the hand–object distance in $h_t$.

The final demonstration tuple per timestep thus comprises $(\hat{r}_t, \Delta F_t, g_t)$, enabling end-to-end imitation learning.

5. Algorithmic Pipeline Summary

The entire RwoR methodology is summarized in the following sequence:

1. Collect human-hand videos $H = \{h_t\}$ and paired gripper videos $R = \{r_s\}$ with SLAM poses.
2. Train TCC embeddings and generate temporally aligned pairs.
3. Segment and composite aligned image pairs for training data.
4. Train the diffusion generative model to map $(h_t, l) \rightarrow \hat{r}_t$.
5. For novel hand demonstration sequences, apply the generative model and pose transformations to render full robot demonstration tuples.
6. Train the visuomotor policy $\pi$ on the generated robot demonstrations according to the diffusion policy framework.

At each policy step, the input comprises a history of robot-view images $[\hat{r}_{t-k+1}, \ldots, \hat{r}_t]$, the current end-effector pose $F_t$, and binary gripper state $g_t$, with the network predicting $\Delta F_t$ and $g_{t+1}$.

6. Experimental Evaluation and Key Results

RwoR was validated via real-robot deployments on nine daily-life manipulation tasks using Franka+UMI hardware. Each task involved 50 human and 50 UMI gripper demonstrations as benchmarks. Performance was measured by success rate over 15 trials per task:

Method	Success Rate (mean)
UMI baseline	82%
RwoR	78%
Texture swap	37%

The generated demonstrations' visual quality was evaluated on 2000 held-out aligned pairs using PSNR (33.8 dB) and SSIM (0.86). Removal of observational alignment dropped PSNR and SSIM to 31.5 and 0.77, respectively.

Generalization experiments demonstrated robust transfer to unseen actions (rotate, unstack; 80–83% success) and minimal degradation with new objects. Qualitative assessment confirmed temporal coherence, correct gripper-object relationships, and consistent backgrounds.

7. Limitations and Directions for Future Research

Principal sources of error in RwoR include generative model failure modes (such as premature gripper closure) that degrade downstream policy performance. Out-of-distribution errors may arise when human demonstrations exceed the workspace or kinematic envelope of the target robot; thus, controlled collection protocols are required. The training dataset, comprising 200 paired demos and 60 object instances, constrains instance-level generalization, suggesting performance gains are likely with larger, more diverse pairing corpora.

The current method is tailored for parallel-jaw grippers. Extending RwoR to dexterous, multi-finger robotic hands is identified as an open area for future development.

In summary, RwoR demonstrates that scalable collection of wrist-mounted human hand videos, paired with diffusion-based hand-to-gripper translation, can generate high-fidelity, manipulation-ready robot demonstrations for imitation learning entirely without physical robots in the data collection loop (Heng et al., 5 Jul 2025).