Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments

Abstract: Vision-language-action (VLA) models across robot embodiments require high-quality observation--action supervision to learn deployable action distributions, yet scaling such robot data remains difficult, especially for high-DoF humanoids. Teleoperation provides controller-aligned supervision, while human egocentric videos capture diverse bimanual manipulation but do not directly provide executable robot actions. We introduce Human-as-Humanoid, a human-to-humanoid supervision framework that enables near-real-time human-centric action generation, making human demonstrations usable for high-DoF humanoid VLA training by jointly aligning the robot embodiment, the sensing setup, and the action-label interface. Built on PrimeU, a human-aligned 60-DoF upper-body humanoid, Human-as-Humanoid uses synchronized ego-exo videos to pair deployment-aligned egocentric observations with exocentric motion recovery, retargets the recovered human motion through staged Inverse Kinematics (IK) into controller-aligned 60-DoF action chunks, and trains the VLA model with Forward Kinematics (FK)-aware supervision to preserve wrist and fingertip task-space geometry. This converts large-scale human demonstrations from visual observations into executable observation--action supervision for the target humanoid. Experiments validate the conversion chain at the motion-recovery, robot-action-space, and real-robot deployment levels. Human-as-Humanoid yields a 4.8--7.2x raw demonstration-throughput gain over humanoid teleoperation in our data-collection analysis, and on several downstream tasks, policies post-trained only with the converted human labels generalize to real-robot deployment without target-task robot demonstrations. The official project website is available at https://zgc-embodyai.github.io/Human-as-Humanoid.

• The paper presents a novel pipeline that transforms synchronized egocentric human videos into robot-executable 60-DoF action vectors for zero-shot policy deployment.
• It details a four-stage methodology including human motion recovery, inverse kinematics retargeting, and DS-HKC supervision to ensure precise action alignment.
• Empirical evaluations demonstrate up to 7.2x throughput gains and accurate end-effector placements, validating effective zero-shot transfer on diverse manipulation tasks.

Human-as-Humanoid: Zero-Shot Humanoid Learning from Ego-Exo Human Videos

Motivation and Problem Formulation

High-DoF (Degrees of Freedom) humanoid robots hold significant promise for human-scale bimanual manipulation, but their training is fundamentally constrained by the necessity for high-quality, scalable observation-action supervision. Conventional teleoperation, while task-aligned, is slow, labor-intensive, and limited in diversity and throughput, especially for dexterous systems. In contrast, human demonstrations—particularly those captured via egocentric video—offer richness, diversity, and speed, but are not natively formatted as robot-executable trajectories due to fundamental mismatches in embodiment, sensing, and action interface.

The core challenge addressed by this work is to develop a pipeline that bridges these gaps, converting synchronized, camera-only ego-exo human demonstrations into executable, high-DoF robot actions at scale and near real-time rates. The resulting data unlocks new possibility for zero-shot deployment of humanoid VLA (Vision-Language-Action) policies—i.e., transfer to new manipulation tasks without the need for task-specific robot demonstrations.

System Architecture and Data Alignment

The proposed pipeline, Human-as-Humanoid, is architected around PrimeU, a custom-built, 60-DoF upper-body humanoid designed for anthropometric alignment with human demonstrators. PrimeU’s joint structure, scaling, and sensing layout (including head- and wrist-mounted RealSense cameras) substantially minimize the embodiment and sensor gap that has historically hampered human-to-robot imitation and transfer.

The data pipeline operates in four main stages:

1. Synchronized Ego-Exo Data Capture: Egocentric (policy-aligned) and exocentric (motion recovery) RGB videos are collected in synchrony. No motion-capture suits or markers are required, increasing practicality and throughput.
2. Robust Human Motion Recovery: Exocentric video provides stable evidence for the 3D reconstruction of upper-body and hand skeletons. The reconstruction is refined in root-relative coordinates and mapped to a kinematic convention matching PrimeU's structure.
3. Staged Inverse Kinematics Retargeting: The recovered human joint data is mapped into executable robot actions via part-wise IK, sequentially solving for hands, arms, neck, and waist, followed by guarded smoothing subject to robot kinematic limits.
4. Controller-Aligned Labeling: Outputs from the IK are formatted as 60-DoF action vectors directly usable in policy training and deployment, ensuring strict compatibility with PrimeU's control interface.

The result is a corpus of robot-centric action labels transformed from natural human activity, suitable for direct input into VLA model training.

PhysDex: VLA Learning from Human-Derived Action Labels

The policy learning stack, PhysDex, is constructed to leverage this high-DoF data efficiently. Visual and language tokens are produced via PhysBrain, an existing manipulation-aware VLM. Future robot actions are predicted with a conditional flow-matching diffusion transformer, with action supervision enforced both in joint space and, crucially, in task space through the Dual-Space Hierarchical Kinematic Constraint (DS-HKC). DS-HKC applies forward kinematics to enforce wrist and fingertip positional consistency, aligning model outputs with the requirements of contact-rich manipulation.

The action representation remains in full joint space (rather than lower-dimensional end-effector pose), maintaining the capacity to control all 60-DoF while incorporating geometric constraints to mitigate alignment drift and ensure physical feasibility.

Empirical Evaluation and Numerical Results

Multiple axes of evaluation demonstrate both the fidelity and practicality of the Human-as-Humanoid approach:

• Throughput: The pipeline achieves a 4.8–7.2x gain in demonstration collection throughput versus teleoperation. This is due to near-real-time conversion and the lower overhead of collecting human video versus robot manipulation data.
• Controller Alignment: A tokenizer trained exclusively on human-derived action chunks reconstructs held-out real-robot action windows with a mean normalized MAE of 0.008 and 95th percentile of 0.0097. Task-space replay via forward kinematics yields mean end-effector errors of approximately 5.3 mm, underscoring high cross-domain compatibility.
• Policy Performance: On diverse real-world manipulation tasks (e.g., ring placement, bag packing, water pouring), VLA policies pre-trained solely from human data labels successfully deploy zero-shot on the robot—demonstrating that the method enables target-task generalization without task-specific robot demonstrations. Composite task-stage scores regularly surpass those of GR00T N1.7, especially in the robot-data-scarce setting.
• Task-Space Consistency: FK-aware supervision (DS-HKC) significantly improves policy optimization geometry and reliability compared to joint-space-only losses, yielding lower loss curves and more accurate wrist/fingertip placements during execution.

Limitations

Despite the empirical strengths, several practical and theoretical constraints remain:

• Pose Estimation Bottleneck: The quality of human pose/hand reconstruction drives the upper bound on action label fidelity; systematic estimation errors can yield biased policy outcomes.
• Generalization Across Embodiments: The pipeline and labels are tailored to PrimeU; adaptation to new robot bodies requires fresh retargeting and action convention adjustment.
• Contact-Rich Dexterity: While kinematic transfer is robust, contact properties (forces, slip, friction) are less faithfully preserved, meaning fine-grained tasks still benefit from limited real-robot fine-tuning.
• Annotation and Camera Requirements: Current implementation relies on synchronized multi-view capture; future work is outlined to improve egocentric-only pose recovery under occlusions and motion blur.

Implications and Future Directions

The demonstrated pipeline establishes that high-throughput, controller-aligned human demonstrations can significantly reduce the robot-data requirements for scalable humanoid VLA learning. This paradigm shifts the burden of complex, labor-intensive robot teleoperation data collection toward more efficient human video collection, with robot data reserved for essential hardware grounding and contact-rich actuation refinement.

Theoretically, the method provides evidence that the longstanding embodiment gap in robot imitation learning can be systematically reduced by anthropometric robot design and tailored retargeting pipelines. Practically, it paves the way for large-scale training of dexterous policies in human-like manipulation contexts, and could accelerate the development of broadly competent generalist humanoids.

Future research will target the relaxation of sensing assumptions, improved robustness in pose recovery, and the generation of synthetic egocentric hand-object interaction sequences amenable to conversion by the Human-as-Humanoid process—thereby further scaling the diversity and coverage of available action supervision.

Conclusion

Human-as-Humanoid provides a robust framework for converting natural, camera-only human manipulation demonstrations into executable, joint-space-aligned robot actions suitable for high-DoF humanoid learning. The approach enables substantial data-efficiency improvements, strong cross-domain action compatibility, and practical zero-shot deployment on challenging manipulation tasks. While certain limitations remain, the results substantiate the use of human data not just for visual pretraining, but as genuine action supervision for future large-scale humanoid policy learning and deployment.