- The paper introduces HEFT, a teleoperation framework that robustly integrates privileged motion guidance and a windowed payload curriculum to manage VR tracking noise and heavy payloads.
- It employs an asymmetric actor-critic architecture with teacher-student policy distillation to bridge the sim-to-real gap and reduce long-horizon tracking drift.
- Experimental results show significant gains, achieving up to 90% success at 25 kg payload compared to 29% for existing baselines.
HEFT: Heavy-Payload Full-size Humanoid Teleoperation with Privileged Motion Guidance and Windowed Payload Curriculum
Introduction
The paper introduces HEFT, a teleoperation framework for full-size humanoid robots that robustly tracks task-level human references and manipulates heavy payloads using commodity VR trackers. This addresses two long-standing challenges: the high sensitivity of large humanoid platforms to tracking noise and drift induced by sparse, imperfect VR observations; and the complexity of controlling payload-coupled whole-body motions in dynamic, unstructured teleoperation scenarios. Compared to prevailing approaches, HEFT uniquely enables general whole-body teleoperation with two-hand loads up to 24 kg on a 175 cm, 65 kg, 29-DOF humanoid, L7.
Technical Contributions
HEFT's core innovations are twofold. First, Privileged Motion Guidance (PMG) leverages distributional knowledge from physically plausible offline reconstructions during training, but deploys policies on unmodified, noisy VR references. This architectural separation is realized in an asymmetric actor-critic framework, ensuring robustness to real-world VR artifacts without introducing deployment latency or sacrificing feedback responsiveness. Second, the Windowed Payload Curriculum (WPC) tackles the rapidly varying, motion-dependent nature of feasible payloads by partitioning teleoperation trajectories into short temporal windows, annotating each with expert-labeled payload caps derived from simulated rollouts. Payload schedules are then sampled adaptively within these caps during training, facilitating curriculum learning that generalizes to diverse and unforeseen whole-body motions.
Methodology
Privileged Motion Guidance
PMG is operationalized by pairing the deployable actor with raw VR reference windows, while the critic and reward computation operate against cleaned, reconstructed motion trajectories (using RoHM [15]). This set-up enables the actor to internalize the structure of noise and artifacts typical to VR tracking, while learning to align output with physically consistent intent. Notably, the reference windows include both past and (buffered) future targets, capturing local trajectory context and mitigating drift or latency effects at test time.
Windowed Payload Curriculum
For WPC, motions are windowed into 5-second segments, each evaluated by an offline expert policy conditioned on privileged simulation information. For each window, the expert attempts to execute the trajectory under incrementally decreasing payloads, labeling the maximum feasible load. During training, the sampled per-window payload (split randomly between wrists, directionally randomized within a gravity cone) is drawn from a uniform distribution up to the cap, modulated by curriculum progress. This prevents catastrophic failure during early learning and enables the policy to discover the differentiated effect of payload on various motion primitives (e.g., static support versus dynamic squatting).
RMA-Structured Policy Distillation
Training uses a teacher-student paradigm akin to Rapid Motor Adaptation (RMA) [40]. The teacher policy accesses comprehensive privileged state and serves as the source of a latent motor intent representation. The student policy, limited to deployable observations, learns to predict the latent online—bridging the sim-to-real gap and enabling low-latency deployment on hardware.
Experimental Evaluation
Simulation
On both the legacy G1 and the full-size L7 platforms, HEFT is benchmarked against strong baselines (SONIC [32], TWIST2 [10]) under both noise-free and noisy VR teleoperation references. PMG achieves the lowest long-horizon tracking drift on noisy VR inputs (0.544 m G1 / 0.560 m L7), a substantial improvement over mocap-only and generic noise augmentation methods. On standard motion datasets (Drandom), PMG policies match or exceed clean-reference policies in both success rate and MPJPE (0.021 m G1, 0.036 m L7), outperforming SONIC and TWIST2.
WPC delivers significant gains in high-payload robustness. At 25 kg payload, HEFT maintains 90% success versus 29% for the strongest baseline (TWIST2+FC), with only marginally increased MPJPE and velocity error. Even on high-dynamic, no-payload trajectories, the WPC policy improves success rates and pose accuracy over ablations using global or static curricula.
Hardware
HEFT's core policy is deployed unmodified on L7, robustly executing multi-stage, contact-rich object handling tasks under both symmetric and asymmetric loads (up to 24 kg). The policy demonstrates versatility across object pickups, squat motions, horizontal pushing, and coordinated walking, confirming HEFT's ability to unify whole-body teleoperation and high-payload manipulation in a single control primitive.
Theoretical and Practical Implications
HEFT validates that privileged training from reconstructed motion signals enables robust closed-loop deployment on full-size platforms without requiring structured noise models or online reference denoising, resolving a key bottleneck for widespread VR-based humanoid teleoperation. The windowed, motion-conditioned curriculum approach demonstrates that encapsulating motion-dependent payload feasibility as a first-class training signal expands the feasible operational envelope without resorting to task-specific policies or hard-coded controllers.
These advancements imply a paradigm shift for teleoperated, generalist humanoids: robust, deployable policies can now emerge from heterogeneous and imperfect demonstrations, scaling whole-body skills to real-world manipulation workloads. The approach further aligns with trends in reinforcement learning and sim2real transfer, as it leverages offline privileged information for policy shaping while preserving low-latency, sensor-aligned interfaces at runtime.
Limitations and Future Directions
HEFT's reliance on offline trajectory reconstruction and expert labeling entails non-trivial data preparation. Porting to new morphologies or tracker setups necessitates re-annotation and curriculum recalibration. The current force model treats payload as idealized wrist-applied wrenches, abstracting away object geometry, grasp stability, and support contact modeling. Generalizing HEFT across diverse hardware with different actuation and sensing remains future work.
Potential next steps include integrating online or incremental reference denoising, expanding force modeling to non-wrist contacts and complex objects, coupling with tactile and vision-based feedback for advanced manipulation, and validating curriculum generalization over broader humanoid platforms.
Conclusion
HEFT establishes a new state-of-the-art in full-size humanoid teleoperation under heavy payloads, architecting a scalable framework that merges privileged offline reconstruction with motion-conditional payload curricula. It bridges VR teleoperation noise and dynamic load feasibility constraints, supporting robust, versatile whole-body manipulation and locomotion on a single, deployment-ready policy. The methodology advances the design and deployment of generalist humanoids capable of human-scale task execution in the real world.
