DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation

Abstract: Recent advances in video generative models enable the synthesis of realistic human-object interaction videos across a wide range of scenarios and object categories, including complex dexterous manipulations that are difficult to capture with motion capture systems. While the rich interaction knowledge embedded in these synthetic videos holds strong potential for motion planning in dexterous robotic manipulation, their limited physical fidelity and purely 2D nature make them difficult to use directly as imitation targets in physics-based character control. We present DeVI (Dexterous Video Imitation), a novel framework that leverages text-conditioned synthetic videos to enable physically plausible dexterous agent control for interacting with unseen target objects. To overcome the imprecision of generative 2D cues, we introduce a hybrid tracking reward that integrates 3D human tracking with robust 2D object tracking. Unlike methods relying on high-quality 3D kinematic demonstrations, DeVI requires only the generated video, enabling zero-shot generalization across diverse objects and interaction types. Extensive experiments demonstrate that DeVI outperforms existing approaches that imitate 3D human-object interaction demonstrations, particularly in modeling dexterous hand-object interactions. We further validate the effectiveness of DeVI in multi-object scenes and text-driven action diversity, showcasing the advantage of using video as an HOI-aware motion planner.

• The paper introduces a novel pipeline leveraging text-conditioned synthetic videos to generate physically-valid human-object interaction motion plans.
• It employs a hybrid imitation paradigm that combines 3D human pose recovery with robust 2D object tracking for precise visual alignment.
• Empirical results demonstrate significant error reductions (e.g., body MPJPE reduced to 29.73 mm) and enhanced generalization across diverse HOI scenarios.

Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation: An Expert Review

Introduction and Motivation

DeVI ("Dexterous Video Imitation") (2604.20841) systematically addresses key challenges in physically grounded human-object interaction (HOI) policy learning by introducing a novel pipeline that leverages text-conditioned synthetic videos as imitation references. The work is motivated by the prohibitive costs and limited generality of collecting high-quality 3D HOI demonstrations, particularly for dexterous hand-object interactions and multi-object scenarios. In contrast to prior methods relying on 3D mocap data or category-constrained demonstration libraries, DeVI explores the use of state-of-the-art video diffusion models to generate diverse, coherent HOI plans which are then mapped into actionable, physically-valid policy targets for simulated humanoids.

Technical Contributions

DeVI introduces a hybrid imitation paradigm where 3D human motions are reconstructed from synthetic videos with advanced mesh recovery tools, while the object trajectories are kept in 2D via robust video tracking. The method critically addresses the inherent imprecision and depth ambiguity of 2D generative video outputs by optimizing a visual HOI alignment procedure, combining geometric and visual loss terms to synchronize hand-object contacts and human posture with the rendered scene.

The DeVI pipeline operates in three principal stages:

1. Video-driven HOI Planning: Construction of text-conditioned, scene-specific synthetic HOI videos using a video diffusion model, starting from a realistic mesh-based initialization for both the human and object.
2. Hybrid Imitation Target Extraction: Extraction of 3D human pose trajectories with monocular mesh recovery and hand pose estimation, unified and refined to align with the proprioceptive scene, combined with robust 2D video tracking of visible object vertices.
3. Policy Learning via Hybrid Tracking Rewards: Training a physics-based humanoid controller using reinforcement learning, with a hybrid tracking reward that combines 3D human pose targets, 2D object point tracking, and contact-based constraints inferred from the video.

   Figure 1: Given a 3D physics environment and a text prompt, DeVI generates plausible human-object interaction motion using a video diffusion model as a motion planner.

   

   Figure 2: Overview showing scene initialization, mesh replacement, rendering, video generation, and hybrid target extraction for control policy training.

Key technical advances include:

• A reward formulation for RL that blends 3D body/hand tracking and 2D object tracking, circumventing the need for accurate 3D object pose recovery.
• Visual HOI alignment based on geometric projection, temporal consistency, and contact constraints for producing feasible, contact-consistent 3D human reconstructions from purely synthetic videos.
• Empirical validation in both single-object and multi-object scenes, demonstrating broad generalization beyond the restricted settings of typical 3D mocap datasets.

Empirical Results and Analysis

The authors conduct rigorous comparative experiments on the GRAB dataset and internet-derived object collections, benchmarking DeVI against leading methods: PhysHOI, SkillMimic, and InterMimic. The evaluation quantifies both joint accuracy (MPJPE, $T_\text{root}$), and object tracking metrics ($T_\text{obj}$, $O_\text{obj}$), with stringent success criteria.

Notable results include:

• Substantial reduction in MPJPE for both body and hand joints relative to all baselines. For example, DeVI achieves body MPJPE as low as $29.73$ mm compared to PhysHOI's $123.1$ mm (see table in main text).
• Superior 2D and 3D hand-object alignment, as quantified by contact precision and chamfer contact distance; visual HOI alignment proves critical in reducing errors by up to an order of magnitude for hand joints relative to state-of-the-art monocular mesh recovery.
• Robust handling of diverse HOI tasks: DeVI generalizes across a wide spectrum of object categories and interaction types, supporting both single- and multi-object scenes, as well as category-specific functional interactions.

  Figure 3: DeVI's video diffusion model supports a wide range of object categories and interaction types, from grasping to functional manipulation.

  

  Figure 4: DeVI's planning supports both target-awareness (planning interaction with specified objects) and fine-grained text-driven motion variation.

  

  Figure 5: Qualitative comparison showing DeVI outperforms baselines in tracking ground truth human and object motion using only 2D trajectories.

Quantitative ablation demonstrates the necessity of hybrid imitation rewards and the visual HOI alignment optimization; removal of either drastically reduces final policy accuracy and contact realism.

Implications and Future Directions

DeVI's hybrid tracking framework signals a paradigm shift in physical policy learning for dexterous agents, making feasible the imitation of complex, realistic HOI behaviors from widely available 2D generative data rather than costly 3D demonstrations. The fusion of robust 2D object tracking with 3D pose recovery acknowledges current limitations in monocular HOI estimation and exploits the geometric strengths of each modality for reward design. The RL-based approach closes the sim-to-real gap in synthetic-to-physical transfer, enabling learning policies that satisfy both the kinematic and physical constraints of manipulators.

On the theoretical front, DeVI demonstrates that reward formulations combining partial 3D and 2D objectives can induce policies that both mimic high-level task intent and produce feasible physical motions, without over-constraining the solution space or requiring dense, precise supervision. The employment of diffusion models as HOI motion planners also establishes a new modular interface between generative vision models and embodied policy learning.

Figure 6: The network architecture combines a transformer-based actor and MLP-based critic for high-dimensional humanoid control.

In practice, DeVI opens avenues for scalable simulation of dexterous manipulation skills in settings where 3D ground truth is unattainable, with immediate applicability in robotics, embodied AI, and virtual agents. As diffusion and multi-view video models improve, richer and more physically accurate imitation targets may be extracted, reducing the impact of perspective artifacts and monocular ambiguity (as discussed in the paper's limitations section). Automatic contact estimation and affordance learning remain open challenges, with future work likely to integrate multimodal perception and language grounding to further enhance text-controllable HOI policy learning.

Figure 7: Demonstration that DeVI and its hybrid rewards are applicable to non-tabletop, full-body manipulation scenarios.

Conclusion

DeVI (2604.20841) proposes a comprehensive solution for physics-based, dexterous human-object interaction policy learning by capitalizing on synthetic video diffusion models as motion planners and extracting hybrid 3D-2D rewards for reinforcement learning. The framework achieves state-of-the-art results on both qualitative dexterity and quantitative imitation benchmarks, with strong generalization to unseen objects and multi-object situations. The method substantially relaxes the requirement for high-quality 3D demonstrations, paving the way for scalable, vision-driven policy learning for complex HOI in simulation and, prospectively, robotics.