InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion

Abstract: We present InterHandGen, a novel framework that learns the generative prior of two-hand interaction. Sampling from our model yields plausible and diverse two-hand shapes in close interaction with or without an object. Our prior can be incorporated into any optimization or learning methods to reduce ambiguity in an ill-posed setup. Our key observation is that directly modeling the joint distribution of multiple instances imposes high learning complexity due to its combinatorial nature. Thus, we propose to decompose the modeling of joint distribution into the modeling of factored unconditional and conditional single instance distribution. In particular, we introduce a diffusion model that learns the single-hand distribution unconditional and conditional to another hand via conditioning dropout. For sampling, we combine anti-penetration and classifier-free guidance to enable plausible generation. Furthermore, we establish the rigorous evaluation protocol of two-hand synthesis, where our method significantly outperforms baseline generative models in terms of plausibility and diversity. We also demonstrate that our diffusion prior can boost the performance of two-hand reconstruction from monocular in-the-wild images, achieving new state-of-the-art accuracy.

• The paper presents a cascaded reverse diffusion framework that decomposes the complex joint distribution of two-hand interactions into simpler parts.
• It employs classifier-free and anti-penetration guidance to balance fidelity and diversity for generating plausible hand models.
• Benchmark metrics like FHID and KHID demonstrate superior performance, setting a new foundation for VR/AR applications and future research.

Leveraging Cascaded Reverse Diffusion for Two-Hand Interaction Generation

Introduction to Two-Hand Interaction Generation

Attempting to model the complex interactiveness of two hands, whether engaging with each other or with an object, represents a formidable challenge in the field of generative modeling. To date, efforts in reconstruing two-hand interactions have largely concentrated on reconstructing from monocular images. Generating these interactions procedurally, however, remains under-explored, a gap that this work, titled "InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion," aims to fill. At the core of this challenge is the inherent complexity imbued in modeling the combinatorial nature of hand articulations and interactions.

Breaking Down the Complexity

The key innovation in this paper is the proposed framework that simplifies the generative process by decomposing the joint distribution of two interacting hands into separate, more manageable distributions. This decomposition elegantly reduces the learning complexity and enables the generation of highly plausible hand shapes in close interaction contexts. Specifically, the authors leverage a diffusion-based method, incorporating anti-penetration and classifier-free guidance to enhance the generative fidelity and diversity.

Technical Contributions

The authors introduce several noteworthy technical innovations in their approach to two-hand interaction generation. Importantly, they acknowledge the symmetrical nature of hands and utilize this to learn a shared distribution model for both hands, applying conditioning dropout as a means to model both conditional and unconditional single-hand distributions. Additionally, the paper pioneers the use of cascaded reverse diffusion, where the sampling process first generates one hand and then conditions the generation of the second hand on the first. This cascaded approach is fortified with classifier-free guidance to balance between fidelity and diversity and further employs anti-penetration guidance to avert physically implausible intersections between the generated hand models.

Evaluation and Benchmarking

Given the absence of established benchmarks in the domain of two-hand synthesis, the authors construct a rigorous evaluation protocol to quantify the plausibility and diversity of generated hand interactions. This protocol includes metrics adapted from the generative modeling domain, such as Fréchet Hand Interaction Distance (FHID) and Kernel Hand Interaction Distance (KHID), alongside novel metrics specifically designed to quantify the physical plausibility of interaction. The resulting benchmarks not only demonstrate the superior performance of InterHandGen against existing baselines but also establish a methodological foundation for future research in this area.

Implications and Future Directions

The implications of InterHandGen span both practical applications and theoretical advancements in generative AI. From a practical standpoint, high-fidelity hand interaction models hold significant promise for enhancing user experiences in virtual and augmented reality environments. Theoretically, this work propels forward the understanding of complex interaction generation, shedding light on the potential of cascaded diffusion processes in tackling high-dimensional generative tasks.

Speculatively, extending the framework to incorporate dynamic interactions or applying its underlying principles to other domains of complex interaction, such as human-to-human or human-to-object interactions, could yield fascinating avenues for future research. Moreover, the versatility of the proposed model suggests potential in refining optimization or learning-based methods across a broad spectrum of applications in computer vision and beyond.

In conclusion, "InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion" presents a methodologically sound and technically innovative approach to generating plausible two-hand interactions. By decomposing the generative process and employing a cascaded diffusion strategy, this work lays a robust foundation for future explorations into complex interaction modeling and sets a new benchmark in two-hand interaction generation.