This&That: Language-Gesture Controlled Video Generation for Robot Planning

Abstract: Clear, interpretable instructions are invaluable when attempting any complex task. Good instructions help to clarify the task and even anticipate the steps needed to solve it. In this work, we propose a robot learning framework for communicating, planning, and executing a wide range of tasks, dubbed This&That. This&That solves general tasks by leveraging video generative models, which, through training on internet-scale data, contain rich physical and semantic context. In this work, we tackle three fundamental challenges in video-based planning: 1) unambiguous task communication with simple human instructions, 2) controllable video generation that respects user intent, and 3) translating visual plans into robot actions. This&That uses language-gesture conditioning to generate video predictions, as a succinct and unambiguous alternative to existing language-only methods, especially in complex and uncertain environments. These video predictions are then fed into a behavior cloning architecture dubbed Diffusion Video to Action (DiVA), which outperforms prior state-of-the-art behavior cloning and video-based planning methods by substantial margins.

• The paper introduces a novel framework that integrates language-gesture controls with video diffusion models, enhancing clarity and precision in robot planning.
• The paper employs a Transformer-based DiVA model that converts generated video plans into robotic actions, achieving state-of-the-art behavioral cloning.
• Experimental results on Bridge and IsaacGym show superior fidelity and user alignment compared to methods like AVDC and StreamingT2V in complex spatial tasks.

Language-Gesture Controlled Video Generation for Robot Planning

The research paper introduces the "This{content}That" framework, presenting a robot learning methodology that integrates language-gesture controls for enhanced video generation in robot planning. The framework leverages video generative models to address challenges in unambiguous task communication, controllable video generation, and the translation of visual plans into robotic actions. By combining language-gesture conditioning with behavioral cloning for robot execution, the paper claims state-of-the-art performance in planning tasks across complex environments.

The core of "This{content}That" is its innovative use of video diffusion models (VDM), particularly adapted from a large-scale text-to-video diffusion model (SVD) pre-trained on extensive internet data. The modifications involve a unique language-gesture conditioning approach, which surpasses the clarity and precision of language-only methods, especially within complex scenarios. This enhancement allows the generation of video sequences that align closely with human intent, using simple deictic expressions like "this" and "that."

The paper's experimental results, conducted on datasets such as Bridge and IsaacGym simulation, demonstrate the framework's effectiveness. The VDM, refined for robotics, showed superior fidelity and user alignment in video generation compared to existing methods, including AVDC, StreamingT2V, and DragAnything. Notably, the incorporation of gestures alongside natural language commands enables more precise interaction in tasks involving spatial complexity, such as "pick and place" or "stacking."

The proposed behavioral cloning model, DiVA, operates by referencing video frames generated by the VDM, incorporating them into a Transformer-based architecture. This model facilitates the seamless conversion of video plans into robotic actions. DiVA's adaptability was tested in synthetic environments, showcasing its robustness in handling out-of-distribution scenarios where language ambiguities are significant. Such advancements herald promising prospects for multi-task policy learning, underlining a significant contribution to the intersection of generative models and robotics.

The methodological contribution and empirical validation presented in this research hold substantial implications for AI's future trajectory. By refining interaction modes between humans and machines and enhancing task flexibility, "This{content}That" could significantly impact real-world robot planning and execution applications. Additionally, future exploration might focus on expanding this framework's capabilities to address long-duration tasks involving more intricate sequences of actions.

In conclusion, "This{content}That" presents a compelling advancement in robot learning via language-gesture conditioned video generation, offering profound insights and contributions that potentially drive forward the field of human-robot interaction.