NVSPolicy: Adaptive Novel-View Synthesis for Generalizable Language-Conditioned Policy Learning

Abstract: Recent advances in deep generative models demonstrate unprecedented zero-shot generalization capabilities, offering great potential for robot manipulation in unstructured environments. Given a partial observation of a scene, deep generative models could generate the unseen regions and therefore provide more context, which enhances the capability of robots to generalize across unseen environments. However, due to the visual artifacts in generated images and inefficient integration of multi-modal features in policy learning, this direction remains an open challenge. We introduce NVSPolicy, a generalizable language-conditioned policy learning method that couples an adaptive novel-view synthesis module with a hierarchical policy network. Given an input image, NVSPolicy dynamically selects an informative viewpoint and synthesizes an adaptive novel-view image to enrich the visual context. To mitigate the impact of the imperfect synthesized images, we adopt a cycle-consistent VAE mechanism that disentangles the visual features into the semantic feature and the remaining feature. The two features are then fed into the hierarchical policy network respectively: the semantic feature informs the high-level meta-skill selection, and the remaining feature guides low-level action estimation. Moreover, we propose several practical mechanisms to make the proposed method efficient. Extensive experiments on CALVIN demonstrate the state-of-the-art performance of our method. Specifically, it achieves an average success rate of 90.4\% across all tasks, greatly outperforming the recent methods. Ablation studies confirm the significance of our adaptive novel-view synthesis paradigm. In addition, we evaluate NVSPolicy on a real-world robotic platform to demonstrate its practical applicability.

NVSPolicy: Adaptive Novel-View Synthesis and Hierarchical Policy Learning

The paper introduces NVSPolicy, an innovative language-conditioned policy learning methodology designed to improve robotic manipulation in unstructured environments. This research precisely addresses two significant challenges in robotic vision for manipulation tasks: the integration of multi-modal features resulting from deep generative models and the robustness of these features despite visual artifacts.

Context and Motivation

Recently, deep generative models have exhibited remarkable zero-shot generalization capabilities across various modalities, such as text and images. These advances hold potential for robotic tasks wherein robots must operate with partial scene observations. Specifically, the ability of generative models to synthesize unseen parts of a scene can provide robots with enhanced context for better decision-making. However, multimodal integration in these scenarios has been problematic due to visual imperfections in the synthesized view and the complex aggregation of visual and semantic features in policy networks.

NVSPolicy Methodology

NVSPolicy proposes a refined approach through the development of an adaptive novel-view synthesis module coupled with a hierarchical policy network. The framework leverages novel-view synthesis to dynamically curate additional visual context by selecting adaptive viewpoints that enhance the understanding of the robot's operational environment. This synthesized context is then processed by a cycle-consistent Variational Autoencoder (VAE) to separate the robust semantic information from less critical features. The resultant hierarchical policy network employs these disentangled features for informed high-level and low-level decision-making processes.

The proposed system comprises the following key components:

1. Adaptive Novel-View Synthesis: Utilizing GenWarp, a semantic-preserving generative model, NVSPolicy synthesizes images from strategically chosen viewpoints to maximize context while minimizing distortion. This is crucial for obtaining stable and informative visuals without unnecessary computational overhead.
2. Disentangled Feature Representation: The cycle-consistent VAE module refines the fidelity of the synthesized images by isolating semantic features necessary for accurate meta-skill selection from extraneous visual data. This separation ensures that the high-level skills determined by the policy network are robust and less prone to errors from visual anomalies.
3. Hierarchical Policy Network: NVSPolicy implements a hierarchical policy architecture where the semantic features guide meta-skill selection, and remaining features contribute to precise low-level actions. This separation aligns high-level strategy and lower-level execution, improving task success rates.
4. Practical Considerations for Efficiency: To enhance computational efficiency, two strategies are introduced—keyframe selection and policy distillation. Keyframe selection ensures novel-view synthesis is only performed when significant viewpoint changes occur, while policy distillation facilitates a lighter operational model by inferring semantic features directly, bypassing the need for constant synthesis.

Experimental Evaluation

NVSPolicy was rigorously evaluated against leading language-conditioned robotic manipulation algorithms on the CALVIN benchmark, demonstrating superior task success rates and proficiency in managing sequential tasks. The ablation studies further validated the efficacy of each proposed component, showing that the adaptive view synthesis and disentangled feature methodology significantly enhance the policy network's performance.

Moreover, NVSPolicy's robustness was demonstrated in real-world robotic applications, emphasizing the practical applicability of synthesized viewpoints and hierarchical strategy for manipulation tasks.

Implications and Future Work

NVSPolicy contributes substantially to the field of adaptive robotic manipulation by enhancing the integration of generative auxiliary input and structured policy networks. The technique of adaptive view synthesis exemplifies a pivotal avenue for generalizing robot action across diversified scenes and mandates further exploration into real-time efficiency and cross-domain adaptability. Future research directions could expand into broader multimodal integration, exploring deeper interactions between textual, visual, and temporal cues within robotic systems.

Overall, NVSPolicy represents a significant stride forward towards generalizable and robust autonomous manipulation, aligning state-of-the-art generative techniques with hierarchical learning paradigms for improved robotic autonomy.