VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding

Abstract: 3D visual grounding is crucial for robots, requiring integration of natural language and 3D scene understanding. Traditional methods depending on supervised learning with 3D point clouds are limited by scarce datasets. Recently zero-shot methods leveraging LLMs have been proposed to address the data issue. While effective, these methods only use object-centric information, limiting their ability to handle complex queries. In this work, we present VLM-Grounder, a novel framework using vision-LLMs (VLMs) for zero-shot 3D visual grounding based solely on 2D images. VLM-Grounder dynamically stitches image sequences, employs a grounding and feedback scheme to find the target object, and uses a multi-view ensemble projection to accurately estimate 3D bounding boxes. Experiments on ScanRefer and Nr3D datasets show VLM-Grounder outperforms previous zero-shot methods, achieving 51.6% [email protected] on ScanRefer and 48.0% Acc on Nr3D, without relying on 3D geometry or object priors. Codes are available at https://github.com/OpenRobotLab/VLM-Grounder .

• The paper introduces a VLM-based framework for zero-shot 3D visual grounding using only 2D images, bypassing the need for 3D point clouds and predefined vocabularies.
• It employs a dynamic stitching strategy and multi-view ensemble projection to accurately identify and localize objects in complex scenes.
• VLM-Grounder outperforms existing methods with [email protected] scores of 51.6% on ScanRefer and 48.0% on Nr3D, highlighting its enhanced performance.

Leveraging VLMs for Zero-Shot 3D Visual Grounding: The Introduction of VLM-Grounder

The quest to seamlessly integrate natural language with 3D scene understanding is a critical challenge for autonomous systems, particularly in robotics. The paper "VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding" presents a novel approach to this problem by introducing a vision-LLM (VLM) based framework to perform zero-shot 3D visual grounding exclusively using 2D images. This method diverges from traditional techniques which rely heavily on 3D point clouds and predefined vocabularies, an approach often constrained by the scarcity of comprehensive datasets.

The researchers propose a multi-faceted VLM-Grounder framework that dynamically analyzes sequences of images to locate target objects based on user queries. This is achieved through an innovative dynamic stitching strategy, which optimizes image context exposure while minimizing input data loss. By employing a grounding and feedback mechanism, VLM-Grounder leverages the reasoning capabilities of VLMs to ensure accurate target object identification, even incorporating automatic feedback for error correction. Following object identification, a multi-view ensemble projection module synthesizes various perspectives to estimate 3D bounding boxes, thereby overcoming limitations related to field of view and depth accuracy from a single image.

Numerical Results and Achievements

Significantly, VLM-Grounder outperforms existing zero-shot methods, achieving an [email protected] of 51.6% on the ScanRefer benchmark and 48.0% on the Nr3D benchmark. These results are notable given that traditional methods, such as ZS3DVG, reported an [email protected] of 36.4% and Nr3D accuracy of 39.0%, clearly indicating the enhanced performance of VLM-Grounder without the need for 3D geometry or object priors. Moreover, from a 2D perspective, VLM-Grounder achieves superior grounding accuracy against supervised and other zero-shot counterparts, bolstering its application in diverse scenarios, especially those involving complex object relationships.

Theoretical and Practical Implications

The introduction of VLM-Grounder contributes substantially to both theoretical and practical domains within AI and robotics. Theoretically, it pushes the boundary of what is achievable using VLMs in scene understanding by demonstrating that these models can perform complex 3D localization tasks using less information compared to traditional methods. Practically, this paves the way for more adaptable robotic vision systems capable of operating in unstructured environments without reliance on predefined object models.

Prospective Developments

Looking forward, the development of more powerful VLMs and enhanced dynamic stitching algorithms could further improve the accuracy and efficiency of VLM-Grounder. Additionally, refining the robustness of multi-view ensemble projection through advanced image matching techniques could make this approach viable for even more complex tasks, such as dynamic object interaction in real-time scenarios.

In summary, VLM-Grounder proposes a substantial shift from conventional 3D scene understanding methodologies by showcasing the power of leveraging 2D image sequences with advanced VLM frameworks for zero-shot 3D grounding. This work exemplifies a pivotal step towards more resource-efficient and flexible AI systems for real-world applications.