VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models

Abstract: Text-to-point-cloud (T2P) localization aims to infer precise spatial positions within 3D point cloud maps from natural language descriptions, reflecting how humans perceive and communicate spatial layouts through language. However, existing methods largely rely on shallow text-point cloud correspondence without effective spatial reasoning, limiting their accuracy in complex environments. To address this limitation, we propose VLM-Loc, a framework that leverages the spatial reasoning capability of large vision-LLMs (VLMs) for T2P localization. Specifically, we transform point clouds into bird's-eye-view (BEV) images and scene graphs that jointly encode geometric and semantic context, providing structured inputs for the VLM to learn cross-modal representations bridging linguistic and spatial semantics. On top of these representations, we introduce a partial node assignment mechanism that explicitly associates textual cues with scene graph nodes, enabling interpretable spatial reasoning for accurate localization. To facilitate systematic evaluation across diverse scenes, we present CityLoc, a benchmark built from multi-source point clouds for fine-grained T2P localization. Experiments on CityLoc demonstrate VLM-Loc achieves superior accuracy and robustness compared to state-of-the-art methods. Our code, model, and dataset are available at \href{https://github.com/MCG-NKU/nku-3d-vision}{repository}.

• The paper introduces a novel VLM-centric paradigm that transforms 3D point clouds into BEV images and scene graphs for precise text-guided localization.
• It achieves over 14% improvement in Recall@5m on the CityLoc benchmark by leveraging partial node assignment for robust multimodal reasoning.
• The study demonstrates strong cross-domain generalization from LiDAR and photogrammetric data, paving the way for scalable, interpretable urban localization solutions.

Vision-LLMs for Text-to-Point-Cloud Localization: An Expert Review of VLM-Loc

Introduction and Problem Statement

The paper "VLM-Loc: Localization in Point Cloud Maps via Vision-LLMs" (2603.09826) addresses fine-grained text-to-point-cloud (T2P) localization, which seeks to estimate a 2-DoF location in a 3D point cloud map using natural language spatial descriptions. Unlike traditional T2P approaches, which primarily rely on direct correspondence between shallow text and geometric data and are limited by explicit spatial modeling and small-scale testbeds (e.g., KITTI360Pose), this work introduces a novel VLM-centric paradigm. VLM-Loc leverages large-scale vision-LLMs (VLMs) to provide explicit multimodal spatial reasoning using both 2D and structured semantic representations derived from urban-scale point cloud data.

Figure 1: (a) Human-like spatial reasoning for text-to-point-cloud localization; (b) architecture of a typical prior approach (Text2Loc); (c) the proposed VLM-Loc architecture.

Methodology

The VLM-Loc framework introduces a pipeline that transforms 3D point clouds into bird’s-eye-view (BEV) images and scene graphs—bridging the modality gap between geometric data and language for VLMs. The BEV images encapsulate dense geometric and semantic context, while scene graphs encode explicit object relationships and spatial positions. This structured input is then processed by a VLM (Qwen3-VL-8B-Instruct as default backbone) via parameter-efficient LoRA tuning, enabling the model to perform partial node assignment (PNA) and autoregressive spatial localization using natural language queries.

Figure 2: VLM-Loc framework overview: (left) map conversion to BEV and scene graph; (right) input flow for VLM fine-tuning and localization.

Partial Node Assignment (PNA) Mechanism

The PNA module provides a critical component, aligning language-entity cues with visible nodes in the scene graph by thresholding centroid distances (object and "stuff" classes with separate, semantically adaptive thresholds). This mechanism ensures robust reasoning and interpretable grounding, even when some referenced objects are partially or entirely unobservable in the map region—a key challenge absent from previous studies focused on oversimplified submaps.

Figure 3: Node assignment logic in PNA: a textual reference is "groundable" only if its centroid distance to the mapped object is below semantically set thresholds.

Localization proceeds via autoregressive decoding, outputting (1) text-object-to-node assignments (in strict JSON) and (2) the 2D predicted BEV position.

The CityLoc Benchmark

To evaluate the scalability and generalization limits of T2P localization models, the authors introduce the CityLoc benchmark, constructed from multi-source point clouds: CityLoc-K (KITTI-360 LiDAR) and CityLoc-C (SensatUrban/CityRefer photogrammetric data from UAVs). This dual-source design allows systematic testing of models under both domain-congruent and domain-shift scenarios, with substantially larger and more complex maps than those permitted by KITTI360Pose.

Figure 4: Example point clouds from CityLoc: (a) KITTI-360 road LiDAR; (b) photogrammetric urban blocks from CityRefer.

Figure 5: Semantic instance composition of CityLoc-K.

Figure 6: Semantic instance distribution in CityLoc-C.

Experimental Results

Overall Performance

VLM-Loc achieves dominant performance across all recall thresholds on CityLoc-K, with Recall@5m improvements exceeding 14% over the prior SOTA, CMMLoc. Metrics across Recall@5,10,15m on both validation and test splits are consistently higher, affirming both improved grounding accuracy and structured spatial reasoning.

Figure 7: Strong negative correlation between localization error and the number of correctly assigned nodes—demonstrating the effectiveness of explicit node grounding.

The robust alignment of language with object nodes is shown to be critical, as qualitative and statistical results indicate that error distributions narrow and median errors drop as the number of correctly grounded cues increases.

Qualitative Analysis

Qualitative comparisons reveal VLM-Loc's superior ability to interpret complex queries and resolve ambiguous or incomplete spatial language in rich urban maps, consistently estimating positions more accurately than Text2Loc, MNCL, and CMMLoc.

Figure 8: Predicted vs. ground-truth positions for VLM-Loc and baselines on CityLoc-K using semantic label overlays.

Figure 9: Additional scene-level qualitative comparisons on CityLoc-K.

Figure 10: Qualitative results for cross-domain transfer to CityLoc-C.

Ablations

Systematic component-wise and query-attribute ablation shows:

• The full model (BEV+scene graph+PNA) outperforms all reduced configurations;
• Scene graph input alone surpasses BEV-only configurations, but adding dense BEV yields further gains, showing the complementarity of geometric and relational representations;
• Directional cues in queries are the most essential for disambiguation, while color cues yield further incremental improvements;
• Partial node assignment (PNA) outperforms forced (full) assignment by >18% at Recall@5m, confirming the necessity of handling partial observability;
• Larger backbone VLMs (Qwen3-VL-32B) offer moderate additional gains, but even 8B/4B/2B backbones are highly competitive, ensuring scalability across compute budgets.

Cross-Domain Generalization

On CityLoc-C (cross-modality, cross-region), VLM-Loc achieves up to 2× higher Recall@5m than the best baselines, indicating strong cross-domain robustness and adaptability to unseen semantic and geometric distributions.

Implications and Future Directions

The introduction of VLM-Loc establishes a new paradigm for language-driven spatial reasoning in complex, real-world 3D scenes, revealing that (1) explicit alignment between language and structured scene representations is essential for interpretable, reliable localization; (2) large-scale VLMs, even when parameter-efficiently tuned on derived 2D projections and graph structures, exhibit strong multi-modal generalization; (3) future datasets and benchmarks must eschew oversimplified, small submaps for scalable, richly semantic challenges that reflect real-world deployment conditions.

On the theoretical side, the explicit PNA mechanism contributes to the field’s understanding of partial grounding and modular spatial alignment, suggesting a pathway for fine-grained, compositional mapping between open-vocabulary descriptions and multi-object geometric contexts.

Practically, the demonstrated robustness of VLM-Loc under domain shifts (LiDAR ↔ photogrammetry) argues for its deployment in multimodal embodied agents, including those requiring language-based assistance for navigation and localization in unstructured or visually inaccessible environments (e.g., urban GNSS dropout, collaborative robotics, autonomous vehicles with limited perception).

Potential future lines of research include extending the reasoning capacity to handle hierarchical, nested, and multi-turn dialogues, and integrating active querying and navigation modules to enable interactive, closed-loop spatial reasoning, as well as further compressing the VLM-Loc stack for low-latency edge deployment.

Conclusion

VLM-Loc delivers a substantial advance in T2P localization, unifying vision-LLMs with explicit spatial and semantic grounding for scalable, interpretable position estimation in complex scenes. Its architectural design, benchmark innovation, and experimental rigor collectively redefine the state of the art for language-based 3D spatial understanding, and set a trajectory for subsequent work in embodied multimodal localization and planning (2603.09826).