Where Am I? Semantic Map Grounding via Vision-Language Models for Multi-Modal Localization

Abstract: We address robot localization in GPS-denied indoor environments by reframing it as a semantic reasoning task rather than a geometric estimation problem. Motivated by how humans localize using object-level cues and labeled maps, we ask whether a vision-LLM, given a front camera image, a polar LiDAR scan, and a top-down semantic grid map, can infer the robot pose. We fine-tune Qwen2.5-VL-7B with LoRA and attach a lightweight regression head that predicts continuous pose coordinates (x, y, theta) directly from the final hidden state, bypassing text generation. Training uses a composite position-and-direction loss with curriculum learning on a custom Gazebo dataset of 120,112 samples and 527 scenes. On the in-distribution test set of 18,017 samples, the model achieves 98.23 percent position accuracy, 98.00 percent direction accuracy, 96.75 percent full pose accuracy, a mean position error of 0.11 m, and a mean orientation error of 5.7 degrees at 0.62 s per sample. Position accuracy drops by only 7.2 percentage points on seven unseen object categories, reaching 90.99 percent, supporting semantic spatial reasoning rather than appearance memorization. With incomplete maps, fine-tuning recovers performance to 93.72 percent position accuracy, showing adaptability to stale or partial map information. Two ablations highlight cross-modal complementarity. Without LiDAR, using only camera and map inputs, position accuracy remains 95.06 percent, only 3.2 percentage points below the full system. However, when the camera sees no visible objects in a wall-facing view, LiDAR sustains 92.33 percent position accuracy, compared with 70.74 percent when neither LiDAR nor visible objects are available. This shows that LiDAR becomes the primary localization signal when camera semantics are unavailable and provides a reliable fallback under occlusion or sparse layouts.

• The paper presents a multi-modal localization model that integrates camera, LiDAR, and semantic grid maps to regress pose coordinates in a single shot.
• It leverages a fine-tuned Qwen2.5-VL-7B backbone with LoRA, achieving high accuracy with 98.23% position accuracy and a low mean position error of 0.11m.
• The approach demonstrates robust semantic reasoning, adapting to incomplete maps, novel objects, and challenging visual conditions in GPS-denied environments.

Semantic Map Grounding via Vision-LLMs for Multi-Modal Robot Localization

Problem Formulation and Motivation

This work reframes the robot localization challenge in GPS-denied indoor environments as a semantic reasoning problem, leveraging multi-modal sensory data and a semantic grid map as symbolic prior. Unlike classical SLAM and geometric localization frameworks that rely on temporal consistency, dense maps, and metric cues, the proposed method is inspired by human deliberative spatial reasoning—integrating object-level semantics from camera, metric geometry from LiDAR, and structural priors from a top-down semantic map to infer a robot's pose in a single shot. The approach sidesteps text generative paradigms typical in VLMs, focusing instead on direct regression of pose coordinates.

System Architecture

The core model utilizes the Qwen2.5-VL-7B vision-language backbone, fine-tuned via LoRA for efficient adaptation. The input pipeline concatenates three modalities—an overhead semantic grid map, a front-facing RGB camera image, and a bird's-eye-view polar LiDAR scan—to construct a multi-image prompt.

After tokenization and vision encoding, the pooled non-padding hidden state is routed through a lightweight PoseHead regression module consisting of LayerNorm, GELU-activated linear projection, dropout, and a final linear map to four outputs representing normalized position $(x, y)$ and direction cosines $(\cos\theta, \sin\theta)$. This bypasses language modeling and enables direct continuous pose estimation.

Figure 1: System pipeline—spatial observations from a semantic map, camera, and LiDAR are concatenated and processed by a fine-tuned VLM, with pose decoded through a regression head.

The multi-modal fusion depicted in the pipeline enables the model to cross-reference the robot’s current sensory context with persistent map-level priors. The semantic grid map conveys symbolic structure, while the camera and LiDAR provide metric and semantic evidence.

Dataset and Evaluation Protocol

A custom Gazebo 3D simulation corpus is constructed: 527 unique scenes (6×6 m) are annotated with 8–10 objects from 10 training categories, discretized into $8 \times 8$ grid cells. For each cell and heading (8 per cell), corresponding camera and LiDAR images are captured, yielding over 120k multi-modal training examples. The evaluation regimes are carefully structured to probe not only in-distribution generalization but also transfer to unseen object categories, partial/stale maps, occluded views, and domain shifts in appearance.

The main semantic map input is shown below:

Figure 2: Semantic grid map encoding object layout, with blue map cells denoting labeled categories—this serves as the symbolic prior for map grounding.

Performance is quantified using Position Accuracy (PA), Direction Accuracy (DA; within 45°), Full Pose Accuracy (FPA), Mean Position Error (MPE in meters), and Mean Orientation Error (MOE in degrees).

Training Methodology

Training exploits LoRA with full-weight freezing except bottleneck adaptation matrices. A composite loss function emphasizes both position MSE and direction cosine loss, with additional L2 regularization penalizing boundary drift under ambiguous contexts. The training curriculum advances through:

1. Baseline fitting on all modalities with a complete semantic map.
2. Exposure to map incompleteness and unseen category objects, to force reasoning from partial data.
3. Camera + map only finetuning and explicit LiDAR ablations, isolating the contribution of each modality.

Figure 3: Training and validation loss curves indicate stable convergence and negligible overfit under full-modality training.

Experimental Results

After LoRA fine-tuning, the full model achieves 98.23% PA, 96.75% FPA, 0.11 m MPE, and 5.7° MOE on in-distribution samples. Zero-shot prompting yields near-chance (<5%) accuracy, highlighting the necessity of targeted adaptation to the structured pose regression task.

Challenging ablations and regime shifts reveal several key findings:

• Semantic generalization: On 7 novel object categories, PA drops by only 7.2% to 90.99%. This demonstrates robust relational reasoning—semantic associations rather than mere appearance memorization.
• Robustness to incomplete maps: With 50% of map entries missing, PA falls to 72.81%, but can be fine-tuned back to 93.72%, confirming adaptability to stale/partial spatial priors.
• Modality complementarity: Wall-facing (object-free) views with LiDAR intact obtain 92.33% PA. Removing both objects and LiDAR degrades PA to 70.74%, quantifying modality redundancy and fallback behavior.
• Camera+map sufficiency: With no LiDAR, PA remains high (95.06%), but only when visual object cues are present—the reliance on semantics emerges from explicit ablations.
• Domain shift: Under strong synthetic appearance alteration (NanoBanana-generated environments), PA to the correct grid cell drops sharply (25%), though relaxing tolerance to two cells recovers 100% accuracy. Direction accuracy remains high (87.5%), indicating that heading estimation is robust to visual shift while fine-grained position is strongly affected.

Figure 4: Domain adaptation performance on held geometry with extensive visual appearance shift—cell-level localization accuracy is impaired, but heading estimation and coarse pose tolerate visual perturbations.

Implications and Directions

This work substantiates the claim that transformer-based VLMs, when fine-tuned, can perform complex spatial reasoning over multi-modal observations and symbolic map priors, enabling reliable single-shot pose estimation in structured indoor environments. The minimal position error matches or exceeds classical geometric methods under comparable conditions, and the architecture is robust to semantic novelty, partial map priors, and challenging occlusion scenarios. The system critically demonstrates that LiDAR data is not redundant; instead, it provides essential fallback information, especially for ambivalent or visually sparse viewpoints.

From a theoretical standpoint, these results strengthen the case for deliberative multi-modal VLMs as drop-in semantic backends for robotics. Their ability to cross-reference current sensory cues with persistent symbolic representations, and generalize via relational rather than pixel-level features, marks a new regime in semantic map grounding and spatial AI.

Practical deployment will require domain adaptation to real-world shifts, dynamic scenes, and handling continuous (non-discretized) pose spaces. The approach is inherently extensible to richer map constructs (scene graphs, 3D) and online learning in dynamic layouts. Future directions include sim-to-real transfer and integrating incremental map updates with localization.

Conclusion

The approach systematically demonstrates that a fine-tuned VLM with a multi-modal input pipeline and direct pose regression can achieve near-human-level indoor localization accuracy, even in the face of map incompleteness, novel semantic entries, and adverse visibility. The paradigm expands the scope of semantic reasoning in robotics, providing a blueprint for integrating deliberative, language-inspired models into perception and decision pipelines for autonomous agents. The model’s robust cross-modal generalization, symbolic relational inference, and adaptability offer substantial promise for semantic localization beyond the limits of conventional geometric SLAM frameworks.