ImLoc: Revisiting Visual Localization with Image-based Representation

Abstract: Existing visual localization methods are typically either 2D image-based, which are easy to build and maintain but limited in effective geometric reasoning, or 3D structure-based, which achieve high accuracy but require a centralized reconstruction and are difficult to update. In this work, we revisit visual localization with a 2D image-based representation and propose to augment each image with estimated depth maps to capture the geometric structure. Supported by the effective use of dense matchers, this representation is not only easy to build and maintain, but achieves highest accuracy in challenging conditions. With compact compression and a GPU-accelerated LO-RANSAC implementation, the whole pipeline is efficient in both storage and computation and allows for a flexible trade-off between accuracy and highest memory efficiency. Our method achieves a new state-of-the-art accuracy on various standard benchmarks and outperforms existing memory-efficient methods at comparable map sizes. Code will be available at https://github.com/cvg/Hierarchical-Localization.

• The paper introduces an image-based pipeline using dense depth maps, eliminating the need for global 3D models to achieve SOTA localization accuracy.
• It details a modular approach that integrates dense matching, retrieval, and PnP+RANSAC for efficient, robust pose estimation.
• Empirical studies show that aggressive image and depth compression yields minimal accuracy loss while significantly reducing storage requirements.

ImLoc: Revisiting Visual Localization with Image-based Representation

Introduction

ImLoc presents a principled re-examination of the visual localization pipeline, leveraging the advantages of 2D image-based scene representation augmented with dense, precomputed depth maps to unify the strengths of both 2D- and 3D-based localization paradigms. The method addresses the practical trade-off between geometric reasoning capacity and scalability/flexibility that traditionally divides 2D (image-based) and 3D (structure-based) localization techniques. ImLoc proposes that explicit, globally consistent 3D models are not necessary for SOTA localization accuracy, provided that each image is annotated with dense depth. The pipeline is designed for efficiency in storage, modularity in the use of retrieval/matching components, and robustness across diverse, real-world conditions.

Methodology

ImLoc formulates scene representation as a set of posed RGB images, each supplemented with a dense depth map and retrieval features. This architecture is depicted in the comprehensive pipeline schematic, encapsulating both the mapping and localization phases:

Figure 1: Pipeline overview: images, depth maps, poses, and retrieval features are stored; localization is performed via image retrieval, dense matching, 2D-3D correspondences, and PnP+RANSAC.

During mapping, depth maps are inferred for each input image using dense feature matching (specifically, RoMa [edstedt2023roma]), triangulating pixel correspondences across covisible images. These depth maps are compressed aggressively (8-bit quantization, JPEG XL), leveraging the finding that localization performance is insensitive to quantization beyond 8 bits and significant spatial downsampling. All inputs are efficiently stored, yielding memory footprints on par or below contemporary sparse 3D point clouds.

At localization time, global retrieval identifies the top K candidates (with options for state-of-the-art retrieval networks), and for each, dense 2D-2D matches between query and reference are established. These are converted into 2D-3D matches using the stored depth maps, enabling robust pose estimation using a GPU-accelerated LO-RANSAC. Importantly, the dense storage avoids the premature sparsification inherent in traditional pipelines, allowing selection of maximal, high-quality correspondences post-matching.

ImLoc is agnostic to the choice of matcher, supporting fully dense (RoMa), semi-dense (LoFTR), or sparse (SuperPoint, LightGlue), and can exploit the latest advances in dense correspondence and depth estimation.

Compression and Storage Analysis

Empirical studies on Cambridge Landmarks validate that storage for RGB and depth data—with modern compression and low-resolution setting—is not only comparable or superior to classic descriptor storage approaches, but also enables flexible trade-offs between localization accuracy and storage footprint.

Figure 2: Localization performance is robust to aggressive RGB image compression but is more sensitive to keyframe subsampling or drastic downsampling.

Figure 3: Depth quantization saturates at 8 bits; both quantization and resolution can be reduced with minimal accuracy cost.

Further, the Pareto frontier for image/depth compression settings demonstrates that moderate image downsampling/quality and low bit-depth for depth maps yield nearly optimal accuracy, with significant reductions in storage cost.

Benchmark Results

ImLoc decisively establishes new state-of-the-art results across major benchmarks, outperforming both established 3D structure-based pipelines (such as HLoc) and leading Scene Coordinate Regression (SCR) and hybrid approaches. Across LaMAR, Oxford Day and Night, Cambridge Landmarks, and Aachen Day-Night, ImLoc yields the highest recall at fine- and coarse-resolution thresholds, and remains robust under severe day-night appearance and environmental changes.

Figure 4: ImLoc achieves SOTA results on LaMAR, Aachen, Oxford, Cambridge datasets (left). Right: accuracy versus map size, showing superior accuracy-memory trade-off.

Critically, ImLoc's performance does not degrade significantly under aggressive compression, permitting practical deployments at drastically reduced storage cost (“nano” at $2$MB per scene), while SCR-based or heavily compressed 3DGS/mesh/NeRF approaches suffer severe accuracy loss.

Ablations and Analytical Results

Ablation studies systematically establish:

• Dense Match Superiority: Post-matching selection from dense matches (as opposed to matching only at keypoints) consistently yields higher localization accuracy.
• Compression Tolerance: JPEG XL at moderate to low quality, with downsampling to $280^2$-$560^2$ resolution, and 8-bit log-quantized depth, suffices with <1% loss in localization accuracy.
• Retrieval Robustness: SOTA retrieval (MegaLoc) is critical, but the pipeline’s gains are consistent across all retrieval settings given a sufficient number of references.
• Matching Flexibility: The architecture supports sparse, semi-dense, and dense matchers, always outperforming baselines when matched at the same feature density.

Failure Cases and Limitations

Failure cases predominantly arise from ambiguous global scene structure (e.g., visually identical locations, repetitive building structure), where retrieval itself is confounded, and current dense matchers cannot disambiguate. ImLoc, like all pose estimation pipelines, is limited by the accuracy of ground truth and cannot recover from errors in retrieval or strong scene symmetries.

Figure 5: Example of ground truth ambiguity—similar image content but different locations, as found in SfM-derived benchmark datasets.

Supplementary analyses reveal that pseudo ground truth from reconstruction may itself be mislabeled in ambiguous scenarios, confounding both evaluation and further learning.

Theoretical and Practical Implications

ImLoc demonstrates that the need for globally consistent, persistent 3D structure is eliminated when dense, local geometry (depth) is paired with high-quality image retrieval and dense correspondence models. This marks a significant simplification in both storage and pipeline complexity, obsoleting explicit 3D map maintenance for large-scale, dynamic, or mutable environments.

On a theoretical level, ImLoc’s results imply that scene geometry, for the purpose of localization, can be effectively represented by a collection of local, per-image reconstructions, entirely decomposing the parameterization problem and leaving global consistency as a non-essential constraint.

In practice, these findings impact deployment scenarios with memory-constrained devices, dynamic scene updates, or runtime demands for rapid adaptation (e.g., AR/VR, robotics in changing environments). Future developments are likely to further exploit data-driven per-image geometry and matching networks, potentially revisiting foundation models and feed-forward architectures for even more robust global disambiguation.

Conclusion

ImLoc redefines scene representation for visual localization, combining the flexibility and updatability of 2D image databases with the geometric reasoning of structure-based methods by leveraging precomputed dense depths. Extensive experiments confirm consistent SOTA performance, compactness, robustness under compression, and adaptability to changing scenes and retrieval techniques. The method obviates the need for explicit global 3D models for localization, marking a notable evolution in the design and analysis of visual localization systems.

(2601.04185)