SPIDER: Spatial Image CorresponDence Estimator for Robust Calibration

Abstract: Reliable image correspondences form the foundation of vision-based spatial perception, enabling recovery of 3D structure and camera poses. However, unconstrained feature matching across domains such as aerial, indoor, and outdoor scenes remains challenging due to large variations in appearance, scale and viewpoint. Feature matching has been conventionally formulated as a 2D-to-2D problem; however, recent 3D foundation models provides spatial feature matching properties based on two-view geometry. While powerful, we observe that these spatially coherent matches often concentrate on dominant planar regions, e.g., walls or ground surfaces, while being less sensitive to fine-grained geometric details, particularly under large viewpoint changes. To better understand these trade-offs, we first perform linear probe experiments to evaluate the performance of various vision foundation models for image matching. Building on these insights, we introduce SPIDER, a universal feature matching framework that integrates a shared feature extraction backbone with two specialized network heads for estimating both 2D-based and 3D-based correspondences from coarse to fine. Finally, we introduce an image-matching evaluation benchmark that focuses on unconstrained scenarios with large baselines. SPIDER significantly outperforms SoTA methods, demonstrating its strong ability as a universal image-matching method.

• The paper introduces a dual-head framework that combines geometry-aware descriptors with fine spatial details, enabling robust dense correspondence even under extreme view changes.
• It leverages multi-scale feature fusion via attention-driven Fusion Gates to effectively balance global geometric priors and local image details.
• Empirical evaluations show that SPIDER outperforms state-of-the-art methods on diverse benchmarks, reducing planar bias and enhancing match distribution diversity.

SPIDER: A Dual-Head Framework for Universal Dense Image Correspondence Estimation

Introduction and Motivation

Establishing accurate dense correspondences under large intra-domain appearance, scale, and viewpoint variations is foundational to spatial vision applications, notably in camera calibration, structure-from-motion (SfM), SLAM, and multiview geometry. Classical 2D feature-matching pipelines, while mature, suffer significant deficits when facing domains with limited texture, strong visual ambiguities, and large cross-view baselines—conditions often encountered in aerial, indoor, and ground-to-aerial scenes. Recent advances in pretrained vision transformers and 3D Vision Foundation Models (VFMs) have partially addressed these gaps. However, methods employing 2D features prioritize appearance similarity, leading to false matches in geometric outliers, while 3D-pretrained models exhibit strong geometric priors but tend to concentrate matches on dominant planes, lacking fine-grained sensitivity and suffering under wide baselines.

To reconcile the specificity-sensitivity dichotomy, "SPIDER: Spatial Image CorresponDence Estimator for Robust Calibration" (2511.17750) proposes an architecture integrating the geometric awareness of 3D VFMs with the fine spatial detail of 2D ConvNets. A dual-head matcher, operating in a coarse-to-fine manner, is introduced: one head regresses pixel-wise correspondences (warp head), while the other produces robust geometry-driven descriptors (descriptor head). This design yields a robust pipeline for wide-baseline, unconstrained image matching, evaluated on challenging established and novel benchmarks.

Methodology

Architecture Overview

Given image pair inputs, SPIDER leverages a 3D VFM encoder and a high-resolution ConvNet as a multi-scale backbone. The dual-head matching strategy is central:

• Descriptor Head: Aggregates multi-scale features with attention-driven Fusion Gates, yielding geometry-aware descriptors and confidence maps. These are especially crucial for rejecting semantically similar but geometrically inconsistent matches.
• Warp Head: Regresses dense correspondence fields at multiple scales, refining them hierarchically for high-sensitivity matching at the pixel level.

The final correspondences are sampled from the outputs of both heads, balancing sensitivity and specificity via a dual-sampling strategy based on confidence scores.

Figure 1: SPIDER pipeline: dual-head coarse-to-fine correspondence estimation leveraging 3D VFM and ConvNet multi-scale features.

Multi-Scale Feature Fusion

SPIDER's backbone first projects the image pairs into a geometry-rich coarse space using frozen weights from a 3D VFM (notably Aerial-MASt3R decoder), capturing cross-view geometric consistency. These coarse features are then progressively upsampled and fused via a ConvNet-based fine encoder (VGG), ensuring that local high-resolution spatial details are preserved. The Fusion Gate modules perform scale-adaptive mixing, controlled by learned attention maps, between successive coarse and fine feature scales. This disambiguates correspondences in complex scenes by maintaining both global and local geometric cues as shown by the significant reduction in planar matching bias.

Figure 2: Comparison of first-image descriptors from Aerial-MASt3R and SPIDER, demonstrating increased geometric detail and spatial diversity due to multi-scale refinement.

Dual Spatial Image Matcher Heads

The descriptor head uses a symmetric InfoNCE-based objective to enforce discriminability of the geometric embeddings, supervised by ground-truth correspondences. The warp head predicts dense 2D flows, using a transformer-based decoder and convolutional refinement, trained by discrete correspondence assignment at coarse scale and continuous loss at finer scales.

Fusion of the two head outputs is empirically optimized by balanced sampling, preventing dominance by either warp or descriptor correspondences—a clear improvement over naive pooling or ensemble strategies.

Experimental Evaluation

Benchmarks and Implementation

SPIDER is trained on a massively diverse corpus spanning 10 million+ image pairs across domains (including MegaDepth, ScanNet++, Aerial-MegaDepth, Habitat, and more). Training utilizes frozen ViT-Large/ViT-Base weights from Aerial-MASt3R for feature extraction, with only the output heads being finetuned, thereby avoiding overfitting and reducing training cost by over 80%.

Evaluation is reported on both in-domain (MegaDepth, ScanNet, Aerial-MegaDepth) and zero-shot (ZEB: 12 diverse datasets) settings, along with proposed extreme unconstrained multi-elevation scenarios and novel real/synthetic datasets (Urban, Warehouses, multi-campus) containing large scale and viewpoint variations, and visual ambiguities.

Numerical Results

SPIDER consistently achieves state-of-the-art mean AUC@5° on all established two-view geometry datasets and outperforms previous bests by sizable margins. In challenging unconstrained aerial-to-ground and ground-to-ground matching, SPIDER's average performance surpasses competing approaches by over 6-10% in the most difficult scenes. The method is particularly robust to planar bias and visual ambiguities, as evidenced by a lower planarity ratio (0.16 vs. 0.22 for Aerial-MASt3R) and higher diversity in match distribution.

Figure 3: Visualization of Urban scene geometry with camera positions showing the complexity and large viewpoint differences handled by SPIDER.

Qualitative visualizations reveal the dual-head synergy: while pattern-driven matchers (e.g., RoMa) recover spatially dense but ambiguity-prone correspondences, and geometry-only methods (e.g., Aerial-MASt3R) over-concentrate on dominant planes, SPIDER maintains high-confidence, spatially diverse, and geometrically consistent matches even in repetitive or ambiguous structures.

Figure 4: Visual comparison under unconstrained settings. SPIDER achieves spatially diverse yet geometrically accurate correspondences, avoiding the planar collapse and erroneous matches prevalent in single-head alternatives.

Ablation and Analysis

Ablation studies confirm that:

• Frozen backbones outperform end-to-end trainable variants in test generalization, with significantly lower parameter count and better AUC@5°.
• Multi-scale Fusion Gates with adaptive gating ($\alpha$) outperform standard FPN architectures and simple unweighted fusions.
• Match-level confidence ensemble, with balanced spatial sampling from both heads, achieves the highest robustness and accuracy.
• The cost is ~30% higher inference time than pure Aerial-MASt3R, but with consistently higher performance across the full range of tasks.

Implications and Future Directions

SPIDER's dual-head, multi-scale approach effectively bridges the trade-off between geometric specificity and visual pattern sensitivity, achieving SOTA robustness on classic and next-generation matching benchmarks—including extreme wide-baseline and ambiguous scenarios. The framework highlights the critical importance of fusing 3D geometric priors with high-resolution 2D local details—a principle likely to hold for future vision foundation models, particularly as tasks further diversify in complexity and domains.

The practical implications are broad, impacting robust camera calibration, visual relocalization, and all downstream tasks requiring dense, accurate correspondences in unconstrained or cross-domain settings. Theoretically, SPIDER paves a path for unified 2D-3D representation learning architectures, with promising follow-up avenues:

• Integration of more powerful ConvNet/Transformer fine encoders
• Task-driven or scene-conditional adaptive fusion of matching heads
• Extending match-head training for temporal sequences, not just pairwise estimation
• Reducing computational overhead or designing confidence aggregation priors for even better match set calibration

Conclusion

SPIDER unifies geometry-grounded and appearance-driven dense correspondence estimation in a dual-head, coarse-to-fine framework leveraging 3D VFM and multi-scale ConvNet features. Through systematic evaluation, SPIDER demonstrates state-of-the-art generalization and robustness across both classical and highly unconstrained matching regimes. This work represents a significant step towards universal image matching under arbitrary viewing conditions and sets a new high-water mark for robust, geometry-aware spatial perception (2511.17750).