Transferring between sparse and dense matching via probabilistic reweighting

Abstract: Detector-based and detector-free matchers are only applicable within their respective sparsity ranges. To improve adaptability of existing matchers, this paper introduces a novel probabilistic reweighting method. Our method is applicable to Transformer-based matching networks and adapts them to different sparsity levels without altering network parameters. The reweighting approach adjusts attention weights and matching scores using detection probabilities of features. And we prove that the reweighted matching network is the asymptotic limit of detector-based matching network. Furthermore, we propose a sparse training and pruning pipeline for detector-free networks based on reweighting. Reweighted versions of SuperGlue, LightGlue, and LoFTR are implemented and evaluated across different levels of sparsity. Experiments show that the reweighting method improves pose accuracy of detector-based matchers on dense features. And the performance of reweighted sparse LoFTR is comparable to detector-based matchers, demonstrating good flexibility in balancing accuracy and computational complexity.