SOSNet: Second Order Similarity Regularization for Local Descriptor Learning

Abstract: Despite the fact that Second Order Similarity (SOS) has been used with significant success in tasks such as graph matching and clustering, it has not been exploited for learning local descriptors. In this work, we explore the potential of SOS in the field of descriptor learning by building upon the intuition that a positive pair of matching points should exhibit similar distances with respect to other points in the embedding space. Thus, we propose a novel regularization term, named Second Order Similarity Regularization (SOSR), that follows this principle. By incorporating SOSR into training, our learned descriptor achieves state-of-the-art performance on several challenging benchmarks containing distinct tasks ranging from local patch retrieval to structure from motion. Furthermore, by designing a von Mises-Fischer distribution based evaluation method, we link the utilization of the descriptor space to the matching performance, thus demonstrating the effectiveness of our proposed SOSR. Extensive experimental results, empirical evidence, and in-depth analysis are provided, indicating that SOSR can significantly boost the matching performance of the learned descriptor.

• The paper introduces Second Order Similarity Regularization (SOSR) to capture deeper structural relationships in local descriptor learning.
• It integrates SOSR with a modified Quadratic Hinge Triplet loss to adaptively weight errors and improve training efficiency.
• Empirical results show SOSNet significantly reduces false positives and enables more robust and accurate 3D reconstructions.

Overview of SOSNet: Second Order Similarity Regularization for Local Descriptor Learning

The paper introduces a novel approach called Second Order Similarity Network (SOSNet) that enhances local descriptor learning by incorporating Second Order Similarity Regularization (SOSR). The authors address a gap in the domain of local descriptor learning, where traditionally, First Order Similarity (FOS) via Euclidean distances has been the primary focus. The innovative aspect of SOSR lies in its ability to leverage second order similarities, which better capture structural information and relationships in the data, thus offering improvements over existing methods.

Theoretical and Methodological Contributions

SOSNet proposes a new regularization technique for training local descriptors used in computer vision tasks such as 3D reconstruction, image localization, and patch retrieval. The key idea is to harness Second Order Similarity, positing that a positive pair of matching points should exhibit analogous distances with respect to other points in the embedding space. This consideration leads to the development of the SOSR term that integrates seamlessly into the training process, requiring minimal computational overheads compared to traditional approaches utilized in graph matching and clustering that necessitate significant resources.

The training approach combines SOSR with a modified triplet loss known as Quadratic Hinge Triplet (QHT) loss. QHT introduces an adaptive weighting mechanism that enhances the learning of descriptors by emphasizing errors proportional to the loss magnitude, offering a potent improvement over standard hinge losses.

Experimental Results

The empirical evaluation demonstrates that SOSNet achieves state-of-the-art performance across various benchmarks, most notably the UBC Phototour dataset, HPatches benchmark, and the ETH SfM dataset. Noteworthy achievements include significant reductions in false positive rates at 95% recall compared to alternatives like HardNet and GeoDesc, underscoring the efficacy of utilizing SOSR. Additionally, SOSNet performs impressively on the ETH SfM benchmark, highlighting its utility in practical 3D reconstruction tasks by yielding denser and more accurate 3D models.

Impact on Descriptor Space Utilization

A fascinating aspect of the paper is the proposed evaluation using the von Mises-Fisher (vMF) distribution to analyze the utilization of the descriptor space. This analysis reveals that enforcing SOSR results in descriptors that occupy a larger portion of the hypersphere, indicated by a decrease in intra-class concentration and an increase in inter-class dispersion. Consequently, this enhances the discriminative power of SOSNet in high-dimensional spaces.

Implications and Future Directions

The introduction of SOSR is a significant contribution to local descriptor learning, bridging techniques from graph matching and clustering with local feature learning in computer vision. This integration of second order similarities enriches the learned descriptors' representation, making them more robust for numerous applications in computer vision and robotics.

Future research could explore further integration of spatial transformer networks to alleviate geometric distortions alongside SOSR, potentially improving descriptor efficiency in more complex environments. Additionally, experiments may expand to real-time applications where computational efficiency and descriptor robustness are pivotal.

In summary, SOSNet sets a new benchmark for local descriptor learning, providing a robust and computationally efficient framework for leveraging second order relationships in data. This work presents both practical advancements and theoretical insights that are likely to inspire subsequent developments in the field.