Neural Non-Rigid Tracking

Abstract: We introduce a novel, end-to-end learnable, differentiable non-rigid tracker that enables state-of-the-art non-rigid reconstruction by a learned robust optimization. Given two input RGB-D frames of a non-rigidly moving object, we employ a convolutional neural network to predict dense correspondences and their confidences. These correspondences are used as constraints in an as-rigid-as-possible (ARAP) optimization problem. By enabling gradient back-propagation through the weighted non-linear least squares solver, we are able to learn correspondences and confidences in an end-to-end manner such that they are optimal for the task of non-rigid tracking. Under this formulation, correspondence confidences can be learned via self-supervision, informing a learned robust optimization, where outliers and wrong correspondences are automatically down-weighted to enable effective tracking. Compared to state-of-the-art approaches, our algorithm shows improved reconstruction performance, while simultaneously achieving 85 times faster correspondence prediction than comparable deep-learning based methods. We make our code available.

• The paper proposes an end-to-end learnable framework combining CNN-based correspondence prediction with a differentiable Gauss-Newton solver for robust non-rigid tracking.
• Key techniques include a differentiable Gauss-Newton solver and self-supervised correspondence weighting for improved robustness against outliers.
• The method achieves 85x faster correspondence prediction and improved tracking error metrics, with practical implications for AR/VR and robotics.

Summary of "Neural Non-Rigid Tracking" Paper

The paper "Neural Non-Rigid Tracking" presents a novel approach for non-rigid tracking and reconstruction using single RGB-D frames. It leverages a fully end-to-end learnable and differentiable framework that combines deep learning with classical optimization techniques to address challenges in tracking dynamic and deformable objects in computer vision. Given two RGB-D frames, the authors employ a Convolutional Neural Network (CNN) to predict dense correspondences and their associated confidences, which serve as constraints within an As-Rigid-As-Possible (ARAP) optimization framework. The integration of gradient back-propagation into non-linear least squares solvers facilitates learning optimal correspondences and confidences for tracking purposes.

Key Contributions

1. End-to-end Differentiable Gauss-Newton Solver: The paper presents an innovative differentiable Gauss-Newton solver within their tracking pipeline. This enables back-propagation for better-informed correspondence predictions that are vital for effective non-rigid tracking.
2. Self-supervised Correspondence Weighting: The authors propose a self-supervised learning approach for determining correspondence importance weights, which helps in robust outlier rejection during tracking. This approach improves the non-rigid reconstruction results compared to existing state-of-the-art methods.

Numerical Results and Claims

• The paper reports an $85\times$ improvement in correspondence prediction speed while enhancing reconstruction performance compared to existing methods.
• The proposed approach yields substantial improvements in non-rigid tracking error metrics, including both EPE 3D and Graph Error 3D measurements, showcasing its superiority over previous solutions.

Theoretical and Practical Implications

The work bridges classical optimization techniques with modern neural network methodologies to advance non-rigid tracking and reconstruction problems. By integrating differentiable components into the non-rigid reconstruction pipeline, the authors demonstrate potential advancements in AR/VR applications, robotics, and autonomous driving systems. This approach can be viewed as a stepping stone toward fully differentiable non-rigid reconstruction models.

Speculation on Future Developments in AI

This paper suggests significant future directions for AI in computer vision. The successful integration of differentiable solvers into deep learning frameworks may lead to new models that exploit both geometric insights and data-driven patterns efficiently. As AI systems increasingly deal with dynamic and complex environments, the techniques presented here may find extended applications not only in real-world capture but also in simulation and modeling tasks.

The paper "Neural Non-Rigid Tracking" stands out as an insightful contribution to the non-rigid tracking domain, offering promising methodologies and results that may inspire subsequent innovations and applications in AI-driven computer vision systems.