DoubleTake: Geometry Guided Depth Estimation

Abstract: Estimating depth from a sequence of posed RGB images is a fundamental computer vision task, with applications in augmented reality, path planning etc. Prior work typically makes use of previous frames in a multi view stereo framework, relying on matching textures in a local neighborhood. In contrast, our model leverages historical predictions by giving the latest 3D geometry data as an extra input to our network. This self-generated geometric hint can encode information from areas of the scene not covered by the keyframes and it is more regularized when compared to individual predicted depth maps for previous frames. We introduce a Hint MLP which combines cost volume features with a hint of the prior geometry, rendered as a depth map from the current camera location, together with a measure of the confidence in the prior geometry. We demonstrate that our method, which can run at interactive speeds, achieves state-of-the-art estimates of depth and 3D scene reconstruction in both offline and incremental evaluation scenarios.

• The paper introduces DoubleTake, integrating historical 3D geometry via a persistent TSDF to enhance depth estimation.
• It employs a Hint MLP that combines cost volume features with geometric hints for robust, real-time scene reconstruction.
• Empirical results demonstrate state-of-the-art performance across several datasets, improving reconstruction accuracy and depth metrics.

DoubleTake: Geometry Guided Depth Estimation

The paper introduces a novel method called "DoubleTake" for high-quality depth estimation and 3D scene reconstruction, leveraging historical predictions of 3D geometry as guidance. Developed to function at interactive speeds, the method circumvents limitations of traditional Multi-View Stereo (MVS) approaches by incorporating geometry derived from previously seen frames. This integration enhances the depth estimation process and improves the accuracy of scene reconstructions.

Methodology

At the core of DoubleTake is the innovative use of historical 3D geometry data to guide current depth estimations. Traditional MVS methods build cost volumes using input sequences and target frames, which are then processed to predict depth maps. DoubleTake advances this paradigm by utilizing a persistent 3D representation of the scene, encoded as a truncated signed distance function (TSDF). This representation is continuously updated with depth maps derived from prior predictions, captured from various viewpoints over time.

Key Components

1. Hint MLP (Multi-Layer Perceptron):
   • The Hint MLP operates by combining cost volume features with geometric hints—a depth map rendered from the persistent 3D geometry—and a confidence measure for this depth estimate.
   • This architecture allows the network to effectively utilize short-term and long-term geometric information, improving depth prediction robustness even when direct visual matching between frames is challenging.
2. TSDF Representation:
   • The TSDF accumulates depth information incrementally, enabling a constantly evolving, highly detailed geometric representation of the scene.
   • Depth maps are rendered from this TSDF, providing the network with geometric context that spans both recently and previously visited regions within the scene.
3. Training Strategy:
   • The training process includes scenarios both with and without geometric hints to ensure robustness. For improved depth predictions, geometric hints are generated using a variety of depths and confidence levels from pre-existing scene models.
   • The method also incorporates a novel training protocol that carefully considers ground-truth mesh limitations, ensuring realistic evaluation.

Experimental Results

Empirical evaluation demonstrates that DoubleTake achieves state-of-the-art performance in depth estimation and scene reconstruction across several datasets, including ScanNetV2, 7Scenes, and 3RScan.

1. Depth Estimation Metrics:
   • DoubleTake outperforms existing methods like SimpleRecon and DeepVideoMVS on depth estimation accuracy. On ScanNetV2, for example, it scores the best in Abs Rel and Sq Rel metrics, confirming the efficacy of using geometric hints.
   • The method also shows robustness in 7Scenes dataset evaluations, where it maintains top performance compared to other MVS-based techniques.
2. 3D Scene Reconstruction:
   • The mesh evaluation reveals superior reconstruction quality, with DoubleTake achieving lower Chamfer distances and higher precision-recall scores. The method is particularly adept at maintaining high reconstruction fidelity in online, interactive scenarios.
3. Long-term Scene Understanding:
   • A unique aspect of this method is its capacity to incorporate long-term geometric hints, validated using the 3RScan dataset. This capability enables it to handle scenes revisited after significant temporal gaps, maintaining high depth estimation accuracy despite potential changes in the scene.

Implications and Future Directions

The results of this research have significant practical implications for augmented reality (AR), autonomous navigation, and robotics:

• Augmented Reality:
  • By providing instantaneous and highly accurate depth maps, AR applications will benefit from more realistic and immersive experiences.
• Autonomous Systems:
  • For path planning and object avoidance, the improved depth estimation promises safer and more reliable navigation.

Theoretically, this research paves the way for further exploration into the integration of geometric priors in neural networks and enhancing reconstruction quality through incremental updates. Future work could explore optimizing the TSDF update mechanisms for even faster real-time processing and extending the approach to outdoor or more dynamically changing environments.

Conclusion

DoubleTake represents a significant advancement in the field of depth estimation and 3D scene reconstruction. Its ability to leverage historical geometry for guiding current predictions marks a notable improvement over traditional MVS methods, both in accuracy and runtime performance. The research outlines a robust approach to depth prediction that holds promise for a wide range of applications in computer vision and beyond.