LRRU: Long-short Range Recurrent Updating Networks for Depth Completion

Abstract: Existing deep learning-based depth completion methods generally employ massive stacked layers to predict the dense depth map from sparse input data. Although such approaches greatly advance this task, their accompanied huge computational complexity hinders their practical applications. To accomplish depth completion more efficiently, we propose a novel lightweight deep network framework, the Long-short Range Recurrent Updating (LRRU) network. Without learning complex feature representations, LRRU first roughly fills the sparse input to obtain an initial dense depth map, and then iteratively updates it through learned spatially-variant kernels. Our iterative update process is content-adaptive and highly flexible, where the kernel weights are learned by jointly considering the guidance RGB images and the depth map to be updated, and large-to-small kernel scopes are dynamically adjusted to capture long-to-short range dependencies. Our initial depth map has coarse but complete scene depth information, which helps relieve the burden of directly regressing the dense depth from sparse ones, while our proposed method can effectively refine it to an accurate depth map with less learnable parameters and inference time. Experimental results demonstrate that our proposed LRRU variants achieve state-of-the-art performance across different parameter regimes. In particular, the LRRU-Base model outperforms competing approaches on the NYUv2 dataset, and ranks 1st on the KITTI depth completion benchmark at the time of submission. Project page: https://npucvr.github.io/LRRU/.

• The paper introduces the LRRU network that iteratively refines sparse depth maps using adaptive long-short range kernel updates.
• The method reduces computational complexity while achieving competitive performance on the KITTI benchmark with minimal parameters.
• The approach advances iterative refinement in neural networks, enabling efficient real-time depth completion in constrained hardware environments.

An Evaluation of LRRU: Long-short Range Recurrent Updating Networks for Depth Completion

The paper "LRRU: Long-short Range Recurrent Updating Networks for Depth Completion" introduces an innovative approach to depth completion that addresses the significant computational challenges associated with deep learning-based methods. The primary method proposed, the Long-short Range Recurrent Updating (LRRU) network, offers a refined solution that aims to efficiently transform sparse depth inputs into dense depth maps, with the potential use cases in fields such as autonomous driving and augmented reality.

Depth sensors, such as LiDAR, frequently produce sparse depth maps, requiring substantial processing to fill in the missing information. While existing approaches leverage convolutional neural networks (CNNs) and spatial propagation networks (SPNs) to infer dense maps directly, these techniques are often computationally intensive, necessitating large networks and extensive processing power. This paper's contribution is significant in that it proposes a framework that bypasses the need for complex feature learning, instead iteratively updating a coarse initial map using a novel recurrent strategy.

The LRRU network starts by creating an initial dense depth map from sparse inputs. This map is further refined through iterative updates using learned spatially-variant kernels. These kernels are determined by taking into account the RGB guidance images and the depth map requiring enhancement. The innovation lies in dynamically adjusting kernel scopes to capture dependencies ranging from long to short distances, thus optimizing the refinement process. This approach significantly reduces computational load while maintaining high accuracy, demonstrated by the outstanding performance of LRRU variants on the KITTI benchmark, where the LRRU-Base model emerged as the leading method.

The numerical results presented indicate that even the smallest version of LRRU, with only 0.3 million parameters, can outperform several more demanding models, reaching competitive RMSE values on the KITTI dataset. The LRRU-Base model improves upon state-of-the-art results, indicating its robustness and efficiency despite fewer learning parameters.

From a theoretical perspective, the paper advances the understanding of iterative refinement processes within neural networks, suggesting a departure from traditional direct regression. Its implications extend toward more efficient processing capabilities in real-time applications, especially in scenarios constrained by hardware capacities, such as embedded systems in autonomous vehicles.

Looking forward, the research provides a solid foundation for further exploration into lightweight model architectures for depth completion, possibly extending to other domains such as monocular depth estimation or semantic segmentation. The strategy utilized in LRRU could inspire additional methodologies that prioritize efficiency and adaptability in neural network design.

The adaptability of kernel scopes and the integration of target-dependent updates present promising avenues for other dense prediction tasks, potentially revolutionizing current approaches that struggle with computational burdens. Further investigation may involve the application of LRRU principles to different input data types or enhancing its iterative framework to accommodate multi-modal input situations more broadly.