The Devil is in the Details: Boosting Guided Depth Super-Resolution via Rethinking Cross-Modal Alignment and Aggregation

Abstract: Guided depth super-resolution (GDSR) involves restoring missing depth details using the high-resolution RGB image of the same scene. Previous approaches have struggled with the heterogeneity and complementarity of the multi-modal inputs, and neglected the issues of modal misalignment, geometrical misalignment, and feature selection. In this study, we rethink some essential components in GDSR networks and propose a simple yet effective Dynamic Dual Alignment and Aggregation network (D2A2). D2A2 mainly consists of 1) a dynamic dual alignment module that adapts to alleviate the modal misalignment via a learnable domain alignment block and geometrically align cross-modal features by learning the offset; and 2) a mask-to-pixel feature aggregate module that uses the gated mechanism and pixel attention to filter out irrelevant texture noise from RGB features and combine the useful features with depth features. By combining the strengths of RGB and depth features while minimizing disturbance introduced by the RGB image, our method with simple reuse and redesign of basic components achieves state-of-the-art performance on multiple benchmark datasets. The code is available at https://github.com/JiangXinni/D2A2.

• The paper introduces the D2A2 network that dynamically aligns and aggregates RGB and depth modalities to enhance depth clarity.
• It employs learnable domain and dynamic geometrical alignment along with gated convolution and pixel attention to resolve misalignment issues.
• Experimental results show lower RMSE and sharper boundaries across datasets, establishing a new benchmark for guided depth super-resolution.

The Devil is in the Details: Boosting Guided Depth Super-Resolution via Rethinking Cross-Modal Alignment and Aggregation

Introduction

Guided depth super-resolution (GDSR) seeks to enhance depth map resolution using corresponding high-resolution RGB images. Despite advancements in this domain, challenges persist due to the heterogeneity and complementarity of RGB and depth modalities. Previous methodologies have struggled with alignment and aggregation issues, leading to noise and blurred boundaries in the results. This paper introduces a novel network, Dynamic Dual Alignment and Aggregation (D2A2), addressing these challenges through a two-pronged approach.

Methodology

The D2A2 architecture comprises two primary modules:

1. Dynamic Dual Alignment Module (DDA):

The module aims to rectify modal and geometrical misalignments between RGB and depth features. It includes: - Learnable Domain Alignment (LDA): Uses learnable parameters to adjust RGB features to match the depth feature distribution, enhancing cross-modal feature compatibility. - Dynamic Geometrical Alignment (DGA): Employs deformable convolution to adaptively align RGB features to their depth counterparts, utilizing learned offsets and modulation scalars for enhanced spatial alignment.

Figure 1: An overview of the proposed D2A2 network, showcasing the dynamic alignment and feature aggregation modules.

2. Mask-to-Pixel Feature Aggregation Module (MFA):

This module refines feature fusion through: - Gated Convolution (GC): Filters out irrelevant textures from RGB features using dynamically updated masks. - Pixel Attention (PA): Selectively emphasizes pertinent details during feature fusion, allowing pixel-level attention and integration to improve depth map clarity.

(Figure 2 and Figure 3)

Figure 2: Visualization of RGB features before and after dynamic dual alignment, and associated masks and weight maps used in feature aggregation.

Figure 3: Histogram analysis demonstrating the effect of LDA on aligning RGB features closer to the distribution of depth features.

Experimental Results

D2A2 was evaluated against state-of-the-art methods across multiple datasets, including NYUv2, Middlebury, Lu, and RGBDD, at various scaling factors. The network consistently outperformed existing models, achieving lower RMSE across most configurations, underscoring its effectiveness in precise depth reconstruction. Sample visual comparisons reveal sharper boundaries and reduced artifacts in results produced by D2A2.

(Figure 4 and Figure 5)

Figure 4: Comparison of different methods on the Middlebury dataset for $\times$8 depth super-resolution, highlighting the improvements in boundary sharpness by D2A2.

Figure 5: Performance of various methods on the RGBDD dataset at $\times$8 depth super-resolution, showing D2A2's noise reduction capabilities.

Implications and Future Work

The proposed dual alignment and aggregation strategies significantly enhance GDSR accuracy, offering a promising direction for future research. Potential extensions may include more sophisticated alignment mechanisms and integration with other modalities for richer scene understanding. The general approach of dynamic feature alignment and selective aggregation could be applicable across broader vision tasks beyond GDSR, fostering further exploration into cross-modal research avenues.

Conclusion

D2A2 represents a pivotal advancement in addressing core challenges within GDSR by optimizing cross-modal feature alignment and aggregation. The methodologies introduced are validated through comprehensive quantitative and qualitative assessments, demonstrating superior performance and robustness in various scenarios. Given its success, D2A2 lays a foundation for continued innovation in guided depth map enhancement and related fields.