Unsupervised Depth & Edge Learning

• Unsupervised depth and edge learning is a method that recovers scene geometry and sharp boundaries from unlabeled data using photometric consistency and geometric priors.
• Techniques employ encoder–decoder architectures with multi-scale skip connections and tailored loss functions to enhance depth fidelity and preserve contour details.
• Integrating semantic segmentation and motion estimation, these frameworks effectively handle dynamic scenes while mitigating blur and bleed artifacts at object edges.

Unsupervised Depth and Edge Learning encompasses a family of computer vision frameworks that exploit unlabeled video and image data to simultaneously recover scene geometry (depth, normals) and sharp structure boundaries (edges). These methods avoid reliance on ground-truth depth or segmentation annotations, leveraging photometric consistency, geometric priors, and edge-aware smoothness terms to optimize neural networks. Recent advances have shown that embedding edge constraints—either through spatial weighting, explicit border alignment, or feature coupling—yields substantial improvements in depth map fidelity, contour preservation, and efficiency across challenging domains, including dynamic scenes and semantic-rich environments.

1. Fundamental Objectives and Problem Formulation

The core objective in unsupervised depth and edge learning is to infer pixelwise scene depths, surface normals, and edge maps from monocular or stereo video sequences without supervised signals. This is achieved by enforcing geometric consistency (e.g., planar smoothness, depth-normal compatibility), photometric reconstruction, and edge-preserving regularization. Central to this paradigm is the integration of image structure (texture, intensity gradients) as a prior, such that depth discontinuities are permitted at strong edges, while geometric quantities are constrained to be locally smooth elsewhere. For example, in edge-aware frameworks, the smoothness penalty on depth $\mathcal{L}_{\rm smooth}$ is modulated by image gradient magnitudes, allowing for sharp jumps at object boundaries (Zhou et al., 2019).

2. Architectural Innovations and Edge-Aware Modules

Modern unsupervised geometry networks utilize encoder–decoder architectures (e.g., DispNet-style, U-Net variants) with multi-scale skip connections to recover detailed structure. A typical configuration includes a depth decoder to generate disparity maps, a pose estimation branch for camera motion, and often additional modules for surface normals and explanation masks.

Recent contributions emphasize edge-aware and adaptive fusion mechanisms:

• Edge-Preserving Residual Upsampling: RM-Depth (Hui, 2023) divides the upsampler into two pathways: a bilinear branch for smooth content and a learned deconvolution branch for edge detail, summing their outputs to maintain sharp contours without blurring fine structure.
• Recurrent Modulation Units (RMUs): These iteratively combine encoder features and hidden states across pyramidal levels, modulating information flow to enhance both smooth depth inference and edge localization, enabling compact models to outperform larger baselines (Hui, 2023).
• Depth-Normal Consistency Layers: Differentiable modules explicitly convert depths to normals (via cross products of local back-projected 3D points) and normals back to planarity-regularized depths, with edge-aware weights suppressing regularization across image boundaries (Yang et al., 2017).

3. Edge-Aware Loss Functions and Regularizers

A decisive feature of this field is the development of loss functions which account for image edges, geometric compatibility, and border consistency. Prominent formulations include:

• Edge-aware depth smoothness:

$$\mathcal{L}_{\rm smooth} = \sum_p |\partial_u d(p)|\,e^{-|\partial_u I_t(p)|} + |\partial_v d(p)|\,e^{-|\partial_v I_t(p)|}$$

Smoothing is attenuated in regions of high image gradient—preserving edges (Zhou et al., 2019, Hui, 2023).

• Depth–Normal Consistency:

Utilizing a depth-to-normal conversion and vice versa, enforcing local planarity except across edges defined by:

$w_{ji} = \exp(-\alpha|I(x_j)-I(x_i)|)$

This weighting scheme selectively relaxes regularization penalties across semantic boundaries (Yang et al., 2017).

• Explicit Border Alignment Loss: In "The Edge of Depth" (Zhu et al., 2020), the border-consistency loss directly compares semantic segmentation edge pixels to their nearest depth edge, penalizing their distance:

$$L_{edge} = \frac{1}{|\Gamma_s^d|} \sum_{q\in\Gamma_s^d} \min_{p\in T_d}\|p-q\|$$

Iterative "morph-loss" further warps network outputs to local optima that perfectly align depth edges with semantic edges, improving contour fidelity.

4. Integration of Multiple Modalities and Training Strategies

Unsupervised depth and edge learning leverages monocular video and stereo imagery for self-supervision:

• Temporal and Stereo View Synthesis: Depth networks reconstruct both temporally adjacent and stereo views via photometric warping, supporting multi-modal consistency losses (Zhou et al., 2019).
• Disparity Consistency: Jointly predicted left and right disparities are regularized to agree both spatially and photometrically.
• Semantic Segmentation Constraints: Incorporating predicted segmentation maps (trained even with weak labels) as fixed boundaries for depth refinement leads to superior border localization and helps to prevent bleed artifacts (Zhu et al., 2020).

Training schedules typically employ Adam optimizers, multi-scale loss aggregation, and staged fine-tuning. Notably, models exploiting stereo and segmentation cues converge with significantly fewer video sequences, improving practicality (Zhou et al., 2019).

5. Handling Dynamic Scenes and Moving Objects

Classical unsupervised frameworks assume scene rigidity; recent advances address non-static environments:

• Motion Field Estimation: RM-Depth (Hui, 2023) predicts per-pixel 3D object motions in addition to camera pose, enabling geometry recovery in dynamic scenes. Object motion decoders operate in a coarse-to-fine pyramid, regularized by sparsity-promoting losses gated by rigid-flow masks.
• Outlier-Aware Regularization: Pixels exhibiting large discrepancies between rigid and full flow are gated to focus regularization only where motion is required, improving depth accuracy in the presence of moving objects.

This facilitates the use of large-scale monocular video datasets, overcoming former limitations of unsupervised methods.

6. Experimental Benchmarks and Quantitative Improvements

Quantitative evaluations on standard datasets such as KITTI and Cityscapes consistently demonstrate that edge-aware and border-aligned depth learning yields marked gains:

Method	Abs Rel	RMSE	δ<1.25	Edge Alignment Strategy
Zhou et al. (unsup baseline)	0.207	6.658	0.670	None
RM-Depth (Hui, 2023)	0.107	4.476	0.883	Residual upsampling, RMUs
Edge-aware DispNet (Zhou et al., 2019)	0.195	6.505	0.717	$e^{-|\nabla I|}$ weighted smooth
Border-aligned (Ours) (Zhu et al., 2020)	0.091	4.350	0.898	Morph-loss + semantic segmentation

Models employing explicit border alignment achieve state-of-the-art unsupervised results competitive with supervised baselines. Qualitative improvements include sharper depth discontinuities at object boundaries, reduced bleed artifacts, and higher fidelity in thin structures such as poles and foliage.

7. Implications and Future Directions

The data suggest that integration of edge-sensitive priors, semantic border information, and adaptive fusion mechanisms can dramatically improve self-supervised geometry recovery. Plausible implications include the utility of these strategies in downstream tasks such as instance segmentation, robotics navigation, AR content localization, and the construction of edge-resolved scene representations from video or weakly labeled sources.

Future research will likely extend these methods to:

• Scenarios with severe occlusion and non-Lambertian surfaces
• Real-time deployment on resource-constrained devices via efficient architectures (e.g., RMU-enabled compact decoders)
• Joint estimation of depth, edges, normals, semantic masks and possibly texture/material properties within unified unsupervised frameworks

The convergence of geometric and semantic alignment, as evidenced by recent SOTA, delineates a path toward fully annotation-free scene understanding with high spatial fidelity.