Edge-Aware Smoothness Loss

• Edge-aware smoothness loss is a regularization technique that adaptively weights smoothing penalties using local image gradients to preserve key boundaries.
• It employs edge maps from intensity or gradient differences to balance smoothness across homogeneous regions with sharp transitions at object edges.
• Empirical results in depth estimation and image restoration demonstrate up to a 10% error reduction and enhanced perceptual quality using this method.

Edge-aware smoothness loss constitutes a class of regularization techniques in computer vision that adaptively enforce smoothness on outputs such as depth, segmentation, restored images, or perturbations, while explicitly preserving sharp discontinuities around perceptually or geometrically salient edges. This loss leverages local image or signal properties—typically through intensity, color, structural, or prior output gradients—to weight the smoothing penalty, ensuring high-fidelity reconstruction of both homogeneous regions and boundaries where structural information is critical. These methods are prevalent across unsupervised geometry reconstruction, image restoration, adversarial perturbation construction, and edge-preserving image smoothing.

1. Formulation and Mechanisms

The prototypical edge-aware smoothness loss modulates the penalty on spatial gradients by an edge indicator function derived from auxiliary data (usually image intensity or the current estimate). The canonical form, as employed in unsupervised geometry and depth-normal estimation (Yang et al., 2017), is: $$L_s^D(D) = \sum_{p} \sum_{d\in\{x,y\}} \left| \nabla^2_d D(p) \right| \cdot \exp(-\alpha\,|\nabla_d I(p)|)$$ where $D(p)$ is the estimated depth at pixel $p$, $\nabla_d$ is the finite difference operator in direction $d$, $I(p)$ is the input image, and $\alpha$ is a positive constant. The exponential term suppresses the smoothness penalty at strong image edges, permitting discontinuities in $D$ at positions where the underlying image is non-smooth.

Alternate variants include intensity-difference based neighbor weighting (Yang et al., 2017),

$\omega_{ji} = \exp(-\alpha|I(x_j) - I(x_i)|),$

graph-based edge coupling for perturbations (Zhang et al., 2019), and multi-scale or groundtruth-driven edge masks in deep smoothing networks (Kosheleva et al., 2023, Zhu et al., 2019). These mechanisms universally encode the intuition that true boundaries—whether semantic, depth, or photometric—should not be artificially smoothed.

2. Applications in Geometry and Depth Estimation

In monocular and stereo depth or normal estimation, edge-aware smoothness loss is critical to enforcing geometric consistency without oversmoothing object boundaries:

• In unsupervised depth-normal estimation, (Yang et al., 2017) implements both depth and normal smoothness terms, each modulated by the corresponding image gradients. Additionally, edge-aware consistency terms in the depth-to-normal and normal-to-depth conversion layers are weighted by photometric differences, preventing erroneous feature propagation across surfaces.
• In video or stereo depth estimation, edge-aware losses act only on regions with strong geometric gradients identified in an initial depth estimate. For example, (Kosheleva et al., 2023) proposes multi-scale and contrastive gradient-matching terms, computing edge masks from an initial depth map and penalizing only deviations at those masked locations, thereby preserving object boundaries and high-frequency details during test-time adaptation.

Quantitatively, ablations demonstrate that removing edge weighting in smoothness loss increases absolute-relative depth error by 10% (from 0.172 to 0.189 on KITTI) (Yang et al., 2017); on stereo depth, targeted edge-preserving losses yield up to 10% reductions in photometric loss and crisper qualitative boundaries (Kosheleva et al., 2023).

3. Edge-aware Regularization in Image Restoration and Smoothing

Edge-aware smoothness penalties are foundational in image restoration pipelines—especially deblurring, denoising, and smoothing:

• The Edge-Adaptive Hybrid Regularization (EAHR) model (Zhang et al., 2020) combines spatially-varying total variation and Tikhonov regularizers. Edge maps inferred from local gradients dictate pixelwise smoothing strength, with reduced penalties at detected edges via shrink factors $\theta_m \in (0,1)$. Dynamic updating of the edge mask guarantees that true edges are protected throughout iterative optimization and prevents noise-misclassification.
• Deep CNN-based smoothing, as in (Zhu et al., 2019), employs a neighborhood gradient-matching loss,

$$\mathcal{L}_{\mathrm{nb}} = \sum_{...} w_{t,k}\left\| [M_\theta(x^t)_{i,j} - M_\theta(x^t)_{p,q}] - [Y^{t,k}_{i,j} - Y^{t,k}_{p,q}] \right\|_1,$$

which aligns the predicted and groundtruth local gradients in a $5 \times 5$ neighborhood. This penalizes deviations only where the groundtruth is intrinsically smooth and permits matched discontinuities at groundtruth edges, leading to superior edge preservation.

On benchmark datasets, such edge-aware frameworks outperform isotropic TV or $\ell_2$ regularization both numerically (e.g., best PSNR/SSIM on GB(9,5)/$\sigma$=5 for EAHR (Zhang et al., 2020); lowest WRMSE/WMAE for CNN+neighborhood loss (Zhu et al., 2019)) and visually, preserving sharp transitions without over-smoothing artifacts.

4. Graph-based Edge-aware Loss in Adversarial Example Construction

Edge-aware smoothness constraints have been adapted for adversarial perturbation synthesis, notably in perceptually optimized attacks (Zhang et al., 2019). Here, smoothing is imposed on the perturbation $r$ via an underlying image-dependent Laplacian graph:

• A pixel affinity graph is built with weights $w_{ij} = k_f(x_{0i}, x_{0j}) k_s(p_i, p_j)$, encoding both color similarity and spatial proximity.
• The Laplacian regularization penalizes changes in $r$ predominantly between similar, spatially close pixels, while allowing sharp jumps at detected edges.

$$\phi_\alpha(r, z) = \frac{\alpha}{2} \sum_{ij} w_{ij} \|D^{-1/2}r_i - D^{-1/2}r_j\|^2 + (1-\alpha)\|r-z\|^2$$

• The perturbation is constructed by joint minimization subject to this edge-aware smoothing operator, resulting in attacks that are locally constant over flat regions, ride the fine texture in detailed regions, and shift abruptly at object or color boundaries.

Ablations confirm that smooth, edge-aware attacks maintain both invisibility and high attack success—reaching a lower $L_2$ distortion for the same or higher fooling rates compared to unregularized or naively smoothed counterparts.

5. Integration into Training and Optimization Objectives

Edge-aware smoothness loss is integrated additively or as a multiplicative weight within larger learning objectives:

• In geometry learning (Yang et al., 2017), the total loss comprises photometric reconstruction, mask smoothness, and both depth and normal edge-aware regularization, tuned by hyperparameters such as $\lambda_s$ (depth smoothness), $\lambda_n$ (normal smoothness), and a shared edge-weighting exponent $\alpha$.
• For deep smoothing and restoration, neighborhood losses are combined directly with pixelwise $\ell_1$ or $\ell_2$ data terms, often without additional scalar weighting (Zhu et al., 2019).
• In adversarial optimization, the edge-aware operator is either imposed as a hard constraint on the perturbation or as a penalty term, with all gradients computed via backpropagation through the edge-weighted Laplacian operation (Zhang et al., 2019).
• For edge-adaptive variational models, the spatially-varying regularization coefficients are themselves functions of dynamically updated edge maps, increasing model adaptivity during iterative ADMM solvers (Zhang et al., 2020).

6. Empirical Impact and Comparative Evaluation

The practical advantages of edge-aware smoothness loss span several axes:

Application	Baseline	With Edge-aware Loss	Quantitative/Qualitative Gain
Depth estimation (KITTI, rel. error)	Standard smoothness	Edge-aware $\rightarrow$ 0.172	Absolute-relative error improved (Yang et al., 2017)
Stereo video depth (ETH3D/KITTI)	Geom. only ($L_1$)	+ Multi-scale/contrastive loss	$\sim$10% error reduction, sharper edges (Kosheleva et al., 2023)
Deblurring (PSNR/SSIM)	BM3D: 26.52/0.809	EAHR: 26.89/0.830	Highest scores, preserves edges (Zhang et al., 2020)
Image smoothing (WRMSE/WMAE)	$\ell_2$: 10.14/6.92	$\ell_1$+neigh.: 9.78/6.15	Best quantitative+visual smoothing (Zhu et al., 2019)
CW adversarial (MNIST)	Dist: 2.49	Edge-aware: 1.97	Visually imperceptible, equally successful (Zhang et al., 2019)

In all domains, introducing edge-aware regularization yields models that are more faithful to semantic content, less prone to oversmoothing, and empirically superior by objective and subjective criteria. Edge-weighting preserves scene structure and small-scale discontinuities where naive isotropic regularizers fail.

7. Design Choices and Implementation Considerations

Salient implementation features for edge-aware smoothness include:

• Edge weights may be computed from input image gradients, from the model’s prior outputs, or from additional structural cues depending on the application and available data.
• The edge-aware terms are fully differentiable in all cited works, ensuring compatibility with modern optimization pipelines and enabling end-to-end learning.
• For dynamic or iterative algorithms, edge maps and their corresponding smoothness weights are updated at each outer iteration, closely tracking the evolving signal and suppressing spurious edges.
• Hyperparameters such as the edge-threshold exponent ($\alpha$), mask shrink-factors ($\theta$), and loss weights ($\lambda$) are critical for balancing smoothing and edge preservation and are often dataset/noise-level dependent (Yang et al., 2017, Zhang et al., 2020).
• Cost: While neighborhood or graph-based terms incur higher memory and compute costs due to expanded stencils or affinity matrices, modern hardware and efficient graph/Laplacian solvers enable tractable training and inference for large-scale models (Zhang et al., 2019, Zhu et al., 2019).

In sum, edge-aware smoothness losses are indispensable in contemporary visual estimation, restoration, and adversarial generation tasks. By conditioning regularization on local or global edge information, they achieve a fundamental trade-off between denoising/smoothing and structure preservation, backed by robust empirical validation across modalities and architectures.