Spatially Adaptive Self-Supervised Learning for Real-World Image Denoising

Abstract: Significant progress has been made in self-supervised image denoising (SSID) in the recent few years. However, most methods focus on dealing with spatially independent noise, and they have little practicality on real-world sRGB images with spatially correlated noise. Although pixel-shuffle downsampling has been suggested for breaking the noise correlation, it breaks the original information of images, which limits the denoising performance. In this paper, we propose a novel perspective to solve this problem, i.e., seeking for spatially adaptive supervision for real-world sRGB image denoising. Specifically, we take into account the respective characteristics of flat and textured regions in noisy images, and construct supervisions for them separately. For flat areas, the supervision can be safely derived from non-adjacent pixels, which are much far from the current pixel for excluding the influence of the noise-correlated ones. And we extend the blind-spot network to a blind-neighborhood network (BNN) for providing supervision on flat areas. For textured regions, the supervision has to be closely related to the content of adjacent pixels. And we present a locally aware network (LAN) to meet the requirement, while LAN itself is selectively supervised with the output of BNN. Combining these two supervisions, a denoising network (e.g., U-Net) can be well-trained. Extensive experiments show that our method performs favorably against state-of-the-art SSID methods on real-world sRGB photographs. The code is available at https://github.com/nagejacob/SpatiallyAdaptiveSSID.

• The paper presents a novel self-supervised learning framework addressing spatially correlated noise in real-world image denoising through a spatially adaptive supervision mechanism.
• The method employs two networks, a Blind-Neighborhood Network (BNN) for flat areas and a Locally Aware Network (LAN) for textured areas, coupled to provide adaptive supervision for a denoising network.
• This approach enables more effective real-world image denoising without requiring clean-noisy pairs and advances self-supervised learning by incorporating spatial adaptivity based on data characteristics.

Spatially Adaptive Self-Supervised Learning for Real-World Image Denoising

In this work, the authors present a novel self-supervised learning framework aimed at enhancing image denoising in real-world scenarios. Traditional self-supervised image denoising approaches typically fail when confronted with spatially correlated noise inherent in real-world sRGB photographs, largely due to assumptions of spatial independence that do not hold true in practical settings. To address this issue, the authors propose a spatially adaptive supervision mechanism that distinguishes flat regions from textured regions in noisy images and applies different denoising strategies tailored to these distinct characteristics. The proposed method avoids the destructive pixel-shuffle downsampling technique and focuses on maintaining the integrity of the original image information.

The method introduces two major innovations: a Blind-Neighborhood Network (BNN) tailored for flat areas and a Locally Aware Network (LAN) for textured areas. The BNN extends the traditional blind-spot network by creating a larger blind region around each pixel to effectively bypass the influence of spatially correlated noise. This network is trained using a self-supervised approach derived from non-adjacent surrounding pixels, which are unaffected by the noise correlation. For textured areas, LAN emphasizes the importance of adjacent pixels, which are crucial for preserving content integrity and texture details, and is supervised by the BNN's output data.

The framework couples these two networks to train a denoising network utilizing the adaptive supervision obtained from BNN and LAN. The authors demonstrate that their approach offers favorable results compared to existing state-of-the-art SSID methods, in terms of both improved image quality and efficiency, as evidenced by performance metrics such as PSNR and SSIM. Moreover, the adaptive coefficients generated from local variance of BNN output facilitate the dynamic adjustment between flat-area and textured-area denoising strategies.

The implications of this research are significant. Practically, it enables more effective denoising of real-world images without dependence on clean-noisy image pairs, which are cumbersome to obtain due to variations across devices and conditions. Theoretically, it advances the paradigm of self-supervised learning by incorporating spatial adaptivity as a core concept, paving the way for future AI developments that can autonomously tailor their operations based on data characteristics.

In conclusion, the authors' contribution of combining BNN and LAN within a unified framework represents a substantial advancement in self-supervised image denoising. It addresses the challenges posed by spatially correlated noise in real-world scenarios through an innovative adaptive learning strategy. Further research may focus on extending the framework's capabilities to broader types of noise and investigating its applicability in other unsupervised visual learning tasks.