Super-Resolution via Image-Adapted Denoising CNNs: Incorporating External and Internal Learning

Abstract: While deep neural networks exhibit state-of-the-art results in the task of image super-resolution (SR) with a fixed known acquisition process (e.g., a bicubic downscaling kernel), they experience a huge performance loss when the real observation model mismatches the one used in training. Recently, two different techniques suggested to mitigate this deficiency, i.e., enjoy the advantages of deep learning without being restricted by the training phase. The first one follows the plug-and-play (P&P) approach that solves general inverse problems (e.g., SR) by using Gaussian denoisers for handling the prior term in model-based optimization schemes. The second builds on internal recurrence of information inside a single image, and trains a super-resolver network at test time on examples synthesized from the low-resolution image. Our work incorporates these two independent strategies, enjoying the impressive generalization capabilities of deep learning, captured by the first, and further improving it through internal learning at test time. First, we apply a recent P&P strategy to SR. Then, we show how it may become image-adaptive in test time. This technique outperforms the above two strategies on popular datasets and gives better results than other state-of-the-art methods in practical cases where the observation model is inexact or unknown in advance.

• The paper introduces IDBP-CNN-IA, a novel super-resolution method combining Plug-and-Play denoising with image-adaptive internal learning.
• This approach outperforms existing methods like EDSR+ and RCAN, particularly showing superior PSNR for images with unknown downscaling kernels and real-world noise.
• The research demonstrates a robust way to handle variable observation models in image super-resolution, opening doors for applications in other inverse problems.

Super-Resolution via Image-Adapted Denoising CNNs

Introduction

The paper presented explores the domain of image Super-Resolution (SR), an area that has seen substantial advancements in recent years with the evolution of deep learning techniques. While Convolutional Neural Networks (CNNs) have achieved state-of-the-art performance in SR when the acquisition process is known and fixed, challenges arise when there is a mismatch between the real observation model and the one used during training. This is particularly relevant in practical scenarios where the observation model is often inexact or unknown in advance.

Innovation and Methodology

The authors introduce a novel approach combining two independent strategies to overcome these challenges. The first strategy leverages the Plug-and-Play (P&P) framework, utilizing Gaussian denoisers within model-based optimization schemes to handle inference tasks without being restricted by the training phase. The second strategy exploits internal recurrence of information within a single image, allowing for the training of a super-resolver network at test time using synthesized examples from the given low-resolution image.

The proposed solution, IDBP-CNN-IA, integrates these methodologies by applying the iterative denoising and backward projections (IDBP) method to SR, making it image-adaptive at test time. By fine-tuning CNN denoisers using internal image information, the technique demonstrates improved performance over existing P&P methods and strategies that only utilize internal learning.

Results and Findings

The paper provides strong empirical evidence supporting the effectiveness of their combined approach. The IDBP-CNN-IA method outperforms both the baseline IDBP and other contemporary methods such as EDSR+ and RCAN in various experiments. Notably, the approach yields superior PSNR results in cases where the downscaling kernel is unknown or non-ideal, and it also shows robustness when applied to real, low-quality images with unknown acquisition models.

For ideal observation models, IDBP-CNN-IA achieved significant improvements over IRCNN and ZSSR, further substantiating its ability to generalize well across different acquisition settings.

Implications and Future Work

This research carries notable implications for both theoretical understanding and practical applications in the field of image super-resolution. The incorporation of image-adaptive learning into the P&P framework opens new avenues for developing more versatile and robust SR methods capable of handling variabilities in real-world observation models. The concept of fine-tuning denoisers based on internal information also presents opportunities for refinement and application in other inverse imaging tasks.

Looking ahead, this method could be further optimized for computational efficiency, facilitating faster real-time applications. Additionally, the principles of image-adaptive learning can be extended to other domains, particularly where observation models are uncertain or variable. As deep learning continues to advance, integrating adaptability into network training and inference could prove invaluable in numerous fields, driving the development of more intelligent and resilient models.