External Patch-Based Image Restoration Using Importance Sampling

Abstract: This paper introduces a new approach to patch-based image restoration based on external datasets and importance sampling. The Minimum Mean Squared Error (MMSE) estimate of the image patches, the computation of which requires solving a multidimensional (typically intractable) integral, is approximated using samples from an external dataset. The new method, which can be interpreted as a generalization of the external non-local means (NLM), uses self-normalized importance sampling to efficiently approximate the MMSE estimates. The use of self-normalized importance sampling endows the proposed method with great flexibility, namely regarding the statistical properties of the measurement noise. The effectiveness of the proposed method is shown in a series of experiments using both generic large-scale and class-specific external datasets.

• The paper introduces an SNIS-based image restoration method that uses external patches to approximate MMSE estimates under diverse noise conditions.
• It leverages self-normalized importance sampling with a Hellinger distance criterion to optimize weights for clean patch selection and efficient computation.
• Experimental results show significant PSNR improvements on both generic and class-specific datasets compared to established methods like BM3D and NL-PCA.

"External Patch-Based Image Restoration Using Importance Sampling" (1807.03018)

Introduction

Patch-based image restoration is crucial in addressing the degradation induced by imaging processes. The paper "External Patch-Based Image Restoration Using Importance Sampling" presents a novel approach leveraging external datasets and importance sampling to approximate the Minimum Mean Squared Error (MMSE) estimates of image patches. This method improves upon traditional techniques, particularly in its flexibility regarding the statistical properties of measurement noise. The proposed approach is validated through experiments demonstrating its efficacy across generic and class-specific scenarios.

Methodology

External Datasets and MMSE Estimation

The restoration process considers image degradation models where observed pixel values $\mathbf{y}$ serve as noisy linear projections of original pixel values $\mathbf{x}$. This is expressed in Equation \eqref{eq:LIP}, with noise modeled as additive Gaussian or potentially non-linear and Poissonian. The MMSE estimates intrinsically rely on solving a multidimensional integral, which is traditionally intractable for complex noise models.

The paper innovates by using importance sampling, specifically self-normalized importance sampling (SNIS), to approximate this estimation. SNIS allows sampling from a proposal distribution rather than the target distribution, facilitating efficient computations even with unknown normalization constants.

Importance Sampling for Image Restoration

Given a noisy patch $\mathbf{y}$, the goal is to recover the central pixel of its clean counterpart $\mathbf{x}_c$. Importance sampling bridges this by utilizing external clean patches $\mathbf{x}_1, \ldots, \mathbf{x}_N$ as surrogates for the prior distribution $p_X$. By optimizing the sampling weights $\alpha_k(\mathbf{y})$ for each clustered patch subset—according to a Hellinger distance criterion—restoration becomes constrained not by specific noise assumptions but rather by maximized statistical similarity.

Figure 1: Examples of recovering images with missing pixels and contaminated by noise using SNIS. (a) Degraded image: 70\% randomly available pixels with additive i.i.d. Gaussian noise with standard deviation 15. (b) Restored image (PSNR=33.26\,dB). (c) Degraded image with peak of 5 contaminated by Poisson noise and observation of 80\% of pixels observed (PSNR=4.92 \,dB). (d) Restored image (PSNR=20.18\,dB)

Experimental Results

Class-Specific Datasets

For datasets where image class is identifiable (e.g., face, text, or handwritten digits), SNIS showcases remarkable performance enhancement over generic competitors. The method distinguishes by its adaptability to variance in noise type and degrees of missing data. This class-specific superiority is quantified by PSNR improvements over existing methods such as EPLL and BM3D under Gaussian noise, and against NL-PCA and VST+BM3D for Poissonian noise scenarios.

Large Scale Generic Images

The SNIS advantage also extends to large-scale datasets where computational efficiency is critical. With $2\times10^6$ patches from a large dataset, SNIS demonstrates enhanced algorithmic speed—mitigating the historical bottleneck associated with patch comparison in external NLM. Even when sampling is random or guided by Bernoulli selection, SNIS maintains competitive PSNR scores against exhaustive MMSE estimates.

Conclusion

The proposed SNIS method revolutionizes image restoration tasks by marrying Monte Carlo sampling techniques with Bayesian inference models in the presence of external datasets. It opens pathways for non-linear noise model applications and redefines scalability in dataset utilization. Future research may further explore adaptive clustering strategies, fine-tuning proposal distributions, and expanding SNIS utility across varied multi-modal imaging applications.