Unsupervised Deep Image Prior

• Unsupervised Deep Image Prior is an approach that exploits the inherent bias of untrained CNNs to capture low-level image statistics for restoration and decomposition.
• It uses a randomly initialized encoder-decoder network to fit noisy images, achieving high-quality results via early stopping before overfitting noise.
• Coupled DIP frameworks, like Double-DIP, further decompose images into distinct layers, enabling applications in segmentation, dehazing, and watermark removal.

Unsupervised Deep Image Prior (DIP) is a paradigm in computational imaging which leverages the architectural inductive bias of convolutional networks to perform image restoration, layer decomposition, and inverse problems, all without external training data. Unlike supervised deep learning approaches requiring extensive datasets and explicit regularization, the unsupervised DIP framework exploits the intrinsic statistical preferences of randomly initialized CNNs, enabling effective modeling and restoration from a single noisy, corrupted, or composite image. The approach generalizes to coupled networks for tasks such as image decomposition, segmentation, and unsupervised layer separation.

1. The Deep Image Prior Hypothesis

The foundational hypothesis of Deep Image Prior is that an untrained, randomly initialized convolutional generator network—typically an encoder–decoder with skip connections and down-up-sampling operations—sufficiently encodes the low-level statistics of a single natural image. Training such a network to map fixed noise vectors to the target image, it rapidly fits the dominant image content, capturing prominent patch recurrence and local smoothness well before overfitting spurious noise or artifacts. This early fit, due to the multi-scale and translation-invariant structure of convolutional networks, is exploited to recover high-fidelity reconstructions in classical inverse problems (denoising, inpainting, super-resolution) by simply optimizing network weights on the available, possibly corrupted, observation and stopping the optimization before noise fitting occurs (Gandelsman et al., 2018).

2. Mathematical Formulation

For a single image $x \in \mathbb{R}^{H \times W \times 3}$ and a generator network $$G_\theta: \mathbb{R}^d \to \mathbb{R}^{H\times W\times 3}$$ parametrized by weights $\theta$, one fixes a random input code $z\in\mathbb{R}^d$ and solves: $\min_{\theta} L(\theta) = \|G_\theta(z) - x\|_2^2.$ In this unsupervised setting, as gradient-based optimization proceeds, $G_\theta(z)$ first approximates the structural content in $x$ (low-entropy recurring patches) and, over longer training, starts reproducing noise. Early stopping empirically yields denoised or completed images with minimal or no explicit regularization.

For inverse problems with measurement operator $A$ and observed data $y$, the data-fidelity term is $\|A\,G_\theta(z) - y\|_2^2$, possibly augmented by explicit regularization such as total variation, but the essential prior remains encoded in the CNN architecture (Barbano et al., 2021, Cheng et al., 2023, Gandelsman et al., 2018).

3. Double-DIP: Unsupervised Layer Decomposition

The Double-DIP framework generalizes the classic DIP to unsupervised separation of an observed image $I$ into two or more latent layers, covering tasks such as segmentation, reflection/transparency separation, watermark removal, and dehazing. The layer composition at pixel $x$ is written as: $I(x) \approx m(x) y_1(x) + (1-m(x)) y_2(x),$ with $m(x)$ a soft mask and $y_i(x)$ the intensity at $x$ for layer $i$. Three DIPs are instantiated:

• $G_1(z_1) = y_1$, $G_2(z_2) = y_2$ (image layers)
• $G_m(z_m) = m$ (mask, passed through a sigmoid for [0,1] range)

The composite loss is: $$\text{Loss} = \text{Loss}_{\text{Reconst}} + \alpha\,\text{Loss}_{\text{Excl}} + \beta\,\text{Loss}_{\text{Reg}}$$ with

• $$\text{Loss}_{\text{Reconst}} = \sum_x \|m(x) y_1(x) + (1-m(x)) y_2(x) - I(x)\|_2^2$$
• $$\text{Loss}_{\text{Excl}} = \sum_x \|\nabla y_1(x) \odot \nabla y_2(x)\|_1$$ (exclusion, penalizing shared gradients across layers)
• $\text{Loss}_{\text{Reg}}$ task-specific, e.g., binarizing the mask for segmentation, smoothness on airlight/transmission for dehazing, or bounding box for watermark removal

This coupled optimization is executed entirely unsupervised, fitting all generator weights from scratch to the observed $I$ (and any auxiliary constraints) via gradient descent (Gandelsman et al., 2018).

4. Architectural Design and Training Protocol

Each generator in (Double-)DIP is an encoder–decoder "hourglass" network with 5 scales, 3×3 convolutions, LeakyReLU nonlinearities, and skip connections reminiscent of the U-Net architecture. Downsampling is implemented via strided convolution, upsampling via nearest-neighbor or learned upsampling. Channel dimensions typically follow 64→128→256→512→512 (bottleneck), and the architecture is fully randomly initialized per input.

The training procedure involves:

1. Initializing all generator weights $\theta_i$ randomly.
2. Sampling fixed input noises $z_1, z_2, z_m$.
3. Optionally adding Gaussian perturbations to $z_i$ for stabilization.
4. Computing forward passes to obtain layer outputs and mask.
5. Computing each loss term and the total loss.
6. Updating all weights with Adam.
7. (Optional) Early stopping based on loss behavior or post-processing masks with guided filtering (Gandelsman et al., 2018).

5. Theoretical Rationale for Coupling

A single DIP is empirically biased toward outputs with low local patch entropy (high internal recurrence). When two (or more) DIPs are jointly optimized so that their sum matches a mixed observation $I$, the coupled networks preferentially partition the image such that each network covers the regions it can most parsimoniously represent, guided by internal patch statistics. This results in automatic separation, since the entropy of a mixture $H(X+Y)$ exceeds that of each component ($H(X+Y) \geq \max\{H(X), H(Y)\}$ per Cover–Thomas). The exclusion loss further reduces redundant structure across layers, and in video settings, temporal sharing resolves assignment ambiguities without supervision (Gandelsman et al., 2018).

6. Empirical Performance Across Tasks

Experimental results demonstrate the efficacy of the unsupervised Double-DIP approach:

• Foreground–background segmentation: Produces plausible masks rivaling unsupervised graph-cut methods.
• Video segmentation and transparency separation: Layer consistency enforced across frames; capable of dynamic and static separation.
• Watermark removal: Achieves clean background recovery and precise mask estimation, outperforming prior unsupervised methods in few-shot scenarios.
• Image dehazing: Unlike priors assuming spatially uniform airlight, Double-DIP recovers non-uniform airlight and transmission maps; on the O-HAZE benchmark, achieves PSNR 18.82 dB, second among classical and learning-based unsupervised dehazing algorithms, with improved color fidelity and fewer artifacts (Gandelsman et al., 2018).

7. Synthesis and Outlook

Unsupervised Deep Image Prior establishes that the convolutional network architecture itself imposes a strong and effective prior on natural image statistics, sufficient for a broad range of restoration and layer-separation tasks without any data-driven learning. Double-DIP extends this by coupling multiple DIPs, exploiting the statistical simplicity of each layer relative to their mixture to achieve self-organized factorization of the input into interpretable components. The paradigm remains fully unsupervised—optimized per-image/video—requiring no pretraining or labeled datasets, and is widely applicable across segmentation, dehazing, watermark removal, and transparency separation tasks. Its success motivates further development of unsupervised, architecture-driven approaches to highly underdetermined imaging problems (Gandelsman et al., 2018).