Optimal Eye Surgeon: Finding Image Priors through Sparse Generators at Initialization

Abstract: We introduce Optimal Eye Surgeon (OES), a framework for pruning and training deep image generator networks. Typically, untrained deep convolutional networks, which include image sampling operations, serve as effective image priors (Ulyanov et al., 2018). However, they tend to overfit to noise in image restoration tasks due to being overparameterized. OES addresses this by adaptively pruning networks at random initialization to a level of underparameterization. This process effectively captures low-frequency image components even without training, by just masking. When trained to fit noisy images, these pruned subnetworks, which we term Sparse-DIP, resist overfitting to noise. This benefit arises from underparameterization and the regularization effect of masking, constraining them in the manifold of image priors. We demonstrate that subnetworks pruned through OES surpass other leading pruning methods, such as the Lottery Ticket Hypothesis, which is known to be suboptimal for image recovery tasks (Wu et al., 2023). Our extensive experiments demonstrate the transferability of OES-masks and the characteristics of sparse-subnetworks for image generation. Code is available at https://github.com/Avra98/Optimal-Eye-Surgeon.git.

• The paper introduces a pruning method, Sparse-DIP, at initialization to reduce overfitting in deep image priors.
• The paper demonstrates that adaptive pruning masks transfer across datasets, significantly improving image recovery outcomes.
• The paper shows that Sparse-DIP outperforms existing methods like LTH, GraSP, and Synflow in preserving image fidelity.

An Analytical Overview of "Optimal Eye Surgeon: Finding Image Priors Through Sparse Generators at Initialization"

The study presents a compelling framework known as Optimal Eye Surgeon (OES) which targets the persistent problem of overfitting in deep image generator networks, specifically focusing on deep convolutional neural networks used for image restoration tasks. The core issue addressed by this research is the susceptibility of untrained deep convolutional networks, utilized as image priors, to overfit noise due to their overparameterized nature.

Methodological Innovations

OES innovatively tackles this overfitting challenge by introducing a novel pruning approach at the random initialization phase, termed as Sparse-DIP. The pruning is adaptive and achieves a state of underparameterization, allowing the pruned network to effectively capture low-frequency image components even prior to undergoing training. Through automatic masking at initialization, these networks inherently provide a form of regularization, thus minimizing the risk of overfitting to noise when trained on corrupted images.

The quintessential differentiation of OES from other contemporary pruning techniques, such as the Lottery Ticket Hypothesis (LTH), lies in its adaptability and performance in image recovery tasks. Unlike LTH, which does not suitably address the overfitting aspect in such unsupervised learning scenarios, OES is validated through its robustness against overfitting and superior image recovery capabilities via exhaustive experimental evaluations.

Experimental Insights and Contributions

The extensive experimental analysis conducted within this study offers numerous key insights:

1. Pruned Network Efficiency: The study showcases that the pruned networks achieved through OES, termed Sparse-DIP, are highly effective in reducing overfitting when compared to dense networks like the deep image prior, without loss of critical image representation fidelity.
2. Transferability of Pruning Masks: A notable finding is the generalizability and transferability of the pruning masks determined through OES. These masks are not only effective on the source image but can also be generalized across different images and datasets, which was extensively demonstrated in both inter-dataset and cross-dataset scenarios.
3. Comparison with Existing Methods: Sparse-DIP networks not only outperform well-known pruning methods like GraSP and Synflow at initialization but also exhibit less pronounced overfitting compared to LTH-based subnetworks when trained with image noise.
4. Architecture Insights: The study provides an analytical dissection of the architectural sparsity induced by OES, showing its efficiency in pruning non-essential layers, particularly within the encoder segment of the Unet architecture.
5. Masking at Initialization: The fundamental analysis of masking at initialization unveils that masks not only induce a strong image prior at zero-training state but also maintain substantial fidelity for image reconstruction throughout the training.

Implications and Future Directions

This research holds significant implications for the practical applications in image restoration and potentially enhances the theoretical understanding of deep network parameterization. The robust performance of Sparse-DIPs signals a paradigm where training efficiency and network performance can be simultaneously enhanced. Practically, the portability of these masks can streamline computational demands across various image reconstruction tasks and facilitate accelerated convergence in machine learning pipelines.

Theoretically, this research opens venues to explore deeper into the phenomena of strong lottery tickets within the field of image processing tasks, and further stems into exploring sparse networks' potential in emerging diffusion models and generative frameworks.

In conclusion, while addressing a fundamental limitation in deep image processing networks, the study paves the way for exploring sparse computational paradigms, thereby contributing a significant methodological advancement with OES. Future work will likely focus on extending this framework towards even broader domains and complex image distortion challenges, enhancing both computational efficiency and reconstruction accuracy.