Papers
Topics
Authors
Recent
Search
2000 character limit reached

Diverse super-resolution with pretrained deep hiererarchical VAEs

Published 20 May 2022 in cs.CV, cs.LG, and eess.IV | (2205.10347v4)

Abstract: We investigate the problem of producing diverse solutions to an image super-resolution problem. From a probabilistic perspective, this can be done by sampling from the posterior distribution of an inverse problem, which requires the definition of a prior distribution on the high-resolution images. In this work, we propose to use a pretrained hierarchical variational autoencoder (HVAE) as a prior. We train a lightweight stochastic encoder to encode low-resolution images in the latent space of a pretrained HVAE. At inference, we combine the low-resolution encoder and the pretrained generative model to super-resolve an image. We demonstrate on the task of face super-resolution that our method provides an advantageous trade-off between the computational efficiency of conditional normalizing flows techniques and the sample quality of diffusion based methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. A decoder suffices for query-adaptive variational inference. In The 39th Conference on Uncertainty in Artificial Intelligence, 2023.
  2. Y. Bahat and T. Michaeli. Explorable super resolution. In the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2716–2725, 2020.
  3. Glean: Generative latent bank for large-factor image super-resolution. In IEEE/CVF conference on computer vision and pattern recognition, pages 14245–14254, 2021.
  4. R. Child. Very deep vaes generalize autoregressive models and can outperform them on images. arXiv preprint arXiv:12011.10650, 2021.
  5. Image super-resolution with deep variational autoencoders, 2022.
  6. Ilvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.02938, 2021.
  7. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations, 2022.
  8. Score-guided intermediate level optimization: Fast langevin mixing for inverse problems. In International Conference on Machine Learning, pages 4722–4753. PMLR, 2022.
  9. Intermediate layer optimization for inverse problems using deep generative models. In International Conference on Machine Learning, pages 2421–2432. PMLR, 2021.
  10. K. Deck and T. Bischoff. Easing color shifts in score-based diffusion models. arXiv preprint arXiv:2306.15832, 2023.
  11. Super-resolution variational auto-encoders. arXiv preprint arXiv:2006.05218, 2020.
  12. Solving inverse problems by joint posterior maximization with autoencoding prior, 2021.
  13. Conditional image generation by conditioning variational auto-encoders. In International Conference on Learning Representations, 2022.
  14. Hierarchical vaes know what they don’t know. In International Conference on Machine Learning, pages 4117–4128. PMLR, 2021.
  15. Efficient-vdvae: Less is more, 2022.
  16. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  17. L. Jiang. Image super-resolution via iterative refinement. https://github.com/Janspiry/Image-Super-Resolution-via-Iterative-Refinement, 2022.
  18. Scaling up gans for text-to-image synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10124–10134, 2023.
  19. A style-based generator architecture for generative adversarial networks. In IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  20. Denoising diffusion restoration models. arXiv preprint arXiv:2201.11793, 2022.
  21. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
  22. D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  23. Image super-resolution: A comprehensive review, recent trends, challenges and applications. Information Fusion, 2022.
  24. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 479:47–59, 2022.
  25. Hierarchical conditional flow: A unified framework for image super-resolution and image rescaling. In IEEE/CVF International Conference on Computer Vision, pages 4076–4085, 2021.
  26. Srflow: Learning the super-resolution space with normalizing flow. In European conference on computer vision, pages 715–732. Springer, 2020.
  27. Ntire 2021 learning the super-resolution space challenge. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 596–612, 2021.
  28. Ntire 2022 challenge on learning the super-resolution space. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 786–797, 2022.
  29. Bayesian image reconstruction using deep generative models. arXiv preprint arXiv:2012.04567, 2020.
  30. P.-A. Mattei and J. Frellsen. Leveraging the exact likelihood of deep latent variable models. Advances in Neural Information Processing Systems, 31, 2018.
  31. Pulse: Self-supervised photo upsampling via latent space exploration of generative models. In IEEE/CVF conference on computer vision and pattern recognition, pages 2437–2445, 2020.
  32. No-reference image quality assessment in the spatial domain. IEEE Transactions on image processing, 21(12):4695–4708, 2012.
  33. Exploiting deep generative prior for versatile image restoration and manipulation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7474–7489, 2021.
  34. Y. Poirier-Ginter and J.-F. Lalonde. Robust unsupervised stylegan image restoration. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22292–22301, 2023.
  35. Inverse problem regularization with hierarchical variational autoencoders. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 22894–22905, 2023.
  36. Encoding in style: a stylegan encoder for image-to-image translation. In IEEE/CVF conference on computer vision and pattern recognition, pages 2287–2296, 2021.
  37. High-resolution image synthesis with latent diffusion models. In IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  38. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636, 2021.
  39. How to train deep variational autoencoders and probabilistic ladder networks. In Advances in Neural Information Processing Systems, volume 29, 2016.
  40. Pseudoinverse-Guided Diffusion Models for Inverse Problems. In (ICLR) International Conference on Learning Representations, 2023.
  41. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
  42. A. Vahdat and J. Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03898, 2020.
  43. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004.
  44. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018.
  45. Vspsr: Explorable super-resolution via variational sparse representation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 373–381, 2021.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up