Papers
Topics
Authors
Recent
Search
2000 character limit reached

Diffusion Models, Image Super-Resolution And Everything: A Survey

Published 1 Jan 2024 in cs.CV, cs.AI, cs.LG, and cs.MM | (2401.00736v3)

Abstract: Diffusion Models (DMs) have disrupted the image Super-Resolution (SR) field and further closed the gap between image quality and human perceptual preferences. They are easy to train and can produce very high-quality samples that exceed the realism of those produced by previous generative methods. Despite their promising results, they also come with new challenges that need further research: high computational demands, comparability, lack of explainability, color shifts, and more. Unfortunately, entry into this field is overwhelming because of the abundance of publications. To address this, we provide a unified recount of the theoretical foundations underlying DMs applied to image SR and offer a detailed analysis that underscores the unique characteristics and methodologies within this domain, distinct from broader existing reviews in the field. This survey articulates a cohesive understanding of DM principles and explores current research avenues, including alternative input domains, conditioning techniques, guidance mechanisms, corruption spaces, and zero-shot learning approaches. By offering a detailed examination of the evolution and current trends in image SR through the lens of DMs, this survey sheds light on the existing challenges and charts potential future directions, aiming to inspire further innovation in this rapidly advancing area.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (159)
  1. W. Sun and Z. Chen, “Learned image downscaling for upscaling using content adaptive resampler,” IEEE TIP, vol. 29, pp. 4027–4040, 2020.
  2. R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using sparse-representations,” in International conference on curves and surfaces.   Springer, 2010, pp. 711–730.
  3. D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in ICCV, vol. 2.   IEEE, 2001, pp. 416–423.
  4. D. Valsesia and E. Magli, “Permutation invariance and uncertainty in multitemporal image super-resolution,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–12, 2021.
  5. S. M. A. Bashir, Y. Wang, M. Khan, and Y. Niu, “A comprehensive review of deep learning-based single image super-resolution,” PeerJ Computer Science, vol. 7, p. e621, 2021.
  6. J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” NeurIPS, vol. 33, pp. 6840–6851, 2020.
  7. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” Communications of the ACM, vol. 63, no. 11, pp. 139–144, 2020.
  8. Y. Song and S. Ermon, “Generative modeling by estimating gradients of the data distribution,” NeurIPS, vol. 32, 2019.
  9. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” arXiv preprint arXiv:2011.13456, 2020.
  10. D. Lee, C. Kim, S. Kim, M. Cho, and W.-S. Han, “Autoregressive image generation using residual quantization,” in CVPR, 2022, pp. 11 523–11 532.
  11. P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high-resolution image synthesis,” in CVPR, 2021, pp. 12 873–12 883.
  12. B. Guo, X. Zhang, H. Wu, Y. Wang, Y. Zhang, and Y.-F. Wang, “Lar-sr: A local autoregressive model for image super-resolution,” in CVPR, 2022, pp. 1909–1918.
  13. S. Frolov, T. Hinz, F. Raue, J. Hees, and A. Dengel, “Adversarial text-to-image synthesis: A review,” Neural Networks, vol. 144, pp. 187–209, 2021.
  14. B. B. Moser, F. Raue, S. Frolov, S. Palacio, J. Hees, and A. Dengel, “Hitchhiker’s guide to super-resolution: Introduction and recent advances,” IEEE TPAMI, pp. 1–21, 2023.
  15. E. Agustsson and R. Timofte, “Ntire 2017 challenge on single image super-resolution: Dataset and study,” in CVPRW, 2017, pp. 1122–1131.
  16. ——, “Ntire 2017 challenge on single image super-resolution: Dataset and study,” in CVPRW, July 2017.
  17. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in CVPR, 2009, pp. 248–255.
  18. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, “The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results,” http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html, 2012.
  19. T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” arXiv preprint arXiv:1710.10196, 2017.
  20. T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in CVPR, 2019, pp. 4401–4410.
  21. K. I. Kim and Y. Kwon, “Single-image super-resolution using sparse regression and natural image prior,” IEEE TPAMI, vol. 32, no. 6, pp. 1127–1133, 2010.
  22. G. Freedman and R. Fattal, “Image and video upscaling from local self-examples,” ACM Trans. Graph., vol. 30, no. 2, apr 2011.
  23. J. Sun, Z. Xu, and H.-Y. Shum, “Image super-resolution using gradient profile prior,” in CVPR, 2008, pp. 1–8.
  24. H. Chang, D.-Y. Yeung, and Y. Xiong, “Super-resolution through neighbor embedding,” in Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., vol. 1.   IEEE, 2004, pp. I–I.
  25. W. Freeman, T. Jones, and E. Pasztor, “Example-based super-resolution,” IEEE Computer Graphics and Applications, vol. 22, no. 2, pp. 56–65, 2002.
  26. R. Keys, “Cubic convolution interpolation for digital image processing,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 29, no. 6, pp. 1153–1160, 1981.
  27. M. Irani and S. Peleg, “Improving resolution by image registration,” CVGIP: Graphical Models and Image Processing, vol. 53, no. 3, pp. 231–239, 1991.
  28. J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE TIP, vol. 19, no. 11, pp. 2861–2873, 2010.
  29. C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE TPAMI, vol. 38, no. 2, pp. 295–307, 2015.
  30. C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in ECCV.   Springer, 2016, pp. 391–407.
  31. W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in CVPR, 2016, pp. 1874–1883.
  32. C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang et al., “Photo-realistic single image super-resolution using a generative adversarial network,” in CVPR, 2017.
  33. G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in CVPR, 2017, pp. 4700–4708.
  34. T. Tong, G. Li, X. Liu, and Q. Gao, “Image super-resolution using dense skip connections,” in ICCV, 2017, pp. 4799–4807.
  35. X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “Esrgan: Enhanced super-resolution generative adversarial networks,” in Proceedings of the European conference on computer vision (ECCV) workshops, 2018.
  36. J. Kim, J. K. Lee, and K. M. Lee, “Deeply-recursive convolutional network for image super-resolution,” in CVPR, 2016, pp. 1637–1645.
  37. Y. Tai, J. Yang, and X. Liu, “Image super-resolution via deep recursive residual network,” in CVPR, 2017, pp. 3147–3155.
  38. N. Ahn, B. Kang, and K.-A. Sohn, “Fast, accurate, and lightweight super-resolution with cascading residual network,” in ECCV, 2018, pp. 252–268.
  39. A. Lugmayr, M. Danelljan, L. Van Gool, and R. Timofte, “Srflow: Learning the super-resolution space with normalizing flow,” in ECCV.   Springer, 2020.
  40. C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, “Image super-resolution via iterative refinement,” IEEE TPAMI, vol. 45, no. 4, pp. 4713–4726, 2023.
  41. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  42. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Communications of the ACM, vol. 60, no. 6, pp. 84–90, 2017.
  43. L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui, and M.-H. Yang, “Diffusion models: A comprehensive survey of methods and applications,” ACM Computing Surveys, vol. 56, no. 4, pp. 1–39, 2023.
  44. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in ICML.   PMLR, 2015, pp. 2256–2265.
  45. G. Parisi, “Correlation functions and computer simulations,” Nuclear Physics B, vol. 180, no. 3, pp. 378–384, 1981.
  46. P. Vincent, “A connection between score matching and denoising autoencoders,” Neural computation, vol. 23, no. 7, pp. 1661–1674, 2011.
  47. A. Hyvärinen and P. Dayan, “Estimation of non-normalized statistical models by score matching.” Journal of Machine Learning Research, vol. 6, no. 4, 2005.
  48. Y. Song, S. Garg, J. Shi, and S. Ermon, “Sliced score matching: A scalable approach to density and score estimation,” in Uncertainty in Artificial Intelligence.   PMLR, 2020, pp. 574–584.
  49. B. D. Anderson, “Reverse-time diffusion equation models,” Stochastic Processes and their Applications, vol. 12, no. 3, pp. 313–326, 1982.
  50. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” NeurIPS, vol. 27, 2014.
  51. D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013.
  52. D. Rezende and S. Mohamed, “Variational inference with normalizing flows,” in ICML.   PMLR, 2015, pp. 1530–1538.
  53. Q. Zhang and Y. Chen, “Diffusion normalizing flow,” in NeurIPS, M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34.   Curran Associates, Inc., 2021, pp. 16 280–16 291.
  54. G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan, “Normalizing flows for probabilistic modeling and inference,” The Journal of Machine Learning Research, vol. 22, no. 1, pp. 2617–2680, 2021.
  55. “Oxford vggface implementation using keras functional framework v2+,” https://github.com/rcmalli/keras-vggface.
  56. D. Watson, J. Ho, M. Norouzi, and W. Chan, “Learning to efficiently sample from diffusion probabilistic models,” arXiv preprint arXiv:2106.03802, 2021.
  57. D. Watson, W. Chan, J. Ho, and M. Norouzi, “Learning fast samplers for diffusion models by differentiating through sample quality,” arXiv preprint arXiv:2202.05830, 2022.
  58. C. Meng, R. Rombach, R. Gao, D. Kingma, S. Ermon, J. Ho, and T. Salimans, “On distillation of guided diffusion models,” in CVPR, 2023, pp. 14 297–14 306.
  59. E. Luhman and T. Luhman, “Knowledge distillation in iterative generative models for improved sampling speed,” arXiv preprint arXiv:2101.02388, 2021.
  60. T. Salimans and J. Ho, “Progressive distillation for fast sampling of diffusion models,” arXiv preprint arXiv:2202.00512, 2022.
  61. Z. Xiao, K. Kreis, and A. Vahdat, “Tackling the generative learning trilemma with denoising diffusion gans,” arXiv preprint arXiv:2112.07804, 2021.
  62. Z. Lyu, X. Xu, C. Yang, D. Lin, and B. Dai, “Accelerating diffusion models via early stop of the diffusion process,” arXiv preprint arXiv:2205.12524, 2022.
  63. L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in CVPR, 2023, pp. 3836–3847.
  64. J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” arXiv preprint arXiv:2010.02502, 2020.
  65. A. Jolicoeur-Martineau, K. Li, R. Piché-Taillefer, T. Kachman, and I. Mitliagkas, “Gotta go fast when generating data with score-based models,” arXiv preprint arXiv:2105.14080, 2021.
  66. J. Ho, E. Lohn, and P. Abbeel, “Compression with flows via local bits-back coding,” NeurIPS, vol. 32, 2019.
  67. Z. Dai, Z. Yang, F. Yang, W. W. Cohen, and R. R. Salakhutdinov, “Good semi-supervised learning that requires a bad gan,” NeurIPS, vol. 30, 2017.
  68. Y. Song, C. Durkan, I. Murray, and S. Ermon, “Maximum likelihood training of score-based diffusion models,” NeurIPS, vol. 34, pp. 1415–1428, 2021.
  69. D. Kingma, T. Salimans, B. Poole, and J. Ho, “Variational diffusion models,” NeurIPS, vol. 34, pp. 21 696–21 707, 2021.
  70. A. Q. Nichol and P. Dhariwal, “Improved denoising diffusion probabilistic models,” in ICML.   PMLR, 2021, pp. 8162–8171.
  71. H. Li, Y. Yang, M. Chang, S. Chen, H. Feng, Z. Xu, Q. Li, and Y. Chen, “Srdiff: Single image super-resolution with diffusion probabilistic models,” Neurocomputing, vol. 479, pp. 47–59, 2022.
  72. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in CVPR, 2022, pp. 10 684–10 695.
  73. A. Vahdat, K. Kreis, and J. Kautz, “Score-based generative modeling in latent space,” NeurIPS, vol. 34, pp. 11 287–11 302, 2021.
  74. Z. Luo, F. K. Gustafsson, Z. Zhao, J. Sjölund, and T. B. Schön, “Refusion: Enabling large-size realistic image restoration with latent-space diffusion models,” in CVPR, 2023, pp. 1680–1691.
  75. ——, “Image restoration with mean-reverting stochastic differential equations,” arXiv preprint arXiv:2301.11699, 2023.
  76. L. Chen, X. Chu, X. Zhang, and J. Sun, “Simple baselines for image restoration,” in ECCV.   Springer, 2022, pp. 17–33.
  77. Z. Chen, Y. Zhang, D. Liu, B. Xia, J. Gu, L. Kong, and X. Yuan, “Hierarchical integration diffusion model for realistic image deblurring,” arXiv preprint arXiv:2305.12966, 2023.
  78. S. Menon, A. Damian, S. Hu, N. Ravi, and C. Rudin, “Pulse: Self-supervised photo upsampling via latent space exploration of generative models,” in CVPR, 2020, pp. 2437–2445.
  79. Y. Chen, Y. Tai, X. Liu, C. Shen, and J. Yang, “Fsrnet: End-to-end learning face super-resolution with facial priors,” in CVPR, 2018, pp. 2492–2501.
  80. B. Moser, S. Frolov, F. Raue, S. Palacio, and A. Dengel, “Waving goodbye to low-res: A diffusion-wavelet approach for image super-resolution,” 2023.
  81. B. B. Moser, S. Frolov, F. Raue, S. Palacio, and A. Dengel, “Dwa: Differential wavelet amplifier for image super-resolution,” in Artificial Neural Networks and Machine Learning – ICANN 2023, L. Iliadis, A. Papaleonidas, P. Angelov, and C. Jayne, Eds.   Cham: Springer Nature Switzerland, 2023, pp. 232–243.
  82. T. Guo, H. Seyed Mousavi, T. Huu Vu, and V. Monga, “Deep wavelet prediction for image super-resolution,” in CVPRW, 2017, pp. 104–113.
  83. Y. Huang, J. Huang, J. Liu, Y. Dong, J. Lv, and S. Chen, “Wavedm: Wavelet-based diffusion models for image restoration,” arXiv preprint arXiv:2305.13819, 2023.
  84. F. Guth, S. Coste, V. De Bortoli, and S. Mallat, “Wavelet score-based generative modeling,” NeurIPS, vol. 35, pp. 478–491, 2022.
  85. S. Shang, Z. Shan, G. Liu, and J. Zhang, “Resdiff: Combining cnn and diffusion model for image super-resolution,” arXiv preprint arXiv:2303.08714, 2023.
  86. J. Whang, M. Delbracio, H. Talebi, C. Saharia, A. G. Dimakis, and P. Milanfar, “Deblurring via stochastic refinement,” in CVPR, 2022, pp. 16 293–16 303.
  87. Z. Yue, J. Wang, and C. C. Loy, “Resshift: Efficient diffusion model for image super-resolution by residual shifting,” 2023.
  88. B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in CVPRW, 2017, pp. 136–144.
  89. S. H. Park, Y. S. Moon, and N. I. Cho, “Flexible style image super-resolution using conditional objective,” IEEE Access, vol. 10, pp. 9774–9792, 2022.
  90. W. Zhang, Y. Liu, C. Dong, and Y. Qiao, “Ranksrgan: Generative adversarial networks with ranker for image super-resolution,” in CVPR, 2019, pp. 3096–3105.
  91. A. Lugmayr, M. Danelljan, L. V. Gool, and R. Timofte, “Srflow: Learning the super-resolution space with normalizing flow,” in ECCV.   Springer, 2020, pp. 715–732.
  92. C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans et al., “Photorealistic text-to-image diffusion models with deep language understanding,” NeurIPS, vol. 35, pp. 36 479–36 494, 2022.
  93. J. Choi, S. Kim, Y. Jeong, Y. Gwon, and S. Yoon, “Ilvr: Conditioning method for denoising diffusion probabilistic models,” 2021.
  94. A. Niu, K. Zhang, T. X. Pham, J. Sun, Y. Zhu, I. S. Kweon, and Y. Zhang, “Cdpmsr: Conditional diffusion probabilistic models for single image super-resolution,” 2023.
  95. K. Pandey, A. Mukherjee, P. Rai, and A. Kumar, “Diffusevae: Efficient, controllable and high-fidelity generation from low-dimensional latents,” arXiv preprint arXiv:2201.00308, 2022.
  96. J. Wang, Z. Yue, S. Zhou, K. C. Chan, and C. C. Loy, “Exploiting diffusion prior for real-world image super-resolution,” arXiv preprint arXiv:2305.07015, 2023.
  97. S. Zhou, K. Chan, C. Li, and C. C. Loy, “Towards robust blind face restoration with codebook lookup transformer,” NeurIPS, vol. 35, pp. 30 599–30 611, 2022.
  98. T. Yang, P. Ren, X. Xie, and L. Zhang, “Pixel-aware stable diffusion for realistic image super-resolution and personalized stylization,” arXiv preprint arXiv:2308.14469, 2023.
  99. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in ICML.   PMLR, 2021, pp. 8748–8763.
  100. K. Zhang, J. Liang, L. Van Gool, and R. Timofte, “Designing a practical degradation model for deep blind image super-resolution,” in ICCV, 2021, pp. 4791–4800.
  101. X. Wang, L. Xie, C. Dong, and Y. Shan, “Real-esrgan: Training real-world blind super-resolution with pure synthetic data,” in CVPR, 2021, pp. 1905–1914.
  102. J. Liang, H. Zeng, and L. Zhang, “Details or artifacts: A locally discriminative learning approach to realistic image super-resolution,” in CVPR, 2022, pp. 5657–5666.
  103. C. Chen, X. Shi, Y. Qin, X. Li, X. Han, T. Yang, and S. Guo, “Real-world blind super-resolution via feature matching with implicit high-resolution priors,” in Proceedings of the 30th ACM International Conference on Multimedia, ser. MM ’22.   New York, NY, USA: Association for Computing Machinery, 2022, p. 1329–1338.
  104. J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “Swinir: Image restoration using swin transformer,” in CVPR, 2021, pp. 1833–1844.
  105. J. Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans, “Cascaded diffusion models for high fidelity image generation.” J. Mach. Learn. Res., vol. 23, no. 47, pp. 1–33, 2022.
  106. P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” NeurIPS, vol. 34, pp. 8780–8794, 2021.
  107. A. Brock, J. Donahue, and K. Simonyan, “Large scale gan training for high fidelity natural image synthesis,” arXiv preprint arXiv:1809.11096, 2018.
  108. J. Ho and T. Salimans, “Classifier-free diffusion guidance,” arXiv preprint arXiv:2207.12598, 2022.
  109. C. Luo, “Understanding diffusion models: A unified perspective,” arXiv preprint arXiv:2208.11970, 2022.
  110. X. Li, Y. Ren, X. Jin, C. Lan, X. Wang, W. Zeng, X. Wang, and Z. Chen, “Diffusion models for image restoration and enhancement–a comprehensive survey,” arXiv preprint arXiv:2308.09388, 2023.
  111. B. Kawar, G. Vaksman, and M. Elad, “Snips: Solving noisy inverse problems stochastically,” NeurIPS, vol. 34, pp. 21 757–21 769, 2021.
  112. B. Kawar, M. Elad, S. Ermon, and J. Song, “Denoising diffusion restoration models,” NeurIPS, vol. 35, pp. 23 593–23 606, 2022.
  113. H. Chung, J. Kim, M. T. Mccann, M. L. Klasky, and J. C. Ye, “Diffusion posterior sampling for general noisy inverse problems,” arXiv preprint arXiv:2209.14687, 2022.
  114. Y. Wang, J. Yu, and J. Zhang, “Zero-shot image restoration using denoising diffusion null-space model,” arXiv preprint arXiv:2212.00490, 2022.
  115. B. Fei, Z. Lyu, L. Pan, J. Zhang, W. Yang, T. Luo, B. Zhang, and B. Dai, “Generative diffusion prior for unified image restoration and enhancement,” in CVPR, 2023, pp. 9935–9946.
  116. A. Shocher, N. Cohen, and M. Irani, ““zero-shot” super-resolution using deep internal learning,” in CVPR, 2018, pp. 3118–3126.
  117. A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool, “Repaint: Inpainting using denoising diffusion probabilistic models,” in CVPR, 2022, pp. 11 461–11 471.
  118. B. B. Moser, S. Frolov, F. Raue, S. Palacio, and A. Dengel, “Yoda: You only diffuse areas. an area-masked diffusion approach for image super-resolution,” arXiv preprint arXiv:2308.07977, 2023.
  119. M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in CVPR, 2021, pp. 9650–9660.
  120. H. Chung, B. Sim, and J. C. Ye, “Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction,” in CVPR, 2022, pp. 12 413–12 422.
  121. J. Schwab, S. Antholzer, and M. Haltmeier, “Deep null space learning for inverse problems: convergence analysis and rates,” Inverse Problems, vol. 35, no. 2, p. 025008, 2019.
  122. Y. Wang, Y. Hu, J. Yu, and J. Zhang, “Gan prior based null-space learning for consistent super-resolution,” in AAAI, vol. 37, no. 3, 2023, pp. 2724–2732.
  123. H. Chung, B. Sim, D. Ryu, and J. C. Ye, “Improving diffusion models for inverse problems using manifold constraints,” NeurIPS, vol. 35, pp. 25 683–25 696, 2022.
  124. J. Song, A. Vahdat, M. Mardani, and J. Kautz, “Pseudoinverse-guided diffusion models for inverse problems,” in ICLR, 2022.
  125. T. Karras, M. Aittala, T. Aila, and S. Laine, “Elucidating the design space of diffusion-based generative models,” NeurIPS, vol. 35, pp. 26 565–26 577, 2022.
  126. G. Daras, M. Delbracio, H. Talebi, A. G. Dimakis, and P. Milanfar, “Soft diffusion: Score matching for general corruptions,” arXiv preprint arXiv:2209.05442, 2022.
  127. A. Bansal, E. Borgnia, H.-M. Chu, J. S. Li, H. Kazemi, F. Huang, M. Goldblum, J. Geiping, and T. Goldstein, “Cold diffusion: Inverting arbitrary image transforms without noise,” arXiv preprint arXiv:2208.09392, 2022.
  128. G.-H. Liu, A. Vahdat, D.-A. Huang, E. A. Theodorou, W. Nie, and A. Anandkumar, “I 22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT sb: Image-to-image schrödinger bridge,” arXiv preprint arXiv:2302.05872, 2023.
  129. J. Choi, J. Lee, C. Shin, S. Kim, H. Kim, and S. Yoon, “Perception prioritized training of diffusion models,” in CVPR, 2022, pp. 11 472–11 481.
  130. B. Xia, Y. Zhang, S. Wang, Y. Wang, X. Wu, Y. Tian, W. Yang, and L. Van Gool, “Diffir: Efficient diffusion model for image restoration,” arXiv preprint arXiv:2303.09472, 2023.
  131. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” NeurIPS, vol. 30, 2017.
  132. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
  133. H. Chung, E. S. Lee, and J. C. Ye, “Mr image denoising and super-resolution using regularized reverse diffusion,” IEEE Transactions on Medical Imaging, vol. 42, no. 4, pp. 922–934, 2022.
  134. Y. Mao, L. Jiang, X. Chen, and C. Li, “Disc-diff: Disentangled conditional diffusion model for multi-contrast mri super-resolution,” arXiv preprint arXiv:2303.13933, 2023.
  135. Z. Yue and C. C. Loy, “Difface: Blind face restoration with diffused error contraction,” arXiv preprint arXiv:2212.06512, 2022.
  136. X. Wang, Y. Li, H. Zhang, and Y. Shan, “Towards real-world blind face restoration with generative facial prior,” in CVPR, 2021, pp. 9168–9178.
  137. T. Yang, P. Ren, X. Xie, and L. Zhang, “Gan prior embedded network for blind face restoration in the wild,” in CVPR, 2021, pp. 672–681.
  138. X. Qiu, C. Han, Z. Zhang, B. Li, T. Guo, and X. Nie, “Diffbfr: Bootstrapping diffusion model towards blind face restoration,” arXiv preprint arXiv:2305.04517, 2023.
  139. Z. Wang, Z. Zhang, X. Zhang, H. Zheng, M. Zhou, Y. Zhang, and Y. Wang, “Dr2: Diffusion-based robust degradation remover for blind face restoration,” in CVPR, 2023, pp. 1704–1713.
  140. X. Wang, S. López-Tapia, and A. K. Katsaggelos, “Atmospheric turbulence correction via variational deep diffusion,” in 2023 IEEE 6th International Conference on Multimedia Information Processing and Retrieval (MIPR).   IEEE, 2023, pp. 1–4.
  141. N. G. Nair, K. Mei, and V. M. Patel, “At-ddpm: Restoring faces degraded by atmospheric turbulence using denoising diffusion probabilistic models,” in WACV, 2023, pp. 3434–3443.
  142. J. Liu, Z. Yuan, Z. Pan, Y. Fu, L. Liu, and B. Lu, “Diffusion model with detail complement for super-resolution of remote sensing,” Remote Sensing, vol. 14, no. 19, 2022.
  143. A. M. Ali, B. Benjdira, A. Koubaa, W. Boulila, and W. El-Shafai, “Tesr: Two-stage approach for enhancement and super-resolution of remote sensing images,” Remote Sensing, vol. 15, no. 9, 2023.
  144. M. Xu, J. Ma, and Y. Zhu, “Dual-diffusion: Dual conditional denoising diffusion probabilistic models for blind super-resolution reconstruction in rsis,” arXiv preprint arXiv:2305.12170, 2023.
  145. D. Ganguli, D. Hernandez, L. Lovitt, A. Askell, Y. Bai, A. Chen, T. Conerly, N. Dassarma, D. Drain, N. Elhage et al., “Predictability and surprise in large generative models,” in Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022, pp. 1747–1764.
  146. N. Chen, Y. Zhang, H. Zen, R. J. Weiss, M. Norouzi, and W. Chan, “Wavegrad: Estimating gradients for waveform generation,” arXiv preprint arXiv:2009.00713, 2020.
  147. Z. Cheng, “Sampler scheduler for diffusion models,” arXiv preprint arXiv:2311.06845, 2023.
  148. T. Chen, “On the importance of noise scheduling for diffusion models,” arXiv preprint arXiv:2301.10972, 2023.
  149. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in CVPR, 2018, pp. 586–595.
  150. X. Liu, J. Van De Weijer, and A. D. Bagdanov, “Rankiqa: Learning from rankings for no-reference image quality assessment,” in ICCV, 2017, pp. 1040–1049.
  151. K. Ma, W. Liu, T. Liu, Z. Wang, and D. Tao, “dipiq: Blind image quality assessment by learning-to-rank discriminable image pairs,” IEEE TIP, vol. 26, no. 8, pp. 3951–3964, 2017.
  152. N. Ponomarenko, L. Jin, O. Ieremeiev, V. Lukin, K. Egiazarian, J. Astola, B. Vozel, K. Chehdi, M. Carli, F. Battisti et al., “Image database tid2013: Peculiarities, results and perspectives,” Signal processing: Image communication, vol. 30, pp. 57–77, 2015.
  153. J. Kim and S. Lee, “Deep learning of human visual sensitivity in image quality assessment framework,” in CVPR, 2017, pp. 1676–1684.
  154. H. Talebi and P. Milanfar, “Nima: Neural image assessment,” IEEE TIP, vol. 27, no. 8, pp. 3998–4011, 2018.
  155. S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in CVPR, 2019, pp. 6023–6032.
  156. B. Jing, G. Corso, R. Berlinghieri, and T. Jaakkola, “Subspace diffusion generative models,” in ECCV.   Springer, 2022, pp. 274–289.
  157. M. Kwon, J. Jeong, and Y. Uh, “Diffusion models already have a semantic latent space,” in ICLR, 2023.
  158. Q. Wu, Y. Liu, H. Zhao, A. Kale, T. Bui, T. Yu, Z. Lin, Y. Zhang, and S. Chang, “Uncovering the disentanglement capability in text-to-image diffusion models,” in CVPR, 2023, pp. 1900–1910.
  159. G. Kim, T. Kwon, and J. C. Ye, “Diffusionclip: Text-guided diffusion models for robust image manipulation,” in CVPR, 2022, pp. 2426–2435.
Citations (20)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up