Papers
Topics
Authors
Recent
Search
2000 character limit reached

Latent Abstractions in Generative Diffusion Models

Published 4 Oct 2024 in cs.LG and stat.ML | (2410.03368v1)

Abstract: In this work we study how diffusion-based generative models produce high-dimensional data, such as an image, by implicitly relying on a manifestation of a low-dimensional set of latent abstractions, that guide the generative process. We present a novel theoretical framework that extends NLF, and that offers a unique perspective on SDE-based generative models. The development of our theory relies on a novel formulation of the joint (state and measurement) dynamics, and an information-theoretic measure of the influence of the system state on the measurement process. According to our theory, diffusion models can be cast as a system of SDE, describing a non-linear filter in which the evolution of unobservable latent abstractions steers the dynamics of an observable measurement process (corresponding to the generative pathways). In addition, we present an empirical study to validate our theory and previous empirical results on the emergence of latent abstractions at different stages of the generative process.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. Enlargement of filtration with finance in view. 2017.
  2. Enlarged filtrations for diffusions. Stochastic Processes and their Applications, 24(1):99–107, 1987.
  3. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797, 2023.
  4. Brian D. O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313–326, 1982.
  5. Fundamentals of stochastic filtering, volume 3. Springer, 2009.
  6. Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp.  5185–5198, 2020.
  7. An optimal control perspective on diffusion-based generative modeling. arXiv preprint arXiv:2211.01364, 2022.
  8. Experience grounds language. arXiv preprint arXiv:2004.10151, 2020.
  9. Beyond surface statistics: Scene representations in a latent diffusion model, 2023. URL https://arxiv.org/abs/2306.05720.
  10. Sylvain Corlay. Properties of the ornstein-uhlenbeck bridge. arXiv preprint arXiv:1310.5617, 2013.
  11. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp.  8780–8794. Curran Associates, Inc., 2021.
  12. Tyrone E Duncan. On the calculation of mutual information. SIAM Journal on Applied Mathematics, 19(1):215–220, 1970.
  13. Tyrone E. Duncan. Mutual information for stochastic differential equations. Information and Control, 19(3):265–271, 1971. ISSN 0019-9958. doi: https://doi.org/10.1016/S0019-9958(71)90135-5. URL https://www.sciencedirect.com/science/article/pii/S0019995871901355.
  14. Nonlinear filtering with degenerate noise. Electronic Communications on Probabability, 22, 2017.
  15. Minde: Mutual information neural diffusion estimation. arXiv preprint arXiv:2310.09031, 2023.
  16. MINDE: Mutual information neural diffusion estimation. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=0kWd8SJq8d.
  17. Enlargement of filtrations: An exposition of core ideas with financial examples. arXiv preprint arXiv:2303.03573, 2023.
  18. Likelihood-based diffusion language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=e2MCL6hObn.
  19. Discovering interpretable directions in the semantic latent space of diffusion models, 2024. URL https://arxiv.org/abs/2303.11073.
  20. U. G. Haussmann and E. Pardoux. Time Reversal of Diffusions. The Annals of Probability, 14(4):1188 – 1205, 1986. doi: 10.1214/aop/1176992362. URL https://doi.org/10.1214/aop/1176992362.
  21. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.  770–778, 2016. doi: 10.1109/CVPR.2016.90.
  22. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221, 2022.
  23. DiffusionBERT: Improving generative masked language models with diffusion models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023.
  24. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.  6840–6851. Curran Associates, Inc., 2020.
  25. Imagen video: High definition video generation with diffusion models, 2022. URL https://arxiv.org/abs/2210.02303.
  26. Equivariant diffusion for molecule generation in 3D. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.  8867–8887. PMLR, 17–23 Jul 2022.
  27. Olav Kallenberg. Foundations of modern probability. Probability and its Applications (New York). Springer-Verlag, New York, second edition, 2002. ISBN 0-387-95313-2. doi: 10.1007/978-1-4757-4015-8. URL https://doi.org/10.1007/978-1-4757-4015-8.
  28. Disentangling by factorising. In International conference on machine learning, pp.  2649–2658. PMLR, 2018.
  29. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015.
  30. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
  31. The hitchhiker’s guide to nonlinear filtering. Journal of Mathematical Psychology, 94:102307, 2020.
  32. Diffusion models already have a semantic latent space, 2023. URL https://arxiv.org/abs/2210.10960.
  33. Emergent world representations: Exploring a sequence model trained on a synthetic task. arXiv preprint arXiv:2210.13382, 2022a.
  34. Diffusion-LM improves controllable text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022b. URL https://openreview.net/forum?id=3s9IrEsjLyk.
  35. An analysis of human alignment of latent diffusion models. arXiv preprint arXiv:2403.08469, 2024.
  36. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11020–11028, Jun. 2022.
  37. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.
  38. Discrete diffusion modeling by estimating the ratios of the data distribution. In International conference on machine learning, 2024.
  39. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  2837–2845, June 2021.
  40. Alain Mazzolo. Constraint ornstein-uhlenbeck bridges. Journal of Mathematical Physics, 58(9), 2017.
  41. A variational approach to nonlinear estimation. SIAM journal on control and optimization, 42(5):1813–1833, 2003.
  42. Nigel J Newton. Interactive statistical mechanics and nonlinear filtering. Journal of Statistical Physics, 133(4):711–737, 2008.
  43. Bernt Øksendal. Stochastic differential equations. Springer, 2003.
  44. Peter Ouwehand. Enlargement of filtrations–a primer. arXiv preprint arXiv:2210.07045, 2022.
  45. E Pardoux. Grossissement d’une filtration et retournement du temps d’une diffusion. In Séminaire de Probabilités XX 1984/85: Proceedings, pp.  48–55. Springer, 2006.
  46. Understanding the latent space of diffusion models through the lens of riemannian geometry. Advances in Neural Information Processing Systems, 36:24129–24142, 2023.
  47. Maxim Raginsky. A variational approach to sampling in diffusion processes. arXiv preprint arXiv:2405.00126, 2024.
  48. Improved sampling via learned diffusions. arXiv preprint arXiv:2307.01198, 2023.
  49. Diffusions, Markov processes, and martingales: Itô calculus, volume 2. Cambridge university press, 2000.
  50. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pp.  234–241, 2015.
  51. A phase transition in diffusion models reveals the hierarchical nature of data, 2024. URL https://arxiv.org/abs/2402.16991.
  52. On the stratification of multi-label data. Machine Learning and Knowledge Discovery in Databases, pp.  145–158, 2011.
  53. Steven E Shreve. Stochastic calculus for finance II: Continuous-time models, volume 11. Springer, 2004.
  54. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
  55. A network perspective on stratification of multi-label data. In Luís Torgo, Bartosz Krawczyk, Paula Branco, and Nuno Moniz (eds.), Proceedings of the First International Workshop on Learning with Imbalanced Domains: Theory and Applications, volume 74 of Proceedings of Machine Learning Research, pp.  22–35, ECML-PKDD, Skopje, Macedonia, 2017. PMLR.
  56. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36:1363–1389, 2023.
  57. Diffusion probabilistic modeling of protein backbones in 3d for the motif-scaffolding problem, 2022. URL https://arxiv.org/abs/2206.04119.
  58. Ramon Van Handel. Filtering, stability, and robustness. PhD thesis, California Institute of Technology, 2007.
  59. Invariance properties of the conditional independence relation. The Annals of Probability, pp.  934–945, 1985.
  60. Denoising diffusion autoencoders are unified self-supervised learners. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  15802–15812, 2023.
  61. J. Xiong. An Introduction to Stochastic Filtering Theory. Oxford University Press, 2008.
  62. First hitting diffusion models for generating manifold, graph and categorical data. Advances in Neural Information Processing Systems, 35:27280–27292, 2022.
  63. Lion: Latent point diffusion models for 3d shape generation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
  64. Markov bridges: Sde representation. Stochastic Processes and their Applications, 126(3):651–679, 2016. ISSN 0304-4149. doi: https://doi.org/10.1016/j.spa.2015.09.015. URL https://www.sciencedirect.com/science/article/pii/S0304414915002434.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up