Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections

Abstract: In recent years there has been increased interest in understanding the interplay between deep generative models (DGMs) and the manifold hypothesis. Research in this area focuses on understanding the reasons why commonly-used DGMs succeed or fail at learning distributions supported on unknown low-dimensional manifolds, as well as developing new models explicitly designed to account for manifold-supported data. This manifold lens provides both clarity as to why some DGMs (e.g. diffusion models and some generative adversarial networks) empirically surpass others (e.g. likelihood-based models such as variational autoencoders, normalizing flows, or energy-based models) at sample generation, and guidance for devising more performant DGMs. We carry out the first survey of DGMs viewed through this lens, making two novel contributions along the way. First, we formally establish that numerical instability of likelihoods in high ambient dimensions is unavoidable when modelling data with low intrinsic dimension. We then show that DGMs on learned representations of autoencoders can be interpreted as approximately minimizing Wasserstein distance: this result, which applies to latent diffusion models, helps justify their outstanding empirical results. The manifold lens provides a rich perspective from which to understand DGMs, and we aim to make this perspective more accessible and widespread.

• The paper establishes the manifold hypothesis as a core framework explaining why DGMs effectively learn low-dimensional structures from high-dimensional data.
• It demonstrates that diffusion models, through iterative Gaussian noise addition, outperform likelihood-based approaches in managing manifold-supported data.
• It introduces a novel reinterpretation using autoencoder representations with Wasserstein minimization to enhance generative stability.

Deep Generative Models through the Lens of the Manifold Hypothesis

This paper revisits the theoretical and practical facets of Deep Generative Models (DGMs) by exploring them through the lens of the manifold hypothesis. The authors delineate the manifold hypothesis as a central framework for understanding why certain DGMs excel in learning distributions embedded on unknown manifolds in high-dimensional spaces while others falter. They aim to provide clarity on how some models like diffusion and latent diffusion models yield superior empirical results. Moreover, the paper introduces novel contributions that investigate the inherent limitations of high-dimensional likelihood models and reinterpret DGMs on learned representations of autoencoders.

The Manifold Hypothesis and Its Relevance

The manifold hypothesis posits that high-dimensional data typically resides on a low-dimensional manifold embedded within the ambient space, indicated here as $d^* < D$. This hypothesis provides a robust explanation for why DGMs such as diffusion models often outperform their likelihood-based counterparts, such as VAEs, NFs, or EBMs, on complex data. This is because the hypothesis allows for understanding the pitfalls of high-dimensional likelihood estimations, which are prone to numerical instability when approximating low-dimensional structures.

The manifold hypothesis is supported by various arguments. For instance, theoretical advancements have demonstrated that under this hypothesis, the sample complexity becomes feasible, correlating with intrinsic dimension rather than the high-dimensional dataset space. This has motivated the development of models that effectively incorporate the manifold hypothesis into their architecture, guiding performance improvements.

Numerical Instability in High-Dimensional Likelihood Models

One of the crucial insights offered by the paper is the unavoidable numerical instability associated with high-dimensional likelihoods when modeling low-dimensional data. The authors mathematically prove that a full-dimensional density model cannot entirely capture the structure of manifold-supported data without incurring numerical instability. This inherent flaw underscores the necessity for DGMs to integrate manifold-awareness into their design, thereby ensuring stability in representing the data's interesting structure.

Figure 1: An illustration emphasizing how full-dimensional densities encounter numerical instability when approximating data on a low-dimensional manifold.

Evaluating and Designing DGMs with the Manifold Hypotheses

The authors critically analyze different model families through the manifold lens, demonstrating why diffusion models surpass traditional likelihood-based approaches. Uniquely, diffusion models add Gaussian noise to data iteratively, smoothing the learning landscape and enabling these models to handle manifold-supported data effectively. This can be seen in contrast to NFs and VAEs, which suffer from manifold overfitting due to their inability to adapt naturally to such structures.

Furthermore, this work includes a novel reinterpretation of DGMs on autoencoder representations, suggesting that these models approximately minimize Wasserstein distance—a widely acknowledged metric in the context of optimal transport. The reexamination allows for deeper introspective thought on encoding-decoding strategies, enhancing the generative quality of manifold-aware DGMs.

Figure 2: A representation of the manifold overfitting problem where high-dimensional models inadequately approximate low-dimensional data structures.

Practical Implications and Future Directions

The practical implications of this work point toward designing DGMs that inherently include manifold-awareness. Future research avenues may involve exploring new architectures or training methods that jointly optimize data representation on low-dimensional manifolds without relying solely on high-dimensional parameterizations.

The findings also suggest robust methods for analyzing the convergence properties of DGMs under the manifold assumption, potentially leading to more performant and stable generative processes that are less dependent on vast computational resources.

Conclusion

Overall, the paper provides a compelling argument for leveraging the manifold hypothesis in the design and evaluation of DGMs. By focusing on the manifold-supported representations of data, researchers can improve consistency, robustness, and performance of generative models across various applications. Ultimately, these insights pave the way for developing next-generation DGMs that more accurately reflect the complex distributions they aim to model, promising to enhance applications in machine learning domains where data lies on intricate manifolds.