A note on the evaluation of generative models

Abstract: Probabilistic generative models can be used for compression, denoising, inpainting, texture synthesis, semi-supervised learning, unsupervised feature learning, and other tasks. Given this wide range of applications, it is not surprising that a lot of heterogeneity exists in the way these models are formulated, trained, and evaluated. As a consequence, direct comparison between models is often difficult. This article reviews mostly known but often underappreciated properties relating to the evaluation and interpretation of generative models with a focus on image models. In particular, we show that three of the currently most commonly used criteria---average log-likelihood, Parzen window estimates, and visual fidelity of samples---are largely independent of each other when the data is high-dimensional. Good performance with respect to one criterion therefore need not imply good performance with respect to the other criteria. Our results show that extrapolation from one criterion to another is not warranted and generative models need to be evaluated directly with respect to the application(s) they were intended for. In addition, we provide examples demonstrating that Parzen window estimates should generally be avoided.

• The paper reveals that common metrics like average log-likelihood, Parzen window estimates, and visual fidelity are largely independent in high-dimensional settings.
• The paper illustrates that different training objectives, including JSD and MMD, lead to distinct performance trade-offs as demonstrated with a toy Gaussian example.
• The paper recommends aligning evaluation methods with specific application needs to avoid overreliance on potentially misleading metrics such as Parzen window estimates.

Evaluation of Generative Models: An In-depth Analysis

The paper, "A note on the evaluation of generative models," authored by Lucas Theis, Aäron van den Oord, and Matthias Bethge, addresses the multifaceted nature of evaluating probabilistic generative models, specifically within the image modeling domain. The authors systematically examine the often-overlooked intricacies and limitations associated with the prevalent methodologies used to train and assess these models.

Key Metrics and Their Independence

The paper explores three commonly used criteria for evaluating generative models:

1. Average Log-Likelihood: Often serves as the default metric for quantifying performance in density estimation tasks.
2. Parzen Window Estimates: A method for estimating the model’s likelihood using a kernel density estimator.
3. Visual Fidelity of Samples: Assesses the sample quality generated by the model through visual inspection.

The authors present empirical evidence showing that these criteria are largely independent of one another, especially in high-dimensional data settings. For instance, a generative model might excel in one metric but perform poorly in another, undermining any assumptions that good performance in one criterion implies similar performance in others.

Training Objectives and Their Implications

Different training objectives, such as Jensen-Shannon Divergence (JSD) and Maximum Mean Discrepancy (MMD), lead to significantly different outcomes compared to traditional log-likelihood-based metrics. The paper presents a detailed examination using a toy example where an isotropic Gaussian distribution is fit to data drawn from a mixture of Gaussians. Through this illustration, the authors highlight the distinct trade-offs and implications each optimization criterion offers.

Figure 1 of the paper exemplifies this by depicting how minimizing Kullback-Leibler divergence (KLD), MMD, and JSD results in significantly different fits to the data, further emphasizing the necessity of selecting the appropriate metric for the specific application at hand.

Practical Implications of Evaluation Metrics

Log-Likelihood

While average log-likelihood is extensively used, the paper cautions that it needs careful handling to ensure meaningful results. For correct interpretation, the discrete nature of image data should be considered, and proper dequantization techniques should be applied.

Limitations of Parzen Window Estimates

Parzen window estimates, though often used as proxies for model likelihood, are shown to be unreliable, particularly for high-dimensional data. The paper provides a compelling argument with empirical data demonstrating that Parzen window estimates are far from true log-likelihood values, even with a substantial number of samples.

Visual Fidelity and Overfitting

Evaluation based on visual fidelity can mislead researchers towards overfitted models. The authors present a nuanced discussion on how models with great visual fidelity do not necessarily possess high log-likelihood and vice versa. For instance, samples generated with a simple lookup table approach might appear visually convincing but perform poorly on unseen data.

Future Directions and Recommendations

The paper concludes with a strong emphasis on the alignment between the evaluation metric and the application context. It dissuades the use of Parzen window estimates unless specifically required by the application, proposing that evaluations should match the intended use case of the generative model.

Significantly, the authors highlight that understanding the distinct trade-offs of different training objectives enables better model interpretation and empirical results correlation. They also underscore the inadequacy of a one-size-fits-all approach in the context of generative model evaluation, calling for a nuanced assessment tailored to the specific domain application.

Conclusion

This paper delivers an insightful and thorough analysis of the existing evaluation metrics for generative models. It underscores the orthogonality of popular evaluation criteria, the implications of various training objectives, and the importance of application-specific evaluations. For future research, these insights pave the way for more informed, application-driven development and assessment of generative models in AI.