How to Construct Energy for Images? Denoising Autoencoder Can Be Energy Based Model

Abstract: Energy-based models parameterize the unnormalized log-probability of data samples, but there is a lack of guidance on how to construct the "energy". In this paper, we propose a Denoising-EBM which decomposes the image energy into "semantic energy" and "texture energy". We define the "semantic energy" in the latent space of DAE to model the high-level representations, and define the pixel-level reconstruction error for denoising as "texture energy". Inspired by score-based model, our model utilizes multi-scale noisy samples for maximum-likelihood training and it outputs a vector instead of a scalar for exploring a larger set of functions during optimization. After training, the semantics are first synthesized by fast MCMC through "semantic energy", and then the pixel-level refinement of semantic image will be performed to generate perfect samples based on "texture energy". Ultimately, our model can outperform most EBMs in image generation. And we also demonstrate that Denoising-EBM has top performance among EBMs for out-of-distribution detection.