Papers
Topics
Authors
Recent
Search
2000 character limit reached

Generative adversarial learning with optimal input dimension and its adaptive generator architecture

Published 6 May 2024 in cs.LG, stat.ME, and stat.ML | (2405.03723v1)

Abstract: We investigate the impact of the input dimension on the generalization error in generative adversarial networks (GANs). In particular, we first provide both theoretical and practical evidence to validate the existence of an optimal input dimension (OID) that minimizes the generalization error. Then, to identify the OID, we introduce a novel framework called generalized GANs (G-GANs), which includes existing GANs as a special case. By incorporating the group penalty and the architecture penalty developed in the paper, G-GANs have several intriguing features. First, our framework offers adaptive dimensionality reduction from the initial dimension to a dimension necessary for generating the target distribution. Second, this reduction in dimensionality also shrinks the required size of the generator network architecture, which is automatically identified by the proposed architecture penalty. Both reductions in dimensionality and the generator network significantly improve the stability and the accuracy of the estimation and prediction. Theoretical support for the consistent selection of the input dimension and the generator network is provided. Third, the proposed algorithm involves an end-to-end training process, and the algorithm allows for dynamic adjustments between the input dimension and the generator network during training, further enhancing the overall performance of G-GANs. Extensive experiments conducted with simulated and benchmark data demonstrate the superior performance of G-GANs. In particular, compared to that of off-the-shelf methods, G-GANs achieves an average improvement of 45.68% in the CT slice dataset, 43.22% in the MNIST dataset and 46.94% in the FashionMNIST dataset in terms of the maximum mean discrepancy or Frechet inception distance. Moreover, the features generated based on the input dimensions identified by G-GANs align with visually significant features.

Authors (3)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Towards principled methods for training generative adversarial networks. In International Conference on Learning Representations.
  2. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 214–223. PMLR.
  3. Generalization and equilibrium in generative adversarial nets (gans). In 34th International Conference on Machine Learning, pages 322–349. International Machine Learning Society (IMLS).
  4. Size-noise tradeoffs in generative networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 6490–6500.
  5. Barron, A. R. (1993). Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory, 39(3):930–945.
  6. Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research, 20(1):2285–2301.
  7. Bellido, I. (1993). Do backpropagation trained neural networks have normal weight distributions? In ICANN’93: Proceedings of the International Conference on Artificial Neural Networks Amsterdam, The Netherlands 13–16 September 1993 3, pages 772–775. Springer.
  8. Intrinsic dimension estimation using wasserstein distance. The Journal of Machine Learning Research, 23(1):14124–14160.
  9. Bottou, L. (2010). Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT’2010: 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers, pages 177–186. Springer.
  10. Convex optimization. Cambridge university press.
  11. Distribution approximation and statistical estimation guarantees of generative adversarial networks. arXiv preprint arXiv:2002.03938.
  12. Gans can play lottery tickets too. In International Conference on Learning Representations.
  13. Inferential wasserstein generative adversarial networks. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(1):83–113.
  14. Lojasiewicz inequalities and applications. Surveys in Differential Geometry, 19(1):63–82.
  15. On deep generative models for approximation and estimation of distributions on manifolds. Advances in Neural Information Processing Systems, 35:10615–10628.
  16. Deng, L. (2012). The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141–142.
  17. Consistent feature selection for analytic deep neural networks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 2420–2431.
  18. Consistent feature selection for neural networks via adaptive group lasso. arXiv preprint arXiv:2006.00334.
  19. Uci machine learning repository, 2017. URL http://archive. ics. uci. edu/ml, 7(1).
  20. Training generative neural networks via maximum mean discrepancy optimization. In Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, pages 258–267.
  21. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific reports, 7(1):12140.
  22. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, volume 9, pages 249–256.
  23. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems-Volume 2, pages 2672–2680.
  24. Improved training of wasserstein gans. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 5769–5779.
  25. Array programming with NumPy. Nature, 585(7825):357–362.
  26. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 1, page 770. IEEE.
  27. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6629–6640.
  28. An error analysis of generative adversarial networks for learning distributions. Journal of Machine Learning Research, 23(116):1–43.
  29. Rethinking the pruning criteria for convolutional neural network. Advances in Neural Information Processing Systems, 34:16305–16318.
  30. A global łojasiewicz inequality for algebraic varieties. Transactions of the American Mathematical Society, 329(2):813–818.
  31. Deep nonparametric regression on approximate manifolds: Nonasymptotic error bounds with polynomial prefactors. The Annals of Statistics, 51(2):691–716.
  32. A style-based generator architecture for generative adversarial networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4396–4405. IEEE Computer Society.
  33. Adam: A method for stochastic optimization. In International Conference on Learning Representations.
  34. On the ability of neural nets to express distributions. In Conference on Learning Theory, pages 1271–1296. PMLR.
  35. Maximum likelihood estimation of intrinsic dimension. In Proceedings of the 17th International Conference on Neural Information Processing Systems, pages 777–784.
  36. Generative moment matching networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning-Volume 37, pages 1718–1727. PMLR.
  37. A universal approximation theorem of deep neural networks for expressing probability distributions. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 3094–3105.
  38. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations.
  39. Sobolev gan. In International Conference on Learning Representations.
  40. Müller, A. (1997). Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):429–443.
  41. Adaptive approximation and generalization of deep neural network with intrinsic dimensionality. The Journal of Machine Learning Research, 21(1):7018–7055.
  42. Pytorch: an imperative style, high-performance deep learning library. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 8026–8037.
  43. Constructive universal high-dimensional distribution generation through deep relu networks. In International Conference on Machine Learning, pages 7610–7619. PMLR.
  44. The intrinsic dimension of images and its impact on learning. In International Conference on Learning Representations.
  45. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Representations.
  46. Group sparse regularization for deep neural networks. Neurocomputing, 241:81–89.
  47. Statistical guarantees for generative models without domination. In Algorithmic Learning Theory, pages 1051–1071. PMLR.
  48. Shen, Z. (2020). Deep network approximation characterized by number of neurons. Communications in Computational Physics, 28(5):1768–1811.
  49. Training very deep networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 2, pages 2377–2385.
  50. Going deeper with convolutions. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1–9.
  51. Villani, C. (2009). Optimal transport: old and new, volume 338. Springer.
  52. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747.
  53. On the capacity of deep generative networks for approximating distributions. Neural Networks, 145:144–154.
  54. Yarotsky, D. (2017). Error bounds for approximations with deep relu networks. Neural Networks, 94:103–114.
  55. Yarotsky, D. (2018). Optimal approximation of continuous functions by very deep relu networks. In Conference On Learning Theory, pages 639–649. PMLR.
  56. Can we find strong lottery tickets in generative models? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 3267–3275.
  57. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1):49–67.
  58. On the discrimination-generalization tradeoff in GANs. In International Conference on Learning Representations.
  59. A deep generative approach to conditional sampling. Journal of the American Statistical Association, 118(543):1837–1848.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 3 tweets with 9 likes about this paper.

Sign Up for Free