CatCMA : Stochastic Optimization for Mixed-Category Problems
Abstract: Black-box optimization problems often require simultaneously optimizing different types of variables, such as continuous, integer, and categorical variables. Unlike integer variables, categorical variables do not necessarily have a meaningful order, and the discretization approach of continuous variables does not work well. Although several Bayesian optimization methods can deal with mixed-category black-box optimization (MC-BBO), they suffer from a lack of scalability to high-dimensional problems and internal computational cost. This paper proposes CatCMA, a stochastic optimization method for MC-BBO problems, which employs the joint probability distribution of multivariate Gaussian and categorical distributions as the search distribution. CatCMA updates the parameters of the joint probability distribution in the natural gradient direction. CatCMA also incorporates the acceleration techniques used in the covariance matrix adaptation evolution strategy (CMA-ES) and the stochastic natural gradient method, such as step-size adaptation and learning rate adaptation. In addition, we restrict the ranges of the categorical distribution parameters by margin to prevent premature convergence and analytically derive a promising margin setting. Numerical experiments show that the performance of CatCMA is superior and more robust to problem dimensions compared to state-of-the-art Bayesian optimization algorithms.
- Optuna: A Next-Generation Hyperparameter Optimization Framework. In The 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2623–2631.
- Bidirectional Relation between CMA Evolution Strategies and Natural Evolution Strategies. In Parallel Problem Solving from Nature, PPSN XI. 154–163.
- Theoretical foundation for CMA-ES from information geometry perspective. Algorithmica 64 (2012), 698–716.
- Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search. In Proceedings of the 36th International Conference on Machine Learning (ICML), Vol. 97. 171–180.
- Algorithms for Hyper-Parameter Optimization. In Advances in Neural Information Processing Systems, Vol. 24.
- Peter I Frazier. 2018. A Tutorial on Bayesian Optimization. arXiv preprint arXiv:1807.02811 (2018).
- CMA-ES with Margin: Lower-Bounding Marginal Probability for Mixed-Integer Black-Box Optimization. In Proceedings of the Genetic and Evolutionary Computation Conference (Boston, Massachusetts) (GECCO ’22). Association for Computing Machinery, New York, NY, USA, 639–647. https://doi.org/10.1145/3512290.3528827
- Nikolaus Hansen. 2016. The CMA evolution strategy: A tutorial. arXiv preprint arXiv:1604.00772 (2016).
- Nikolaus Hansen and Anne Auger. 2014. Principled Design of Continuous Stochastic Search: From Theory to Practice. In Theory and Principled Methods for the Design of Metaheuristics. 145–180. https://doi.org/10.1007/978-3-642-33206-7_8
- Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES). Evolutionary Computation 11, 1 (March 2003), 1–18. https://doi.org/10.1162/106365603321828970
- N. Hansen and A. Ostermeier. 1996. Adapting Arbitrary Normal Mutation Distributions in Evolution Strategies: The Covariance Matrix Adaptation. In Proceedings of IEEE International Conference on Evolutionary Computation. 312–317. https://doi.org/10.1109/ICEC.1996.542381
- Hyperparameter optimization: a spectral approach. In International Conference on Learning Representations (ICLR).
- Automated Machine Learning: Methods, Systems, Challenges (1st ed.). Springer Publishing Company, Incorporated. https://doi.org/10.1007/978-3-030-05318-5
- A Mixed-Variable Bayesian Optimization Approach for Analog Circuit Synthesis. In 2020 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, 1–4.
- The Knowledge-Gradient Algorithm for Sequencing Experiments in Drug Discovery. INFORMS Journal on Computing 23, 3 (2011), 346–363.
- CMA-ES with Learning Rate Adaptation. arXiv preprint arXiv:2401.15876 (2024).
- Masahiro Nomura and Masashi Shibata. 2024. cmaes : A Simple yet Practical Python Library for CMA-ES. arXiv preprint arXiv:2402.01373 (2024).
- Information-Geometric Optimization Algorithms: A Unifying Picture via Invariance Principles. Journal of Machine Learning Research 18, 1 (2017), 564–628.
- Konstantinos Touloupas and Paul P Sotiriadis. 2022. Mixed-Variable Bayesian Optimization for Analog Circuit Sizing using Variational Autoencoders. In 2022 18th International Conference on Synthesis, Modeling, Analysis and Simulation Methods and Applications to Circuit Design (SMACD). IEEE, 1–4.
- Think Global and Act Local: Bayesian Optimisation over High-Dimensional Categorical and Mixed Search Spaces. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 10663–10674. https://proceedings.mlr.press/v139/wan21b.html
- Optimizing Chemical Reactions with Deep Reinforcement Learning. ACS central science 3, 12 (2017), 1337–1344.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
