Abstract
Natural Evolution Strategies (NES) is a promising framework for black-box continuous optimization problems. NES optimizes the parameters of a probability distribution based on the estimated natural gradient, and one of the key parameters affecting the performance is the learning rate. We argue that from the viewpoint of the natural gradient method, the learning rate should be determined according to the estimation accuracy of the natural gradient. To do so, we propose a new learning rate adaptation mechanism for NES. The proposed mechanism makes it possible to set a high learning rate for problems that are relatively easy to optimize, which results in speeding up the search. On the other hand, in problems that are difficult to optimize (e.g., multimodal functions), the proposed mechanism makes it possible to set a conservative learning rate when the estimation accuracy of the natural gradient seems to be low, which results in the robust and stable search. The experimental evaluations on unimodal and multimodal functions demonstrate that the proposed mechanism works properly depending on a search situation and is effective over the existing method, i.e., using the fixed learning rate.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Akimoto, Y., Auger, A., Hansen, N.: Comparison-Based Natural Gradient Optimization in High Dimension. In: Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation. pp. 373–380. ACM (2014)
Akimoto, Y., Nagata, Y., Ono, I., Kobayashi, S.: Bidirectional relation between CMA evolution strategies and natural evolution strategies. In: Schaefer, R., Cotta, C., Kołodziej, J., Rudolph, G. (eds.) PPSN 2010. LNCS, vol. 6238, pp. 154–163. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15844-5_16
Amari, S.I., Douglas, S.C.: Why natural gradient? In: Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 1998 (Cat. No. 98CH36181), vol. 2, pp. 1213–1216. IEEE (1998)
Auger, A., Hansen, N.: A restart CMA evolution strategy with increasing population size. In: 2005 IEEE Congress on Evolutionary Computation, vol. 2, pp. 1769–1776. IEEE (2005)
BANGS II, W.J.: Array Processing with Generalized Beam-Formers. Yale University, New Haven (1971)
Beyer, H.G.: Convergence analysis of evolutionary algorithms that are based on the paradigm of information geometry. Evol. Comput. 22(4), 679–709 (2014)
Fukushima, N., Nagata, Y., Kobayashi, S., Ono, I.: Proposal of distance-weighted exponential natural evolution strategies. In: 2011 IEEE Congress of Evolutionary Computation (CEC), pp. 164–171. IEEE (2011)
Glasmachers, T., Schaul, T., Yi, S., Wierstra, D., Schmidhuber, J.: Exponential natural evolution strategies. In: Proceedings of the 12th Annual Conference on Genetic and Evolutionary Computation, pp. 393–400 (2010)
Hansen, N.: The CMA evolution strategy: a comparing review. In: Lozano, J.A., Larrañaga, P., Inza, I., Bengoetxea, E. (eds.) Towards a New Evolutionary Computation. SFSC, vol. 192, pp. 75–102. Springer, Heidelberg (2006). https://doi.org/10.1007/3-540-32494-1_4
Hansen, N.: The CMA evolution strategy: a tutorial. ar**v preprint ar**v:1604.00772 (2016)
Hansen, N., Kern, S.: Evaluating the CMA evolution strategy on multimodal test functions. In: Yao, X., et al. (eds.) PPSN 2004. LNCS, vol. 3242, pp. 282–291. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30217-9_29
Hansen, N., Müller, S.D., Koumoutsakos, P.: Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evol. Comput. 11(1), 1–18 (2003)
Loshchilov, I., Schoenauer, M., Sebag, M., Hansen, N.: Maximum likelihood-based online adaptation of hyper-parameters in CMA-ES. In: Bartz-Beielstein, T., Branke, J., Filipič, B., Smith, J. (eds.) PPSN 2014. LNCS, vol. 8672, pp. 70–79. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10762-2_7
Nishida, K., Akimoto, Y.: Population size adaptation for the CMA-ES based on the estimation accuracy of the natural gradient. In: Proceedings of the Genetic and Evolutionary Computation Conference 2016, pp. 237–244 (2016)
Nishida, K., Akimoto, Y.: PSA-CMA-ES: CMA-ES with population size adaptation. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 865–872 (2018)
Nomura, M., Ono, I.: Natural evolution strategy for unconstrained and implicitly constrained problems with ridge structure. In: 2021 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1–7. IEEE (2021)
Nomura, M., Sakai, N., Fukushima, N., Ono, I.: Distance-weighted exponential natural evolution strategy for implicitly constrained black-box function optimization. In: 2021 IEEE Congress on Evolutionary Computation (CEC), pp. 1099–1106. IEEE (2021)
Ollivier, Y., Arnold, L., Auger, A., Hansen, N.: Information-geometric optimization algorithms: a unifying picture via invariance principles. J. Mach. Learn. Res. 18(1), 564–628 (2017)
Otwinowski, J., LaMont, C.H., Nourmohammad, A.: Information-geometric optimization with natural selection. Entropy 22(9), 967 (2020)
Slepian, D.: Estimation of signal parameters in the presence of noise. Trans. IRE Prof. Group Inf. Theory 3(3), 68–89 (1954)
Sun, Y., Wierstra, D., Schaul, T., Schmidhuber, J.: Efficient natural evolution strategies. In: Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation, pp. 539–546. ACM (2009)
Wierstra, D., Schaul, T., Glasmachers, T., Sun, Y., Peters, J., Schmidhuber, J.: Natural evolution strategies. J. Mach. Learn. Res. 15(1), 949–980 (2014)
Yi, S., Wierstra, D., Schaul, T., Schmidhuber, J.: Stochastic search using the natural gradient. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 1161–1168. ACM (2009)
Acknowledgement
The authors thank anonymous reviewers for their helpful comments. This work was partially supported by JSPS KAKENHI Grant Number JP20K11986.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Nomura, M., Ono, I. (2022). Towards a Principled Learning Rate Adaptation for Natural Evolution Strategies. In: Jiménez Laredo, J.L., Hidalgo, J.I., Babaagba, K.O. (eds) Applications of Evolutionary Computation. EvoApplications 2022. Lecture Notes in Computer Science, vol 13224. Springer, Cham. https://doi.org/10.1007/978-3-031-02462-7_45
Download citation
DOI: https://doi.org/10.1007/978-3-031-02462-7_45
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-02461-0
Online ISBN: 978-3-031-02462-7
eBook Packages: Computer ScienceComputer Science (R0)