Abstract
This chapter provides an overview of evolutionary approaches to supervised learning. It starts with the definition and scope of the opportunity, and then reviews three main areas: evolving general neural network designs, evolving solutions that are explainable, and forming a synergy of evolutionary and gradient-based methods.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Aharonov-Barki, R., Beker, T., Ruppin, E.: Emergence of memory-driven command neurons in evolved artificial agents. Neural Comput. 13, 691–716 (2001)
Aitkenhead, M.J.: A co-evolving decision tree classification method. Expert Syst. Appl. 34, 19–25 (2008)
Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: Precup, D., Teh, Y.W. (eds.), Proceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 214–223 (2017)
Barros, R.C., Basgalupp, M.P., De Carvalho, A.C., Freitas, A.A.: A survey of evolutionary algorithms for decision-tree induction. IEEE Trans. Syst., Man, Cybern., Part C (Appl. Rev.) 42, 291–312 (2012)
Bi, Y., Xue, B., Zhang, M.: Genetic programming-based evolutionary deep learning for data-efficient image classification. IEEE Trans. Evolut. Comput. (2022). https://doi.org/10.1109/TEVC.2022.3214503
Bingham, G., Macke, W., Miikkulainen, R.: Evolutionary optimization of deep learning activation functions. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 289–296 (2020)
Bingham, G., Miikkulainen, R.: Discovering parametric activation functions. Neural Netw. 148, 48–65 (2022)
Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001)
Breiman, L., Friedman, J., Stone, C.J., Olshen, R.A.: Classification and Regression Trees. Chapman and Hall/CRC (1984)
Butz, M.V., Lanzi, P.L., Wilson, S.W.: Function approximation with xcs: Hyperellipsoidal conditions, recursive least squares, and compaction. IEEE Trans. Evolut. Comput. 12, 355–376 (2008)
Canatar, A., Bordelon, B., Pehlevan, C.: Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nat. Commun. 12, 1914 (2021)
Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math. Control Signals Syst. 2, 303–314 (1989)
Dai, E., Zhao, T., Zhu, H., Xu, J., Guo, Z., Liu, H., Tang, J., Wang, S.: A comprehensive survey on trustworthy graph neural networks: privacy, robustness, fairness, and explainability. ar**v:2104.05605, 2020
De Jong, K.: Learning with genetic algorithms: an overview. Mach. Learn. 3, 121–138 10 (1988)
Deb, K., Myburgh, C.: A population-based fast algorithm for a billion-dimensional resource allocation problem with integer variables. Eur. J. Oper. Res. 261, 460–474 (2017)
Dolotov, E., Zolotykh, N.Y.: Evolutionary algorithms for constructing an ensemble of decision trees (2020). ar**v:2002.00721
Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey. J. Mach. Learn. Res. 20, 1–21 (2019)
Gaier, A., Ha, D.: Weight agnostic neural networks. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.), Advances in Neural Information Processing Systems 5364–5378 (2019)
Ganon, Z., Keinan, A., Ruppin, E.: Evolutionary network minimization: adaptive implicit pruning of successful agents. In: Banzhaf, W., Ziegler, J., Christaller, T., Dittrich, P., Kim, J.T. (eds.) Advances in Artificial Life, pp. 319–327. Springer, Berlin (2003)
Gao, B., Gouk, H., Hospedales, T.M.: Searching for robustness: loss learning for noisy classification tasks. IEEE/CVF International Conference on Computer Vision, pp. 6650–6659 (2021)
Gomez, F., Miikkulainen, R.: Incremental evolution of complex general behavior. Adapt. Behav. 5, 317–342 (1997)
Gomez, F., Schmidhuber, J., Miikkulainen, R. and Mitchell, M.: Accelerated neural evolution through cooperatively coevolved synapses. J. Mach. Learn. Res. 937–965 (2008)
Gonzalez, S., Kant, M., Miikkulainen, R.: Evolving GAN formulations for higher quality image synthesis. In: Kozma, R., Alippi, C., Choe, Y., Morabito, F.C. (eds.) Artificial Intelligence in the Age of Neural Networks and Brain Computing, 2nd edn. Elsevier, New York (2023)
Gonzalez, S., Landgraf, J., Miikkulainen, R.: Faster training by selecting samples using embeddings. In: Proceedings of the 2019 International Joint Conference on Neural Networks, pp. 1–7 (2019)
Gonzalez, S., Miikkulainen, R.: Improved training speed, accuracy, and data utilization through loss function optimization. In: Proceedings of the 2020 IEEE Congress on Evolutionary Computation (CEC), pp. 1–8 (2020)
Gonzalez, S., Miikkulainen, R.: Effective regularization through loss-function metalearning (2021). ar**v:2010.00788
Gonzalez, S., Miikkulainen, R.: Optimizing loss functions through multivariate Taylor polynomial parameterization. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 305–313 (2021)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. and Bengio, Y.: Generative adversarial nets. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q., (eds.), Advances in Neural Information Processing Systems 27, pp. 2672–2680 (2014)
Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evol. Comput. 9, 159–195 (2001)
Hanson, S.J., Pratt, L.Y.: Comparing biases for minimal network construction with back-propagation. In: Proceedings of the 1st International Conference on Neural Information Processing Systems, pp. 177–185. MIT Press, Cambridge (1988)
Hayes-Roth, F.: Rule-based systems. Commun. ACM 28, 921–932 (1985)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hemberg, E., Veeramachaneni, K., Wanigarekara, P., Shahrzad, H., Hodjat, B., O’Reilly, U.-M.: Learning decision lists with lagged physiological time series. In: Workshop on Data Mining for Medicine and Healthcare, 14th SIAM International Conference on Data Mining, pp. 82–87 (2014)
Holland, J.H.: Esca** brittleness: the possibilities of general purpose learning algorithms applied to parallel rule-based systems. In: Michalski, R.S., Carbonell, J.G., Mitchell, T.M. (eds.) Machine Learning: An Artificial Intelligence Approach, vol. 2, pp. 593–623. Morgan Kaufmann, Los Altos (1986)
Hoos, H.: Programming by optimization. Commun. ACM 55, 70–80 (2012)
Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw. 2, 359–366 (1989)
Huang, X., Kroening, D., Ruan, W., Sharp, J., Sun, Y., Thamo, E., Min, W., Yi, X.: A survey of safety and trustworthiness of deep neural networks: verification, testing, adversarial attack and defence, and interpretability. Comput. Sci. Rev. 37, 100270 (2020)
Ijspeert, A.J.: Central pattern generators for locomotion control in animals and robots: A review. Neural Netw. 21, 642–653 (2008)
Jankowski, D., Jackowski, K.: Evolutionary algorithm for decision tree induction. In: Saeed, K., Snášel, V. (eds.) Computer Information Systems and Industrial Management, pp. 23–32. Springer, Berlin (2014)
Kashtan, N., Alon, U.: Spontaneous evolution of modularity and network motifs. Proc. Natl. Acad. Sci. 102, 13773–13778 (2005)
Langdon, W.B., Poli, R., McPhee, N.F., Koza, J.R.: Genetic programming: An introduction and tutorial, with a survey of techniques and applications. In: Fulcher, J., Jain, L.C. (eds.) Computational Intelligence: A Compendium, pp. 927–1028. Springer, Berlin (2008)
Liang, J., Meyerson, E. and Miikkulainen, R.: Evolutionary architecture search for deep multitask networks. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 466–473 (2018)
Liang, J., Gonzalez, S., Shahrzad, H., Miikkulainen, R.: Regularized evolutionary population-based training. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 323–331 (2021)
Liang, J., Meyerson, E., Hodjat, B., Fink, D., Mutch, K. and Miikkulainen, R.: Evolutionary neural AutoML for deep learning. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2019), pp. 401–409 (2019)
Liang, J.Z., Miikkulainen, R.: Evolutionary bilevel optimization for complex control tasks. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2015), pp. 871–878 (2015)
Light, W.: Ridge functions, sigmoidal functions and neural networks. In: Approximation Theory VII, pp. 158–201. Academic, Boston (1992)
Liu, Y., Sun, Y., Xue, B., Zhang, M., Yen, G.G., Tan, K.C.: A survey on evolutionary neural architecture search. IEEE Trans. Neural Netw. Learn. Syst. 1–21 (2021)
Liu, Z., Zhang, X., Wang, S., Ma, S., Gao, W.: Evolutionary quantization of neural networks with mixed-precision. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2785–2789 (2021)
Lu, Z., Deb, K., Goodman, E., Banzhaf, W., Boddeti, V.N.: Nsganetv2: evolutionary multi-objective surrogate-assisted neural architecture search. In: European Conference on Computer Vision ECCV-2020, LNCS, vol. 12346, pp. 35–51 (2020)
Mao, X., Li, Q., **e, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2794–2802 (2017)
Meyerson, E., Miikkulainen, R.: Pseudo-task augmentation: from deep multitask learning to intratask sharing—and back. In: Proceedings of the 35th International Conference on Machine Learning, pp. 739–748 (2018)
Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., Hodjat, B.: Evolving deep neural networks. In: Morabito, C.F., Alippi, C., Choe, Y., Kozma, R. (eds.) Artificial Intelligence in the Age of Neural Networks and Brain Computing, 2nd edn., pp. 293–312. Elsevier, New York (2023)
Miikkulainen, R., Meyerson, E., Qiu, X., Sinha, U., Kumar, R., Hofmann, K., Yan, Y.M., Ye, M., Yang, J., Caiazza, D. and Brown, S.M.: Evaluating medical aesthetics treatments through evolved age-estimation models. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1009–1017 (2021)
Montana, D.J., Davis, L.: Training feedforward neural networks using genetic algorithms. In: International Joint Conference on Artificial Intelligene, pp. 762–767 (1989)
Moriarty, D.E., Miikkulainen, R.: Forming neural networks through efficient and adaptive co-evolution. Evol. Comput. 5, 373–399 (1997)
Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 807–814 (2010)
Oymak, S.: Learning compact neural networks with regularization. In: International Conference on Machine Learning, pp. 3963–3972 (2018)
Papavasileiou, E., Cornelis, J., Jansen, B.: A systematic literature review of the successors of “neuroevolution of augmenting topologies.” Evol. Comput. 29, 1–73 (2021)
Park, J., Sandberg, I.W.: Universal approximation using radial-basis-function networks. Neural Comput. 3, 246–257 (1991)
Quinlan, J.R.: Induction of decision trees. Mach. Learn. 1, 81–106 (1986)
Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers Inc., San Francisco (1993)
Ramachandran, P., Zoph, B., Le, Q.V.: Searching for activation functions (2017). ar**v:1710.05941
Rawal, A., Miikkulainen, R.: Discovering gated recurrent neural network architectures. In: Iba, H., Noman, N., (eds.), Deep Neural Evolution - Deep Learning with Evolutionary Computation, pp. 233–251. Springer (2020)
Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4780–4789 (2019)
Real, E., Liang, C., So, D., Le, Q.: AutoML-Zero: evolving machine learning algorithms from scratch. In: Daumé III, H., Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, pp. 8007–8019 (2020)
Reed, R.: Pruning algorithms-a survey. IEEE Trans. Neural Netw. 4, 740–747 (1993)
Routley, N.: Visualizing the trillion-fold increase in computing power. Retrieved 11/17/2022 (2017)
Schaul, T., Schmidhuber, J.: Metalearning. Scholarpedia 5, 4650 (2010)
Schmidhuber, J.: Annotated history of modern ai and deep learning (2022). ar**v:2221.21127
Shahrzad, H., Hodjat, B., Dolle, C., Denissov, A., Lau, S., Goodhew, D., Dyer, J., Miikkulainen, R.: Enhanced optimization with composite objectives and novelty pulsation. In: Banzhaf, W., Goodman, E., Sheneman, L., Trujillo, L., Worzel, B. (eds.), Genetic Programming Theory and Practice XVII, pp. 275–293. Springer, New York (2020)
Shahrzad, H., Hodjat, B., Miikkulainen, R.: EVOTER: evolution of transparent explainable rule-sets (2022). ar**v:2204.10438
Sharma, S., Henderson, J., Ghosh, J.: CERTIFAI: a common framework to provide explanations and analyse the fairness and robustness of black-box models. In: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 166–172, New York 2020. Association for Computing Machinery (2020)
Shayani, H., Bentley, P.J., Tyrrell, A.M.: An fpga-based model suitable for evolution and development of spiking neural networks. In: Proceedings of the European Symposium on Artificial Neural Networks, pp. 197–202 (2008)
Sinha, A., Malo, P., Xu, P. and Deb, K.: A bilevel optimization approach to automated parameter tuning. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2014), pp. 847-854, Vancouver, BC, Canada (2014)
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: Proceedings of the 32nd International Conference on Machine Learning, vol. 37, pp. 2256–2265 (2015)
Srinivasan, S., Ramakrishnan, S.: Evolutionary multi objective optimization for rule mining: a review. Artif. Intell. Rev. 36, 205–248, 10 (2011)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)
Stanley, K.O.: Efficient Evolution of Neural Networks Through Complexification. PhD thesis, Department of Computer Sciences, The University of Texas at Austin (2004)
Stanley, K.O., D’Ambrosio, D.B., Gauci, J.: A hypercube-based encoding for evolving large-scale neural networks. Artif. Life 15, 185–212 (2009)
Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evolut. Comput. 10, 99–127 (2002)
Stanley, K.O., Miikkulainen, R.: Competitive coevolution through evolutionary complexification. J. Artif. Intell. Res. 21, 63–100 (2004)
Such, F.P., Madhavan, V., Conti, E., Lehman, J., Stanley, K.O. and Clune, J.: Deep neuroevolution: genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning (2017). ar**v:1712.06567
Suganuma, M., Kobayashi, M., Shirakawa, S., Nagao, T.: Evolution of deep convolutional neural networks using cartesian genetic programming. Evol. Comput. 28, 141–163 (2020)
Sun, Y., Xue, B., Zhang, M., Yen, G.G.: Evolving deep convolutional neural networks for image classification. IEEE Trans. Evol. Comput. 24, 394–407 (2020)
Sun, Y., Xue, B., Zhang, M., Yen, G.G., Lv, J.: Automatically designing cnn architectures using the genetic algorithm for image classification. IEEE Trans. Cybern. 50, 3840–3854 (2020)
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9 (2015)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818–2826 (2016)
Urbanowicz, R., Moore, J.: Learning classifier systems: a complete introduction, review, and roadmap. J. Artif. Evolut. Appl. 2009, 736398 (2009)
Urbanowicz, R.J., Bertasius, G. and Moore, J.H.: An extended michigan-style learning classifier system for flexible supervised learning, classification, and data mining. In: International Conference on Parallel Problem Solving from Nature, pp. 211–221. Springer (2014)
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Guyon, I., Von Luxburg, U., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R., (eds.), Advances in Neural Information Processing Systems, vol. 30, pp. 6000–6010 (2017)
Vinyals, O., Toshev, A., Bengio, S. and Erhan, D.: Show and tell: a neural image caption generator. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3156–3164 (2015)
Wu, X., Jia, L., Zhang, X., Chen, L., Liang, Y., Zhou, Y., Wu, C.: Neural architecture search based on cartesian genetic programming coding method (2021). ar**v:2103.07173
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Miikkulainen, R. (2024). Evolutionary Supervised Machine Learning. In: Banzhaf, W., Machado, P., Zhang, M. (eds) Handbook of Evolutionary Machine Learning. Genetic and Evolutionary Computation. Springer, Singapore. https://doi.org/10.1007/978-981-99-3814-8_2
Download citation
DOI: https://doi.org/10.1007/978-981-99-3814-8_2
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-3813-1
Online ISBN: 978-981-99-3814-8
eBook Packages: Computer ScienceComputer Science (R0)