Evolutionary Supervised Machine Learning

  • Chapter
  • First Online:
Handbook of Evolutionary Machine Learning

Part of the book series: Genetic and Evolutionary Computation ((GEVO))

  • 1610 Accesses

Abstract

This chapter provides an overview of evolutionary approaches to supervised learning. It starts with the definition and scope of the opportunity, and then reviews three main areas: evolving general neural network designs, evolving solutions that are explainable, and forming a synergy of evolutionary and gradient-based methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (France)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 181.89
Price includes VAT (France)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
EUR 232.09
Price includes VAT (France)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Aharonov-Barki, R., Beker, T., Ruppin, E.: Emergence of memory-driven command neurons in evolved artificial agents. Neural Comput. 13, 691–716 (2001)

    Article  MATH  Google Scholar 

  2. Aitkenhead, M.J.: A co-evolving decision tree classification method. Expert Syst. Appl. 34, 19–25 (2008)

    Article  Google Scholar 

  3. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: Precup, D., Teh, Y.W. (eds.), Proceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 214–223 (2017)

    Google Scholar 

  4. Barros, R.C., Basgalupp, M.P., De Carvalho, A.C., Freitas, A.A.: A survey of evolutionary algorithms for decision-tree induction. IEEE Trans. Syst., Man, Cybern., Part C (Appl. Rev.) 42, 291–312 (2012)

    Google Scholar 

  5. Bi, Y., Xue, B., Zhang, M.: Genetic programming-based evolutionary deep learning for data-efficient image classification. IEEE Trans. Evolut. Comput. (2022). https://doi.org/10.1109/TEVC.2022.3214503

  6. Bingham, G., Macke, W., Miikkulainen, R.: Evolutionary optimization of deep learning activation functions. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 289–296 (2020)

    Google Scholar 

  7. Bingham, G., Miikkulainen, R.: Discovering parametric activation functions. Neural Netw. 148, 48–65 (2022)

    Article  Google Scholar 

  8. Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001)

    Article  MATH  Google Scholar 

  9. Breiman, L., Friedman, J., Stone, C.J., Olshen, R.A.: Classification and Regression Trees. Chapman and Hall/CRC (1984)

    Google Scholar 

  10. Butz, M.V., Lanzi, P.L., Wilson, S.W.: Function approximation with xcs: Hyperellipsoidal conditions, recursive least squares, and compaction. IEEE Trans. Evolut. Comput. 12, 355–376 (2008)

    Article  Google Scholar 

  11. Canatar, A., Bordelon, B., Pehlevan, C.: Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nat. Commun. 12, 1914 (2021)

    Article  Google Scholar 

  12. Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math. Control Signals Syst. 2, 303–314 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  13. Dai, E., Zhao, T., Zhu, H., Xu, J., Guo, Z., Liu, H., Tang, J., Wang, S.: A comprehensive survey on trustworthy graph neural networks: privacy, robustness, fairness, and explainability. ar**v:2104.05605, 2020

  14. De Jong, K.: Learning with genetic algorithms: an overview. Mach. Learn. 3, 121–138 10 (1988)

    Google Scholar 

  15. Deb, K., Myburgh, C.: A population-based fast algorithm for a billion-dimensional resource allocation problem with integer variables. Eur. J. Oper. Res. 261, 460–474 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  16. Dolotov, E., Zolotykh, N.Y.: Evolutionary algorithms for constructing an ensemble of decision trees (2020). ar**v:2002.00721

  17. Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey. J. Mach. Learn. Res. 20, 1–21 (2019)

    MathSciNet  MATH  Google Scholar 

  18. Gaier, A., Ha, D.: Weight agnostic neural networks. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.), Advances in Neural Information Processing Systems 5364–5378 (2019)

    Google Scholar 

  19. Ganon, Z., Keinan, A., Ruppin, E.: Evolutionary network minimization: adaptive implicit pruning of successful agents. In: Banzhaf, W., Ziegler, J., Christaller, T., Dittrich, P., Kim, J.T. (eds.) Advances in Artificial Life, pp. 319–327. Springer, Berlin (2003)

    Chapter  Google Scholar 

  20. Gao, B., Gouk, H., Hospedales, T.M.: Searching for robustness: loss learning for noisy classification tasks. IEEE/CVF International Conference on Computer Vision, pp. 6650–6659 (2021)

    Google Scholar 

  21. Gomez, F., Miikkulainen, R.: Incremental evolution of complex general behavior. Adapt. Behav. 5, 317–342 (1997)

    Article  Google Scholar 

  22. Gomez, F., Schmidhuber, J., Miikkulainen, R. and Mitchell, M.: Accelerated neural evolution through cooperatively coevolved synapses. J. Mach. Learn. Res. 937–965 (2008)

    Google Scholar 

  23. Gonzalez, S., Kant, M., Miikkulainen, R.: Evolving GAN formulations for higher quality image synthesis. In: Kozma, R., Alippi, C., Choe, Y., Morabito, F.C. (eds.) Artificial Intelligence in the Age of Neural Networks and Brain Computing, 2nd edn. Elsevier, New York (2023)

    Google Scholar 

  24. Gonzalez, S., Landgraf, J., Miikkulainen, R.: Faster training by selecting samples using embeddings. In: Proceedings of the 2019 International Joint Conference on Neural Networks, pp. 1–7 (2019)

    Google Scholar 

  25. Gonzalez, S., Miikkulainen, R.: Improved training speed, accuracy, and data utilization through loss function optimization. In: Proceedings of the 2020 IEEE Congress on Evolutionary Computation (CEC), pp. 1–8 (2020)

    Google Scholar 

  26. Gonzalez, S., Miikkulainen, R.: Effective regularization through loss-function metalearning (2021). ar**v:2010.00788

  27. Gonzalez, S., Miikkulainen, R.: Optimizing loss functions through multivariate Taylor polynomial parameterization. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 305–313 (2021)

    Google Scholar 

  28. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. and Bengio, Y.: Generative adversarial nets. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q., (eds.), Advances in Neural Information Processing Systems 27, pp. 2672–2680 (2014)

    Google Scholar 

  29. Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evol. Comput. 9, 159–195 (2001)

    Article  Google Scholar 

  30. Hanson, S.J., Pratt, L.Y.: Comparing biases for minimal network construction with back-propagation. In: Proceedings of the 1st International Conference on Neural Information Processing Systems, pp. 177–185. MIT Press, Cambridge (1988)

    Google Scholar 

  31. Hayes-Roth, F.: Rule-based systems. Commun. ACM 28, 921–932 (1985)

    Article  Google Scholar 

  32. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  33. Hemberg, E., Veeramachaneni, K., Wanigarekara, P., Shahrzad, H., Hodjat, B., O’Reilly, U.-M.: Learning decision lists with lagged physiological time series. In: Workshop on Data Mining for Medicine and Healthcare, 14th SIAM International Conference on Data Mining, pp. 82–87 (2014)

    Google Scholar 

  34. Holland, J.H.: Esca** brittleness: the possibilities of general purpose learning algorithms applied to parallel rule-based systems. In: Michalski, R.S., Carbonell, J.G., Mitchell, T.M. (eds.) Machine Learning: An Artificial Intelligence Approach, vol. 2, pp. 593–623. Morgan Kaufmann, Los Altos (1986)

    Google Scholar 

  35. Hoos, H.: Programming by optimization. Commun. ACM 55, 70–80 (2012)

    Article  Google Scholar 

  36. Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw. 2, 359–366 (1989)

    Article  MATH  Google Scholar 

  37. Huang, X., Kroening, D., Ruan, W., Sharp, J., Sun, Y., Thamo, E., Min, W., Yi, X.: A survey of safety and trustworthiness of deep neural networks: verification, testing, adversarial attack and defence, and interpretability. Comput. Sci. Rev. 37, 100270 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  38. Ijspeert, A.J.: Central pattern generators for locomotion control in animals and robots: A review. Neural Netw. 21, 642–653 (2008)

    Article  Google Scholar 

  39. Jankowski, D., Jackowski, K.: Evolutionary algorithm for decision tree induction. In: Saeed, K., Snášel, V. (eds.) Computer Information Systems and Industrial Management, pp. 23–32. Springer, Berlin (2014)

    Chapter  Google Scholar 

  40. Kashtan, N., Alon, U.: Spontaneous evolution of modularity and network motifs. Proc. Natl. Acad. Sci. 102, 13773–13778 (2005)

    Article  Google Scholar 

  41. Langdon, W.B., Poli, R., McPhee, N.F., Koza, J.R.: Genetic programming: An introduction and tutorial, with a survey of techniques and applications. In: Fulcher, J., Jain, L.C. (eds.) Computational Intelligence: A Compendium, pp. 927–1028. Springer, Berlin (2008)

    Chapter  Google Scholar 

  42. Liang, J., Meyerson, E. and Miikkulainen, R.: Evolutionary architecture search for deep multitask networks. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 466–473 (2018)

    Google Scholar 

  43. Liang, J., Gonzalez, S., Shahrzad, H., Miikkulainen, R.: Regularized evolutionary population-based training. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 323–331 (2021)

    Google Scholar 

  44. Liang, J., Meyerson, E., Hodjat, B., Fink, D., Mutch, K. and Miikkulainen, R.: Evolutionary neural AutoML for deep learning. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2019), pp. 401–409 (2019)

    Google Scholar 

  45. Liang, J.Z., Miikkulainen, R.: Evolutionary bilevel optimization for complex control tasks. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2015), pp. 871–878 (2015)

    Google Scholar 

  46. Light, W.: Ridge functions, sigmoidal functions and neural networks. In: Approximation Theory VII, pp. 158–201. Academic, Boston (1992)

    Google Scholar 

  47. Liu, Y., Sun, Y., Xue, B., Zhang, M., Yen, G.G., Tan, K.C.: A survey on evolutionary neural architecture search. IEEE Trans. Neural Netw. Learn. Syst. 1–21 (2021)

    Google Scholar 

  48. Liu, Z., Zhang, X., Wang, S., Ma, S., Gao, W.: Evolutionary quantization of neural networks with mixed-precision. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2785–2789 (2021)

    Google Scholar 

  49. Lu, Z., Deb, K., Goodman, E., Banzhaf, W., Boddeti, V.N.: Nsganetv2: evolutionary multi-objective surrogate-assisted neural architecture search. In: European Conference on Computer Vision ECCV-2020, LNCS, vol. 12346, pp. 35–51 (2020)

    Google Scholar 

  50. Mao, X., Li, Q., **e, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2794–2802 (2017)

    Google Scholar 

  51. Meyerson, E., Miikkulainen, R.: Pseudo-task augmentation: from deep multitask learning to intratask sharing—and back. In: Proceedings of the 35th International Conference on Machine Learning, pp. 739–748 (2018)

    Google Scholar 

  52. Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., Hodjat, B.: Evolving deep neural networks. In: Morabito, C.F., Alippi, C., Choe, Y., Kozma, R. (eds.) Artificial Intelligence in the Age of Neural Networks and Brain Computing, 2nd edn., pp. 293–312. Elsevier, New York (2023)

    Google Scholar 

  53. Miikkulainen, R., Meyerson, E., Qiu, X., Sinha, U., Kumar, R., Hofmann, K., Yan, Y.M., Ye, M., Yang, J., Caiazza, D. and Brown, S.M.: Evaluating medical aesthetics treatments through evolved age-estimation models. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1009–1017 (2021)

    Google Scholar 

  54. Montana, D.J., Davis, L.: Training feedforward neural networks using genetic algorithms. In: International Joint Conference on Artificial Intelligene, pp. 762–767 (1989)

    Google Scholar 

  55. Moriarty, D.E., Miikkulainen, R.: Forming neural networks through efficient and adaptive co-evolution. Evol. Comput. 5, 373–399 (1997)

    Article  Google Scholar 

  56. Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 807–814 (2010)

    Google Scholar 

  57. Oymak, S.: Learning compact neural networks with regularization. In: International Conference on Machine Learning, pp. 3963–3972 (2018)

    Google Scholar 

  58. Papavasileiou, E., Cornelis, J., Jansen, B.: A systematic literature review of the successors of “neuroevolution of augmenting topologies.” Evol. Comput. 29, 1–73 (2021)

    Article  Google Scholar 

  59. Park, J., Sandberg, I.W.: Universal approximation using radial-basis-function networks. Neural Comput. 3, 246–257 (1991)

    Article  Google Scholar 

  60. Quinlan, J.R.: Induction of decision trees. Mach. Learn. 1, 81–106 (1986)

    Article  Google Scholar 

  61. Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers Inc., San Francisco (1993)

    Google Scholar 

  62. Ramachandran, P., Zoph, B., Le, Q.V.: Searching for activation functions (2017). ar**v:1710.05941

  63. Rawal, A., Miikkulainen, R.: Discovering gated recurrent neural network architectures. In: Iba, H., Noman, N., (eds.), Deep Neural Evolution - Deep Learning with Evolutionary Computation, pp. 233–251. Springer (2020)

    Google Scholar 

  64. Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4780–4789 (2019)

    Google Scholar 

  65. Real, E., Liang, C., So, D., Le, Q.: AutoML-Zero: evolving machine learning algorithms from scratch. In: Daumé III, H., Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, pp. 8007–8019 (2020)

    Google Scholar 

  66. Reed, R.: Pruning algorithms-a survey. IEEE Trans. Neural Netw. 4, 740–747 (1993)

    Article  Google Scholar 

  67. Routley, N.: Visualizing the trillion-fold increase in computing power. Retrieved 11/17/2022 (2017)

    Google Scholar 

  68. Schaul, T., Schmidhuber, J.: Metalearning. Scholarpedia 5, 4650 (2010)

    Article  Google Scholar 

  69. Schmidhuber, J.: Annotated history of modern ai and deep learning (2022). ar**v:2221.21127

  70. Shahrzad, H., Hodjat, B., Dolle, C., Denissov, A., Lau, S., Goodhew, D., Dyer, J., Miikkulainen, R.: Enhanced optimization with composite objectives and novelty pulsation. In: Banzhaf, W., Goodman, E., Sheneman, L., Trujillo, L., Worzel, B. (eds.), Genetic Programming Theory and Practice XVII, pp. 275–293. Springer, New York (2020)

    Google Scholar 

  71. Shahrzad, H., Hodjat, B., Miikkulainen, R.: EVOTER: evolution of transparent explainable rule-sets (2022). ar**v:2204.10438

  72. Sharma, S., Henderson, J., Ghosh, J.: CERTIFAI: a common framework to provide explanations and analyse the fairness and robustness of black-box models. In: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 166–172, New York 2020. Association for Computing Machinery (2020)

    Google Scholar 

  73. Shayani, H., Bentley, P.J., Tyrrell, A.M.: An fpga-based model suitable for evolution and development of spiking neural networks. In: Proceedings of the European Symposium on Artificial Neural Networks, pp. 197–202 (2008)

    Google Scholar 

  74. Sinha, A., Malo, P., Xu, P. and Deb, K.: A bilevel optimization approach to automated parameter tuning. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2014), pp. 847-854, Vancouver, BC, Canada (2014)

    Google Scholar 

  75. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: Proceedings of the 32nd International Conference on Machine Learning, vol. 37, pp. 2256–2265 (2015)

    Google Scholar 

  76. Srinivasan, S., Ramakrishnan, S.: Evolutionary multi objective optimization for rule mining: a review. Artif. Intell. Rev. 36, 205–248, 10 (2011)

    Google Scholar 

  77. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)

    MathSciNet  MATH  Google Scholar 

  78. Stanley, K.O.: Efficient Evolution of Neural Networks Through Complexification. PhD thesis, Department of Computer Sciences, The University of Texas at Austin (2004)

    Google Scholar 

  79. Stanley, K.O., D’Ambrosio, D.B., Gauci, J.: A hypercube-based encoding for evolving large-scale neural networks. Artif. Life 15, 185–212 (2009)

    Article  Google Scholar 

  80. Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evolut. Comput. 10, 99–127 (2002)

    Article  Google Scholar 

  81. Stanley, K.O., Miikkulainen, R.: Competitive coevolution through evolutionary complexification. J. Artif. Intell. Res. 21, 63–100 (2004)

    Article  Google Scholar 

  82. Such, F.P., Madhavan, V., Conti, E., Lehman, J., Stanley, K.O. and Clune, J.: Deep neuroevolution: genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning (2017). ar**v:1712.06567

  83. Suganuma, M., Kobayashi, M., Shirakawa, S., Nagao, T.: Evolution of deep convolutional neural networks using cartesian genetic programming. Evol. Comput. 28, 141–163 (2020)

    Article  Google Scholar 

  84. Sun, Y., Xue, B., Zhang, M., Yen, G.G.: Evolving deep convolutional neural networks for image classification. IEEE Trans. Evol. Comput. 24, 394–407 (2020)

    Article  Google Scholar 

  85. Sun, Y., Xue, B., Zhang, M., Yen, G.G., Lv, J.: Automatically designing cnn architectures using the genetic algorithm for image classification. IEEE Trans. Cybern. 50, 3840–3854 (2020)

    Article  Google Scholar 

  86. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9 (2015)

    Google Scholar 

  87. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818–2826 (2016)

    Google Scholar 

  88. Urbanowicz, R., Moore, J.: Learning classifier systems: a complete introduction, review, and roadmap. J. Artif. Evolut. Appl. 2009, 736398 (2009)

    Google Scholar 

  89. Urbanowicz, R.J., Bertasius, G. and Moore, J.H.: An extended michigan-style learning classifier system for flexible supervised learning, classification, and data mining. In: International Conference on Parallel Problem Solving from Nature, pp. 211–221. Springer (2014)

    Google Scholar 

  90. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Guyon, I., Von Luxburg, U., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R., (eds.), Advances in Neural Information Processing Systems, vol. 30, pp. 6000–6010 (2017)

    Google Scholar 

  91. Vinyals, O., Toshev, A., Bengio, S. and Erhan, D.: Show and tell: a neural image caption generator. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3156–3164 (2015)

    Google Scholar 

  92. Wu, X., Jia, L., Zhang, X., Chen, L., Liang, Y., Zhou, Y., Wu, C.: Neural architecture search based on cartesian genetic programming coding method (2021). ar**v:2103.07173

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Risto Miikkulainen .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Miikkulainen, R. (2024). Evolutionary Supervised Machine Learning. In: Banzhaf, W., Machado, P., Zhang, M. (eds) Handbook of Evolutionary Machine Learning. Genetic and Evolutionary Computation. Springer, Singapore. https://doi.org/10.1007/978-981-99-3814-8_2

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-3814-8_2

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-3813-1

  • Online ISBN: 978-981-99-3814-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Navigation