SpringerLink
Log in

Self-controlled bio-inspired extreme learning machines for scalable regression and classification: a comprehensive analysis with some recommendations

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

The main goal of the current investigation is to answer some open questions regarding the applicability of bio-inspired techniques for simultaneous evolving and training of extreme learning machines (ELMs). Over the past decade, ELMs have successfully been applied to a wide range of regression and classification problems. However, in most of the reports, classical learning systems have been used for training ELMs which can result in cumbersome mathematical formulations and unstable solutions when handling large databases. Moreover, there is no guarantee for converging to global optimum solutions when traditional methods are used for the regularized learning of ELMs, especially when the variables in the database are correlated. To cope with such flaws, some researchers have used bio-inspired computation (BIC) for the topology evolving and parameter tuning of ELMs. However, the research stream still experiences its infancy. In an attempt to take a significant stride towards fulfilling the existing gap, the authors conduct a comprehensive analysis by adopting different BIC and classical methods for training ELMs. Based on the simulation results, some recommendations are given to guide researchers on using appropriate BICs for ELM training for small-scale, medium-scale, large-scale and very large-scale regression and classification problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  • Alexandre E, Cuadra L, Nieto-Borge JC, Candil-Garcia G, Del-Pino M, Salcedo-Sanz S (2015) A hybrid genetic algorithm-extreme learning machine approach for accurate significant wave height reconstruction. Ocean Model 92:115–123

    Article  Google Scholar 

  • An J, Kang Q, Wang L, Wu Q (2013) Mussels wandering optimization: an ecologically inspired algorithm for global optimization. Cogn Comput 5:188–199

    Article  Google Scholar 

  • Azad NL, Mozaffari A, Hedrick K (2015) A hybrid switching predictive controller based on bi-level kernel-based ELM and online trajectory builder for automotive coldstart emissions reduction. Neurocomputing. doi:10.1016/j.neucom.2015.08.070

  • Bartlett PL (1998) The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network. IEEE Trans Inf Theory 44(2):525–536

    Article  MathSciNet  MATH  Google Scholar 

  • Bazi Y, Alajlan N, Melgani F, AlHichri H, Malek S, Yager RR (2014) Differential evolution extreme learning machine for the classification of hyperspectral images. IEEE Geosci Remote Sens Lett 11(6):1066–1070

    Article  Google Scholar 

  • Cambria E, Huang GB, Chamara KLL et al (2013) Extreme learning machines [trends and controversies]. IEEE Intell Syst 28(6):30–59

    Article  Google Scholar 

  • Cao JW, Lin ZP, Huang GB (2011) Composite function wavelet neural networks with differential evolution and extreme learning machine. Neural Process Lett 33(3):251–265

    Article  Google Scholar 

  • Cao J, Zhi** L, Huang GB (2012) Self-adaptive evolutionary extreme learning machine. Neural Process Lett 36:285–305

    Article  Google Scholar 

  • Chang Y, Li Y, Wang F (2007) Soft sensing modeling based on extreme learning machine for biochemical processes. J Syst Simul 19(23):5587–5590

    Google Scholar 

  • Deng W, Zheng Q, Chen L (2009) Regularized extreme learning machine. In: IEEE symposium on computational intelligence and data mining, CIDM ’09, March 30th–April 2nd, pp 289–295

  • Ding S, Su C, Yu J et al (2011) An optimizing BP neural network algorithm based on genetic algorithm. Artif Intell Rev 36(2):153–162

    Article  Google Scholar 

  • Ding S, Xu L, Chunyang SuC et al (2012) An optimizing method of RBF neural network based on genetic algorithm. Neural Comput Appl 21(2):333–336

    Article  Google Scholar 

  • Ding S, Zhao H, Zhang Y, Xu Y, Xu X, Nie R (2013) Extreme learning machine: algorithm, theory and applications. Artif Intell Rev. doi:10.1007/s10462-013-9405-z

  • Diao R, Shen Q (2015) Nature inspired feature selection meta-heuristics. Artif Intell Rev 44:311–340

    Article  Google Scholar 

  • Engelbrecht AP (2015) Particle swarm optimization with crossover: a review and empirical analysis. Artif Intell Rev. doi:10.1007/s10462-015-9445-7

  • Esmin AAA, Coelho RA, Matwin S (2015) A review on particle swarm optimization algorithm and its variants to clustering high-dimensional data. Artif Intell Rev 44:23–45

    Article  Google Scholar 

  • Fathi A, Mozaffari A (2013) Identification of a dynamic model for shape memory alloy actuator using Hammerstein-Wiener gray box and mutable smart bee algorithm. Int J Intell Comput Cybern 6(4):328–357

    Article  MathSciNet  Google Scholar 

  • Fathi A, Mozaffari A (2014) Modeling a shape memory alloy actuator using an evolvable recursive black-box and hybrid heuristic algorithms inspired based on the annual migration of salmons in nature. Appl Soft Comput 14:229–251

    Article  Google Scholar 

  • Frank A, Asuncion A (2010) UCI machine learning repository. http://archive.ics.uci.edu/ml

  • Furtuna R, Curteanu S, Leon F (2012) Multi-objective optimization of a stacked neural network using an evolutionary hyper-heuristic. Appl Soft Comput 12(1):133–144

    Article  Google Scholar 

  • Han F, Yao HF, Ling QH (2013) An improved evolutionary extreme learning machine based on particle swarm optimization. Neurocomputing 116:87–93

    Article  Google Scholar 

  • Hastie T, Tibshirani R, Friedman J (2011) The elements of statistical learning. Springer, New York

    MATH  Google Scholar 

  • Huang GB, Zhu QY, Siew CK (2005) Extreme learning machine: theory and applications. Neurocomputing 70(1–3):489–501

    Google Scholar 

  • Huang GB, Chen L (2008) Enhanced random search based incremental extreme learning machine. Neurocomputing 71(16–18):3460–3468

    Article  Google Scholar 

  • Huang GB, Wang DH, Lan Y (2011) Extreme learning machines: a survey. Int J Mach Learn Cybern 2(2):107–122

    Article  Google Scholar 

  • Huang G, Huang GB, Song S, You K (2015a) Trends in extreme learning machines: a review. Neural Netw 61:32–48

    Article  MATH  Google Scholar 

  • Huang GB, Bai Z, Kasun LLC, Vong MC (2015b) Local receptive fields based extreme learning machine. IEEE Comput Intell Mag 10(2):18–29

    Article  Google Scholar 

  • Jordehi AR, Jasni J (2015) Particle swarm optimization for discrete optimization problems: a review. Artif Intell Rev 43(2):243–258

    Article  Google Scholar 

  • Karaboga D, Basturk B (2007) On the performance of artificial bee colony (ABC) algorithm. Appl Soft Comput 8:687–697

    Article  MATH  Google Scholar 

  • Kockanat S, Karaboga D (2015) The design of two-dimensional digital filters based on metaheuristic optimization algorithms: a review of the literature. Artif Intell Rev 44:265–287

    Article  Google Scholar 

  • Li X, Mao W, Jiang W (2014) Fast sparse approximation of extreme learning machine. Neurocomputing 128:96–103

    Article  Google Scholar 

  • Li S, Wang P, Goel L (2015) Short-term load forecasting by wavelet transform and evolutionary extreme learning machine. Electr Power Syst Res 122:96–103

    Article  Google Scholar 

  • Liao S, Feng C (2014) Meta-ELM: ELM with ELM hidden nodes. Neurocomputing 128:81–87

    Article  Google Scholar 

  • Liu N, Wang H (2013) Evolutionary extreme learning machine and its application to image analysis. J Signal Process Syst 73(1):73–81

    Article  Google Scholar 

  • Luo X, Chang X, Ban X (2015) Regression and classification using extreme learning machine based on \(L_{1}\)-norm and \(L_{2}\)-norm. Neurocomputing. doi:10.1016/j.neucom.2015.03.112

  • Marie-Sainte SL (2015) A survey of particle swarm optimization techniques for solving university examination timetabling problem. Artif Intell Rev. doi:10.1007/s10462-015-9437-7

  • Martinez JM, Escandell-Montero P, Soria-Olivas E, Martin-Guerrero JD, Magdalena-Benedito R, Gomez-Sanchis J (2011) Regularized extreme learning machine for regression problems. Neurocomputing 74:3716–3721

    Article  Google Scholar 

  • Miche Y, Eirola E, Bas P, Simula O, Jutten C, Lendasse A, Verleysen M (2010) Ensemble modeling with a constrained linear system of leave-one-out outputs. In: European symposium on artificial neural networks- computational intelligence and machine learning, Belgium, 28–30 April, pp 19–24

  • Miche Y, Sorjamaa A, Bas P, Simula O, Jutten C, lendasse A (2010b) OP-ELM: optimally pruned extreme learning machine. IEEE Trans Neural Netw 21(1):158–162

    Article  Google Scholar 

  • Miche Y, Heeswijk M, Bas P, Simula O, Lendasse A (2011) TROP-ELM: a double-regularized ELM using LARS and Tikhonov regularization. Neurocomputing 74(16):2413–2421

    Article  Google Scholar 

  • Michalewicz Z (1994) Genetic algorithms + data structures = evolution programs. Springer, New York

    Book  MATH  Google Scholar 

  • Mohapatra P, Chakravarty S, Dash PK (2015) An improved cuckoo search based extreme learning machine for medical data classification. Swarm Evol Comput 24:25–49

    Article  Google Scholar 

  • Mozaffari A, Fathi A, Behzadipour S (2012) The great salmon run: a novel bio-inspired algorithm for artificial system design and optimization. Int J Bio-Inspired Comput 4(5):286–301

    Article  Google Scholar 

  • Mozaffari A, Ramiar A, Fathi A (2013) Optimal design of classic Atkinson engine with dynamic specific heat using adaptive neuro-fuzzy inference system and mutable smart bee algorithm. Swarm Evol Comput 12:74–91

    Article  Google Scholar 

  • Mozaffari A, Azad NL (2014) Optimally pruned extreme learning machine with ensemble of regularization techniques and negative correlation penalty applied to automotive engine coldstart hydrocarbon emission identification. Neurocomputing 131:143–156

    Article  Google Scholar 

  • Mozaffari A, Azad NL, Emami M, Fathi A (2015) Mixed continuous/binary quantum-inspired learning system with non-negative least square optimisation for automated design of regularised ensemble extreme learning machines. Exp Theor Artif Intell. doi:10.1080/0952813X.2015.1020574

  • Neshat M, Sepidnam G, Sargolzaei M, Najaran Toosi A (2012) Artificial fish swarm algorithm: a survey of the state-of-the-art, hybridization, combinatorial and indicative applications. Artif Intell Rev. doi:10.1007/s10462-012-9342-2

  • Neri F, Tirronen V (2009) Scale factor local search in differential evolution. Memet Comput 1(2):153–171

    Article  Google Scholar 

  • Prakasam A, Savarimuthu N (2015) Metaheuristic algorithms and probabilistic behaviour: a comprehensive analysis of ant colony optimization and its variants. Artif Intell Rev. doi:10.1007/s10462-015-9441-y

  • Qu Y, Shang C, Wu W, Shen Q (2011) Evolutionary fuzzy extreme learning machine for mammographic risk analysis. Int J Fuzzy Syst 13(4):282–291

    MathSciNet  Google Scholar 

  • Rashedi E, Nezamabadipour H, Saryazdi S (2009) GSA: a gravitational search algorithm. Inf Sci 179(13):2232–2248

    Article  MATH  Google Scholar 

  • Schmidt M (2005) Least squares optimization with \(\text{ L }_{1}\)-norm regularization, CS542B Project Report, December

  • Shao Z, Er MJ, Wang N (2015) An effective semi-cross-validation model selection method for extreme learning machine with ridge regression. Neurocomputing 151:933–942

    Article  Google Scholar 

  • Silva DNG, Pacifico LDS, Ludermir TB (2011) An evolutionary extreme learning machine based on group search optimizer. In: IEEE Congress on Evolutionary Computation, pp 574–580

  • Storn R, Price K (1997) Differential evolution: a simple and efficient heuristic for global optimization over continuous spaces. J Glob Optim 11:341–459

    Article  MathSciNet  MATH  Google Scholar 

  • Tang J, Deng C, Huang GB (2015) Extreme learning machine for multilayer perceptron. IEEE Trans Neural Netw Learn Syst. doi:10.1109/TNNLS.2015.2424995

  • Taormina R, Chau KW (2015) Data-driven input variable selection for rainfall-runoff modeling using binary-coded particle swarm optimization and extreme learning machines. J Hydrol. doi:10.1016/j.jhydrol.2015.08.022

  • Teoh CK, Wibowo A, Ngadiman MS (2015) Review of state of the art for metaheuristic techniques in academic scheduling problems. Artif Intell Rev 44:1–21

    Article  Google Scholar 

  • Wang D, Alhamdoosh M (2013) Evolutionary extreme learning machine ensembles with size control. Neurocomputing 102:98–110

    Article  Google Scholar 

  • Wang N, Er MJ, Han M (2014a) Parsimonious extreme learning machine using recursive orthogonal least squares. IEEE Trans Neural Netw Learn Syst 25(10):1828–1841

    Article  Google Scholar 

  • Wang N, Han M, Dong N, Er MJ (2014b) Constructive multi-output extreme learning machine with application to large tanker motion dynamics identification. Neurocomputing 128:59–72

    Article  Google Scholar 

  • Wang N, Sun JC, Er MJ, Liu YC (2015) A novel extreme learning control framework of unmanned surface vehicles. IEEE Trans Cybern. doi:10.1109/TCYB.2015.2423635

  • Wang N, Er MJ, Han M (2015b) Generalized single-hidden layer feedforward networks for regression problems. IEEE Trans Neural Netw Learn Syst 26(6):1161–1176

    Article  MathSciNet  Google Scholar 

  • Wang Y, Li D, Du Y, Pan Z (2015) Anomaly detection in traffic using \(L_{1}\)-norm minimization extreme learning machine. Neurocomputing 149:415–425

    Article  Google Scholar 

  • Wolpert DH, Macready WG (1997) No free lunch theorems for optimization. IEEE Trans Evol Comput 1:67–82

    Article  Google Scholar 

  • **ao J, Zhou J, Li C, **ao H, Zhang W, Zhu W (2013) Multi-fault classification based on the two-stage evolutionary extreme learning machine and improved artificial bee colony algorithm. J Mech Eng Sci. doi:10.1177/0954406213496968

  • **n J, Wang Z, Qu L, Wang G (2013) Elastic extreme learning machine for big data classification. Neurocomputing 149:464–471

    Article  Google Scholar 

  • Xu Y, Shu Y (2006) Evolutionary extreme learning machine—based on particle swarm optimization. In: Advances in Neural Networks, Lecture Notes in Computer Science, vol. 3971, pp 644–652

  • Xue X, Yao M, Wu Z, Yang J (2014) Genetic ensemble of extreme learning machine. Neurocomputing 129:175–184

    Article  Google Scholar 

  • Yang XS (2008) Nature-inspired metaheuristic algorithms. Luniver Press, Beckington

    Google Scholar 

  • Yang XS, Cui ZH, **ao RB, Gandomi AH, Karamanoglu M (2013) Swarm intelligence and bio-inspired computation: theory and applications. Elsevier, London

    Book  Google Scholar 

  • Yang H, Yi J, Zhao J, Dong Z (2013) Extreme learning machine based genetic algorithm and its application in power system economic dispatch. Neurocomputing 102:154–162

    Article  Google Scholar 

  • Yu J, Wang S, ** L (2008) Evolving artificial neural networks using an improved PSO and DPSO. Neurocomputing 71(4–6):1054–1060

    Article  Google Scholar 

  • Yu W, Zhuang F, He Q, Shi Z (2015) Learning deep representations via extreme learning machines. Neurocomputing 149:308–315

    Article  Google Scholar 

  • Zhao GP, Hen ZQ, Miao CY, Man ZH (2009) On improving the conditioning of extreme learning machine: a linear case. In: 7th International conference on information, communications and signal processing, 2009 (ICICS2009), Maucu, China, December 8th–10th, pp 1–5

  • Zhu QY, Qin AK, Suganthan PN, Huang GB (2005) Evolutionary extreme learning machine. Pattern Recogn 38:1759–1763

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Systems Design Engineering Department, University of Waterloo, Waterloo, ON, N2L 3G1, Canada

    Ahmad Mozaffari & Nasser L. Azad

Authors
  1. Ahmad Mozaffari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nasser L. Azad
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ahmad Mozaffari.

Appendix: Interactions of the devised controller and BICs

Appendix: Interactions of the devised controller and BICs

In this section, the detailed information regarding the performance of the devised controller and its impact on the performance of BICs is given. In Tables 12, 13 and 14, some parameters are given which should be defined for clarifications:

  • Activation of the first strategy: This column indicates how many times the ill-conditioned ELMs are detected and discarded from the population.

  • Activation of the second strategy: This column indicates how many times the controller spurs the exploration/exploitation over the optimization procedure.

  • Activation of the third strategy: This column indicates how many times the controller selects the fitter ELM based on its generalization capability (Eq. (40)).

  • Performance and robustness improvement: The possible improvement in the performance and robustness of the self-controlled BICs in comparison to the basic BICs are listed in the last two columns.

Table 12 The performance of the controller for \(\hbox {R}_{1}\) to \(\hbox {R}_{6}\) benchmarks
Full size table
Table 13 The performance of the controller for \(\hbox {R}_{7}\) to \(\hbox {R}_{12}\) benchmarks
Full size table
Table 14 The performance of the controller for \(\hbox {C}_{1}\) to \(\hbox {C}_{6}\) benchmarks
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mozaffari, A., Azad, N.L. Self-controlled bio-inspired extreme learning machines for scalable regression and classification: a comprehensive analysis with some recommendations. Artif Intell Rev 46, 167–223 (2016). https://doi.org/10.1007/s10462-016-9461-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-016-9461-2

Keywords

Search

Navigation