SpringerLink
Log in

Convergence of RBF Networks Regression Function Estimates and Classifiers

  • Conference paper
  • First Online:
Artificial Intelligence and Soft Computing (ICAISC 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13588))

Included in the following conference series:

  • 475 Accesses

Abstract

In the paper convergence of the RBF network regression estimates and classifiers with so-called regular radial kernels is investigated. The parameters of the network are trained by minimizing the empirical risk on the training data. We analyze MISE convergence by utilizing the machine learning theory techniques such as VC dimension and covering numbers and the error bounds involving them. The performance of the normalized RBF network regression estimates is also tested in simulations.

A. Krzyżak—Research of the first author was supported by the Alexander von Humboldt Foundation and the Natural Sciences and Engineering Research Council of Canada under Grant RGPIN-2015-06412. He carried out this research at the Westpomeranian University of Technology during his sabbatical leave from Concordia University.

T. Gałkowski—Research of the second author was supported by the project financed with the program of the Polish Minister of Science and Higher Education under the name “Regional Initiative of Excellence" in the years 2019 - 2022 project number 020/RID/2018/19 the amount of financing 12,000,000 PLN.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Anthony, M., Bartlett, P.L.: Neural Network Learning: Theoretical Foundations. Cambridge University Press, Cambridge (1999)

    Book  MATH  Google Scholar 

  2. Barron, A.R.: Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inf. Theory 39, 930–945 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  3. Bauer, B., Kohler, M.: On deep learning as a remedy for the curse of dimensionality in nonparametric regression. Ann. Stat. 47(4), 2261–2285 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  4. Beirlant, J., Györfi, L.: On the asymptotic \({L}_2\)-error in partitioning regression estimation. J. Stat. Plan. Inference 71, 93–107 (1998)

    Article  MATH  Google Scholar 

  5. Bologna, G., Hayashi, Y.: Characterization of symbolic rules embedded in deep DIMLP networks: a challenge to transparency of deep learning. J. Artifi. Intell. Soft Comput. 7(4), 265–286 (2017)

    Google Scholar 

  6. Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001)

    Article  MATH  Google Scholar 

  7. Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J.: Classification and Regression Trees, Wadsworth Advanced Books and Software, CA, Belmont (1984)

    Google Scholar 

  8. Broomhead, D.S., Lowe, D.: Multivariable functional interpolation and adaptive networks. Complex Syst. 2, 321–323 (1988)

    MathSciNet  MATH  Google Scholar 

  9. Cybenko, G.: Approximations by superpositions of sigmoidal functions. Math. Control Signals Syst. 2, 303–314 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  10. Biau, G., Devroye, L.: Lectures on the Nearest Neighbor Method. SSDS, Springer, Cham (2015). https://doi.org/10.1007/978-3-319-25388-6

    Book  MATH  Google Scholar 

  11. Devroye, L., Györfi, L., Lugosi, G.: Probabilistic Theory of Pattern Recognition. Springer-Verlag, New York (1996). https://doi.org/10.1007/978-1-4612-0711-5

  12. Devroye, L., Györfi, L., Krzyżak, A., Lugosi, G.: On the strong universal consistency of nearest neighbor regression function estimates. Ann. Stat. 22, 1371–1385 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  13. Devroye, L., Krzyżak, A.: An equivalence theorem for \(L_1\) convergence of the kernel regression estimate. J. Stat. Plan. Inference 23, 71–82 (1989)

    Article  MATH  Google Scholar 

  14. Devroye, L.P., Wagner, T.J.: Distribution-free consistency results in nonparametric discrimination and regression function estimation. Ann. Stat. 8, 231–239 (1980)

    Article  MathSciNet  MATH  Google Scholar 

  15. Duda, P., Rutkowski, L., Jaworski, M., Rutkowska, D.: On the Parzen kernel-based probability density function learning procedures over time-varying streaming data with applications to pattern classification. IEEE Trans. Cybern. 50(4), 1683–1696 (2020)

    Article  Google Scholar 

  16. Duda, R., Hart, P., Stork, D.: Pattern Classification, Wiley, 2nd ed. (2001)

    Google Scholar 

  17. Gałkowski, T., Krzyżak, A., Patora-Wysocka, Z., Filutowicz, Z., Wang, L.: A new approach to detection of changes in multidimensional patterns - Part II. J. Artifi. Intell. Soft Comput. Res. 11(3), 217–227 (2021)

    Article  Google Scholar 

  18. Girosi, F., Anzellotti, G.: Rates of convergence for radial basis functions and neural networks. In: Mammone, R.J. (ed.) Artificial Neural Networks for Speech and Vision, pp. 97–113. Chapman and Hall, London (1993)

    Google Scholar 

  19. Girosi, F., Jones, M., Poggio, T.: Regularization theory and neural network architectures. Neural Comput. 7, 219–267 (1995)

    Article  Google Scholar 

  20. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. The MIT Press (2016)

    Google Scholar 

  21. Greblicki, W.: Asymptotically Optimal Probabilistic Algorithms for Pattern Recognition and Identification. Monografie No. 3. Prace Naukowe Instytutu Cybernetyki Technicznej Politechniki Wroclawskiej, Nr. 18. Wroclaw, Poland (1974)

    Google Scholar 

  22. Gramacki, A., Gramacki, J.: FFT-based fast bandwidth selector for multivariate kernel density estimation. Comput. Stat. Data Anal. 106, 27–45 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  23. Greblicki, W., Pawlak, M.: Fourier and Hermite series estimates of regression functions. Ann. Inst. Stat. Math. 37, 443–454 (1985)

    Article  MathSciNet  MATH  Google Scholar 

  24. Greblicki, W., Pawlak, M.: Necessary and sufficient conditions for Bayes risk consistency of a recursive kernel classification rule. IEEE Trans. Inf. Theory IT-33, 408–412 (1987)

    Google Scholar 

  25. Györfi, L., Kohler, M., Krzyżak, A., Walk, H.: A Distribution-Free Theory of Nonparametric Regression. Springer Verlag, New York (2002). https://doi.org/10.1007/b97848

  26. Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning. SSS, Springer, New York (2009). https://doi.org/10.1007/978-0-387-84858-7

    Book  MATH  Google Scholar 

  27. Haykin, S.O.: Neural Networks and Learning Machines, 3rd edn. Prentice-Hall, New York (2008)

    Google Scholar 

  28. Homenda, W., Jastrzȩbska, A., Pedrycz, W., Yu, F.: Combining classifiers for foreign pattern rejection. J. Artif. Intell. Soft Comput. Res. 10(2), 75–94 (2020)

    Article  Google Scholar 

  29. Hornik, K., Stinchocombe, S., White, H.: Multilayer feed-forward networks are universal approximators. Neural Netw. 2, 359–366 (1989)

    Article  MATH  Google Scholar 

  30. Jordanov, I., Petrov, N., Petrozziello, A.: Classifiers accuracy improvement based on missing data imputation. J. Artifi. Intell. Soft Comput. 8(8), 31–48 (2018)

    Google Scholar 

  31. Kohler, M., Krzyżak, A.: Nonparametric regression based on hierarchical interaction models. IEEE Trans. Inf. Theory 63, 1620–1630 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  32. Krzyżak, A.: The rates of convergence of kernel regression estimates and classification rules. IEEE Trans. Inf. Theory IT-32, 668–679 (1986)

    Google Scholar 

  33. Krzyżak, A.: Global convergence of recursive kernel regression estimates with applications in classification and nonlinear system estimation. IEEE Trans. Inf. Theory IT-38, 1323–1338 (1992)

    Google Scholar 

  34. Krzyżak, A., Linder, T., Lugosi, G.: Nonparametric estimation and classification using radial basis function nets and empirical risk minimization. IEEE Trans. Neural Netw. 7(2), 475–487 (1996)

    Article  Google Scholar 

  35. Krzyżak, A., Linder, T.: Radial basis function networks and complexity regularization in function learning. IEEE Trans. Neural Netw. 9(2), 247–256 (1998)

    Article  Google Scholar 

  36. Krzyżak, A., Niemann, H.: Convergence and rates of convergence of radial basis functions networks in function learning. Nonlinear Anal. 47, 281–292 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  37. Krzyżak, A., Partyka, M.: Convergence and rates of convergence of normalized recursive radial basis functions networks in function learning and classification. In: Proceedings of 16th International Conference on Artificial Intelligence and Soft Computing, Zakopane, Poland, 11–15 June 2017. Lecture Notes on Artificial Intelligence and Soft Computing, Part I, LNAI, vol. 10245, pp. 107–117. Springer-Verlag, (2017)

    Google Scholar 

  38. Krzyżak, A., Partyka, M.: Learning and convergence of the normalized radial basis functions networks. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds.) ICAISC 2018. LNCS (LNAI), vol. 10841, pp. 118–129. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91253-0_12

    Chapter  Google Scholar 

  39. Krzyżak, A., Partyka, M.: On Learning and convergence of rbf networks in regression estimation and classification. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds.) ICAISC 2019. LNCS (LNAI), vol. 11508, pp. 131–142. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20912-4_13

    Chapter  Google Scholar 

  40. Krzyżak, A., Pawlak, M.: Distribution-free consistency of a nonparametric kernel regression estimate and classification. IEEE Trans. Inf. Theory IT-30, 78–81 (1984)

    Google Scholar 

  41. Krzyżak, A., Schäfer, D.: Nonparametric regression estimation by normalized radial basis function networks. IEEE Trans. Inf. Theory 51, 1003–1010 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  42. Lv, J., Pawlak, M.: Bandwidth selection for kernel generalized regression neural networks in identification of Hammerstein systems. J. Artifi. Intell. Soft Comput. Res. 11(3), 181–194 (2021)

    Article  Google Scholar 

  43. Lugosi, G., Zeger, K.: Nonparametric estimation via empirical risk minimization. IEEE Trans. Inf. Theory 41, 677–687 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  44. Mikołajczyk, A., Grochowski, M., Kwasigroch, A.: Towards explainable classifiers using the counterfactual approach - global explanations for discovering bias in data. J. Artif. Intell. Soft Comput. Res. 11(1), 51–67 (2021)

    Article  Google Scholar 

  45. Moody, J., Darken, J.: Fast learning in networks of locally-tuned processing units. Neural Comput. 1, 281–294 (1989)

    Article  Google Scholar 

  46. Nadaraya, E.A.: On estimating regression. Theory Probability Appli. 9, 141–142 (1964)

    Article  MATH  Google Scholar 

  47. Nowicki, R.K., Grzanek, K., Hayashi, Y.: Rough support vector machine for classification with interval and incomplete data. J. Artif. Intell. Soft Comput. Res. 10(1), 47–56 (2020)

    Article  Google Scholar 

  48. Park, J., Sandberg, I.W.: Universal approximation using Radial-Basis-Function networks. Neural Comput. 3, 246–257 (1991)

    Article  Google Scholar 

  49. Park, J., Sandberg, I.W.: Approximation and radial-basis-function networks. Neural Comput. 5, 305–316 (1993)

    Article  Google Scholar 

  50. Pietruczuk, L., Rutkowski, L., Jaworski, M., Duda, P.: The Parzen kernel approach to learning in non-stationary environment. In: Proceedings 2014 International Joint Conference on Neural Networks (IJCNN), pp. 3319–3323 (2014)

    Google Scholar 

  51. Pollard, D.: Convergence of Stochastic Processes. Springer Verlag, New York (1984). https://doi.org/10.1007/978-1-4612-5254-2

  52. Rafajłowicz, W.: Learning novelty detection outside a class of random curves with application to COVID-19 growth. J. Artif. Intell. Soft Comput. Res. 11(3), 195–215 (2021)

    Article  Google Scholar 

  53. Ripley, B.D.: Pattern Recognition and Neural Networks. Cambridge University Press, Cambridge (2008)

    MATH  Google Scholar 

  54. Rutkowski, L.: Identification of MISO nonlinear regressions in the presence of a wide class of disturbances. IEEE Trans. Inf. Theory 37(1), 214–216 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  55. Rutkowski, L.: Adaptive probabilistic neural networks for pattern classification in time-varying environment. IEEE Trans. Neural Networks 15(4), 811–827 (2004)

    Article  Google Scholar 

  56. Samworth, R.J., Wand, M.P.: Asymptotics and optimal bandwidth selection for highest density region estimation. Ann. Stat. 38(3), 1767–1792 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  57. Scornet, E., Biau, G., Vert, J.-P.: Consistency of random forest. Ann. Stat. 43(4), 1716–1741 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  58. Shorten, R., Murray-Smith, R.: Side effects of normalising radial basis function networks. Int. J. Neural Syst. 7, 167–179 (1996)

    Article  Google Scholar 

  59. Specht, D.F.: Probabilistic neural networks. Neural Netw. 3, 109–118 (1990)

    Article  Google Scholar 

  60. Vapnik, V.N., Chervonenkis, A.Y.: On the uniform convergence of relative frequencies of events to their probabilities. Theory Probab. Appli. 16, 264–280 (1971)

    Article  MATH  Google Scholar 

  61. Vapnik, V.: Estimation of Dependences Based on Empirical Data. ISS, Springer, New York (2006). https://doi.org/10.1007/0-387-34239-7

    Book  MATH  Google Scholar 

  62. van de Geer, S.: Empirical Processes in M-Estimation. Cambridge University Press, New York (2000)

    Google Scholar 

  63. Watson, G.S.: Smooth regression analysis, Sankhya Series A, vol. 26, pp. 359–372

    Google Scholar 

  64. White, H.: Connectionist nonparametric regression: multilayer feedforward networks that can learn arbitrary map**s. Neural Netw. 3, 535–549 (1990)

    Article  Google Scholar 

  65. Wolverton, C.T., Wagner, T.J.: Asymptotically optimal discriminant functions for pattern classification. IEEE Trans. Inf. Theory IT-15 258–265 (1969)

    Google Scholar 

  66. Xu, L., Krzyżak, A., Yuille, A.L.: On radial basis function nets and kernel regression: approximation ability, convergence rate and receptive field size. Neural Netw. 7, 609–628 (1994)

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Software Engineering, Concordia University, 1455 de Maisonneuve Blvd. West, Montreal, H3G 1M8, Canada

    Adam Krzyżak

  2. Department of Electrical Engineering, Westpomeranian University of Technology, ul. Sikorskiego 37, 70-313, Szczecin, Poland

    Adam Krzyżak

  3. Institute of Computational Intelligence, Czȩstochowa University of Technology, Czȩstochowa, al. Armii Krajowej 36, 42-200, Czȩstochowa, Poland

    Tomasz Gałkowski

  4. Department of Knowledge Engineering, Faculty of Production Engineering and Logistics, Opole University of Technology, ul. Ozimska 75, 45-370, Opole, Poland

    Marian Partyka

Authors
  1. Adam Krzyżak
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tomasz Gałkowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marian Partyka
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tomasz Gałkowski .

Editor information

Editors and Affiliations

  1. Systems Research Institute of the Polish Academy of Sciences, Warsaw, Poland

    Leszek Rutkowski

  2. Częstochowa University of Technology, Częstochowa, Poland

    Rafał Scherer

  3. Częstochowa University of Technology, Częstochowa, Poland

    Marcin Korytkowski

  4. University of Alberta, Edmonton, AB, Canada

    Witold Pedrycz

  5. AGH University of Science and Technology, Kraków, Poland

    Ryszard Tadeusiewicz

  6. University of Louisville, Louisville, KY, USA

    Jacek M. Zurada

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Krzyżak, A., Gałkowski, T., Partyka, M. (2023). Convergence of RBF Networks Regression Function Estimates and Classifiers. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2022. Lecture Notes in Computer Science(), vol 13588. Springer, Cham. https://doi.org/10.1007/978-3-031-23492-7_31

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-23492-7_31

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-23491-0

  • Online ISBN: 978-3-031-23492-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation