SpringerLink
Log in

Treating Artificial Neural Net Training as a Nonsmooth Global Optimization Problem

  • Conference paper
  • First Online:
Machine Learning, Optimization, and Data Science (LOD 2019)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11943))

  • 1824 Accesses

Abstract

We attack the classical neural network training problem by successive piecewise linearization, applying three different methods for the global optimization of the local piecewise linear models. The methods are compared to each other and steepest descent as well as stochastic gradient on the regression problem for the Griewank function.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (Germany)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 42.79
Price includes VAT (Germany)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 53.49
Price includes VAT (Germany)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Arora, S., Cohen, N., Golowich, N., Hu, W.: A convergence analysis of gradient descent for deep linear neural networks. CoRR, abs/1810.02281 (2018)

    Google Scholar 

  2. Bagirov, A., Karmitsa, N., Mäkelä, M.: Introduction to Nonsmooth Optimization: Theory, Practice and Software. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-08114-4

    Book  MATH  Google Scholar 

  3. Bölcskei, H., Grohs, P., Kutyniok, G., Petersen, P.: Optimal approximation with sparsely connected deep neural networks. Ar**v:abs/1705.01714 (2019)

  4. Bottou, L., Curtis, F.E., Nocedal, J.: Optimization methods for large-scale machine learning. SIAM Rev. 60, 223–311 (2018)

    Article  MathSciNet  Google Scholar 

  5. Fourer, R., Kernighan, B.W.: AMPL: a modeling language for mathematical programming (2003)

    Google Scholar 

  6. Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. J. Mach. Learn. Res. 15, 315–323 (2011)

    Google Scholar 

  7. Griewank, A.: On stable piecewise linearization and generalized algorithmic differentiation. Optim. Methods Softw. 28(6), 1139–1178 (2013)

    Article  MathSciNet  Google Scholar 

  8. Griewank, A., Walther, A.: First and second order optimality conditions for piecewise smooth objective functions. Optim. Methods Softw. 31(5), 904–930 (2016)

    Article  MathSciNet  Google Scholar 

  9. Griewank, A.: Generalized descent of global optimization. J. Optim. Theory Appl. 34, 11–39 (1981)

    Article  MathSciNet  Google Scholar 

  10. Griewank, A., Walther, A.: Finite convergence of an active signature method to local minima of piecewise linear functions. Optim. Methods Softw. 34, 1035–1055 (2019)

    Article  MathSciNet  Google Scholar 

  11. Gupte, A., Ahmed, S., Cheon, M., Dey, S.: Solving mixed integer bilinear problems using MILP formulations. SIAM J. Optim. 23(2), 721–744 (2013)

    Article  MathSciNet  Google Scholar 

  12. Wright, S.J.: Coordinate descent algorithms. Math. Program. 151, 3–34 (2015)

    Article  MathSciNet  Google Scholar 

  13. Kakade, S.M., Lee, J.D.: Provably correct automatic sub-differentiation for qualified programs. Ar**v:abs/1809.08530 (2018)

  14. Kärkkäinen, T., Heikkola, E.: Robust formulations for training multilayer perceptrons. Neural Comput. 16, 837–862 (2004)

    Article  Google Scholar 

  15. Scholtes, S.: Introduction to Piecewise Differentiable Equations. Springer, New York (2012). https://doi.org/10.1007/978-1-4614-4340-7

    Book  MATH  Google Scholar 

  16. Yarotsky, D.: Error bounds for approximations with deep ReLU networks. Neural Netw. Off. J. Int. Neural Netw. Soc. 94, 103–114 (2017)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Research Center on Mathematical Modelling (MODEMAT), Escuela Politécnica Nacional, Quito, Ecuador

    Andreas Griewank

  2. School of Mathematical and Computational Sciences, Yachay Tech, Urcuquí, Ecuador

    Ángel Rojas

Authors
  1. Andreas Griewank
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ángel Rojas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andreas Griewank .

Editor information

Editors and Affiliations

  1. University of Cambridge, Cambridge, UK

    Giuseppe Nicosia

  2. University of Florida, Gainesville, FL, USA

    Panos Pardalos

  3. Harvard University, Cambridge, MA, USA

    Renato Umeton

  4. Università di Catania, Catania, Catania, Italy

    Giovanni Giuffrida

  5. Almawave, Rome, Roma, Italy

    Vincenzo Sciacca

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Griewank, A., Rojas, Á. (2019). Treating Artificial Neural Net Training as a Nonsmooth Global Optimization Problem. In: Nicosia, G., Pardalos, P., Umeton, R., Giuffrida, G., Sciacca, V. (eds) Machine Learning, Optimization, and Data Science. LOD 2019. Lecture Notes in Computer Science(), vol 11943. Springer, Cham. https://doi.org/10.1007/978-3-030-37599-7_64

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-37599-7_64

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-37598-0

  • Online ISBN: 978-3-030-37599-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation