Abstract
Until now, convolutional neural networks (CNNs) still among the powerful and robust deep neural networks that proved its efficiency through several real applications. However, their functioning requires a large number of parameters which in turn lead to some undesired effects such as the overparametrization, overfitting and the high consumption of computational resources. To deal effectively with these issues, we propose in this paper a new multi-objective optimization model for redundancy reduction in CNNs. The suggested model named MoRR-CNN allows to eliminate the unwanted parameters (kernels and weights) as well as to speeding up the CNN evaluation process. It consists of two objectives, the first one is related to the training task where the solution is the optimal parameters. These parameters are combined with a set of decision variables that controlling their contribution in the training process, making at the end a redundancy-related objective function. Both of the objectives are optimized using the non dominated sorting genetic algorithm NSGA-II. The robustness of MoRR-CNN has been demonstrated through different experimentation applied on three benchmark datasets including MNIST, Fashion-MNIST and CIFAR and using three of the most known CNNs such as VGG-19, Net-in-Net and VGG-16.
Similar content being viewed by others
Data availability
The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.
References
LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324
Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi FE (2017) A survey of deep neural network architectures and their applications. Neurocomputing 234:11–26
Baldeon-Calisto M, Lai-Yuen SK (2020) Adaresu-net: multiobjective adaptive convolutional neural network for medical image segmentation. Neurocomputing 392:325–340
Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: international conference on medical image computing and computer-assisted intervention, pp. 234–241. Springer
Huang W, Zhang L, Wu H, Min F, Song A (2022) Channel-equalization-har: a light-weight convolutional neural network for wearable sensor based human activity recognition. IEEE Trans Mobile Comput. https://doi.org/10.1109/TMC.2022.3174816
Huang W, Zhang L, Wang S, Wu H, Song A (2022) Deep ensemble learning for human activity recognition using wearable sensors via filter activation. ACM Trans Embedded Comput Syst 22(1):1–23
Hssayni Eh, Joudar N-E, Ettaouil M (2022) A deep learning framework for time series classification using normal cloud representation and convolutional neural network optimization. Comput Intell 38(6):2056–2074
Huang W, Zhang L, Teng Q, Song C, He J (2021) The convolutional neural networks training with channel-selectivity for human activity recognition based on sensors. IEEE J Biomed Health Inform 25(10):3834–3843
Zhang Y, Sohn K, Villegas R, Pan G, Lee H (2015) Improving object detection with deep convolutional networks via bayesian optimization and structured prediction. In: proceedings of the IEEE conference on computer vision and pattern recognition, pp. 249–258
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 60(6):84–90
**ong W, Droppo J, Huang X, Seide F, Seltzer ML, Stolcke A, Yu D, Zweig G (2017) Toward human parity in conversational speech recognition. IEEE/ACM Trans Audio Speech Lang Process 25(12):2410–2423
Hssayni EH, Joudar N-E, Ettaouil M (2022) An adaptive drop method for deep neural networks regularization: estimation of dropconnect hyperparameter using generalization gap. Knowl Based Syst 253:109567
Denil M, Shakibi B, Dinh L, Ranzato M, De Freitas N (2013) Predicting parameters in deep learning. In: advances in neural information processing systems, vol 26
Ma R, Miao J, Niu L, Zhang P (2019) Transformed l 1 regularization for learning sparse deep neural networks. Neural Netw 119:286–298
Xu Q, Zhang M, Gu Z, Pan G (2019) Overfitting remedy by sparsifying regularization on fully-connected layers of CNNs. Neurocomputing 328:69–74
Hu H, Peng R, Tai Y-W, Tang C-K (2016) Network trimming: a data-driven neuron pruning approach towards efficient deep architectures. ar**v preprint ar**v:1607.03250
Kim Y-D, Park E, Yoo S, Choi T, Yang L, Shin D (2015) Compression of deep convolutional neural networks for fast and low power mobile applications. ar**v preprint ar**v:1511.06530
Ding H, Chen K, Yuan Y, Cai M, Sun L, Liang S, Huo Q (2017) A compact CNN-DBLSTM based character model for offline handwriting recognition with tucker decomposition. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR), vol. 1, pp. 507–512. IEEE
Lebedev V, Ganin Y, Rakhuba M, Oseledets I, Lempitsky V (2014) Speeding-up convolutional neural networks using fine-tuned cp-decomposition. ar**v preprint ar**v:1412.6553
Denton EL, Zaremba W, Bruna J, LeCun Y, Fergus R (2014) Exploiting linear structure within convolutional networks for efficient evaluation. In: advances in neural information processing systems, vol 27
Hssayni EH, Joudar N-E, Ettaouil M (2022) KRR-CNN: kernels redundancy reduction in convolutional neural networks. Neural Comput Appl 34(3):2443–2454
Hssayni EH, Joudar N-E, Ettaouil M (2022) Localization and reduction of redundancy in CNN using l1-sparsity induction. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-022-04025-2
Reiners M, Klamroth K, Heldmann F, Stiglmayr M (2022) Efficient and sparse neural networks by pruning weights in a multiobjective learning approach. Comput Oper Res 141:105676
Huang J, Sun W, Huang L (2020) Deep neural networks compression learning based on multiobjective evolutionary algorithms. Neurocomputing 378:260–269
Guo Y, Chen G, Jiang M, Gong D, Liang J (2022) A knowledge guided transfer strategy for evolutionary dynamic multiobjective optimization. IEEE Trans Evol Comput. https://doi.org/10.1109/TEVC.2022.322284
Chen G, Guo Y, Huang M, Gong D, Yu Z (2022) A domain adaptation learning strategy for dynamic multiobjective optimization. Inf Sciences 606:328–349
Deb K, Agrawal S, Pratap A, Meyarivan T (2000) A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: Nsga-ii. In: International Conference on Parallel Problem Solving from Nature, pp. 849–858. Springer
Calin O (2020) Deep learning architectures. Springer, New York
LeCun Y, Boser B, Denker J, Henderson D, Howard R, Hubbard W, Jackel L (1989) Handwritten digit recognition with a back-propagation network. In: Advances in neural information processing systems, vol 2
Ranzato M, Boureau Y-L, Cun Y et al (2007) Sparse feature learning for deep belief networks. In: Advances in neural information processing systems, vol 20
Collette Y, Siarry P (2011) Optimisation multiobjectif: algorithmes. Editions Eyrolles, Paris
Miettinen K (2012) Nonlinear multiobjective optimization. Springer, New York
Holland JH (1992) Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence. MIT press, Cambridge
Srinivas N, Deb K (1994) Muiltiobjective optimization using nondominated sorting in genetic algorithms. Evolutionary computation 2(3):221–248
Zitzler E, Deb K, Thiele L (2000) Comparison of multiobjective evolutionary algorithms: Empirical results. Evolutionary computation 8(2):173–195
Zitzler E, Thiele L (1998) Multiobjective optimization using evolutionary algorithms-a comparative case study. In: international conference on parallel problem solving from nature, pp. 292–301. Springer
Bottou L (2010) Large-scale machine learning with stochastic gradient descent. In: proceedings of COMPSTAT’2010, pp. 177–186. Springer
Hoseini F, Shahbahrami A, Bayat P (2019) Adaptahead optimization algorithm for learning deep CNN applied to MRI segmentation. J Digital Imaging 32(1):105–115
Lin M, Chen Q, Yan S (2013) Network in network. ar**v preprint ar**v:1312.4400
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. ar**v preprint ar**v:1409.1556
**ao H, Rasul K, Vollgraf R (2017) Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. ar**v preprint ar**v:1708.07747
Krizhevsky A, Hinton G et al (2009) Learning multiple layers of features from tiny images
Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252
Guo Y-N, Zhang X, Gong D-W, Zhang Z, Yang J-J (2019) Novel interactive preference-based multiobjective evolutionary optimization for bolt supporting networks. IEEE Trans Evol Comput 24(4):750–764
Ji J-J, Guo Y-N, Gao X-Z, Gong D-W, Wang Y-P (2021) Q-learning-based hyperheuristic evolutionary algorithm for dynamic task allocation of crowdsensing. IEEE Trans Cybern. https://doi.org/10.1109/TCYB.2021.3112675
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Boufssasse, A., Hssayni, E.h., Joudar, NE. et al. A Multi-objective Optimization Model for Redundancy Reduction in Convolutional Neural Networks. Neural Process Lett 55, 9721–9741 (2023). https://doi.org/10.1007/s11063-023-11223-2
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-023-11223-2