Abstract
Neural network pruning is a widely used approach for reducing the inference cost of deep models in order to deploy on resource-limited settings. However, current pruning works lack attention to information migration from pruned to the remaining part of the Deep Neural Network, and on balancing model performance and compression rate. On these two issues, in this paper, we propose a novel \(\textbf{E}\)xplicit \(\textbf{I}\)nformation \(\textbf{M}\)igration network \(\textbf{P}\)runing (EIMP) algorithm. Specifically (1) the constrained gradient update method transfers valid information from redundant networks to the preserved, and (2) the newly designed \(\lambda \)-decay regularization method learns the trade-off between the performance and penalty item. Experiments show that our EIMP algorithm achieves state-of-the-art performance on several datasets with various benchmark network architectures. Notably, EIMP achieves \(+1.54\%\) better than SOTA on ImageNet.
Work done during the internship at OPPO.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Chen, T.Y., et al.: Only train once: a one-shot neural network training and pruning framework. In: Advances in Neural Information Processing Systems, vol. 34 (2021)
Chin, T.W., Ding, R.Z., Zhang, C., Marculescu, D.: Towards efficient model compression via learned global ranking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1518–1528 (2020)
Ding, X.H., Hao, T.X., et al.: Resrep: Lossless CNN pruning via decoupling remembering and forgetting. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4510–4520 (2021)
Frankle, J., Carbin, M.: The lottery ticket hypothesis: finding sparse, trainable neural networks. ar**v preprint ar**v:1803.03635 (2018)
Gao, S.Q., Huang, F.H., Cai, W.D., Huang, H.: Network pruning via performance maximization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9270–9280 (2021)
Guo, Y., Yuan, H., Tan, J.C., et al.: GDP: stabilized neural network pruning via gates with differentiable polarization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5239–5250 (2021)
Han, S., Pool, J., et al.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems, vol. 28 (2015)
Hassibi, B., et al.: Optimal brain surgeon and general network pruning. In: IEEE International Conference on Neural Networks, pp. 293–299. IEEE (1993)
He, K.M., Zhang, X.Y., Ren, S.Q., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
He, Y., Kang, G.L., et al.: Soft filter pruning for accelerating deep convolutional neural networks. ar**v preprint ar**v:1808.06866 (2018)
He, Y., Liu, P., Wang, Z.W., Hu, Z.L., Yang, Y.: Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4340–4349 (2019)
He, Y., et al.: Learning filter pruning criteria for deep convolutional neural networks acceleration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2009–2018 (2020)
Hou, Z.J., Qin, M.H., et al.: Chex: channel exploration for CNN model compression. ar**v preprint ar**v:2203.15794 (2022)
Howard, A.G., Zhu, M.L., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. ar**v preprint ar**v:1704.04861 (2017)
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report 0, University of Toronto, Toronto, Ontario (2009)
Li, B., Wu, B., Su, J., Wang, G.: EagleEye: fast sub-net evaluation for efficient neural network pruning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 639–654. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_38
Li, Y.C., Lin, S.H., et al.: Towards compact CNNs via collaborative compression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6438–6447 (2021)
Li, Y.W., Gu, S.H., et al.: Group sparsity: the hinge between filter pruning and decomposition for network compression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8018–8027 (2020)
Lin, M.B., Ji, R.R., et al.: Hrank: filter pruning using high-rank feature map. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1529–1538 (2020)
Liu, Z., Li, J.G., et al.: Learning efficient convolutional networks through network slimming. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2736–2744 (2017)
Liu, L.Y., et al.: Group fisher pruning for practical network compression. In: International Conference on Machine Learning, pp. 7021–7032. PMLR (2021)
Liu, Z., et al.: Rethinking the value of network pruning. ar**v preprint ar**v:1810.05270 (2018)
Liu, Z.C., et al.: MetaPruning: meta learning for automatic neural network channel pruning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3296–3305 (2019)
Miao, L., et al.: Learning pruning-friendly networks via frank-Wolfe: one-shot, any-sparsity, and no retraining. In: International Conference on Learning Representations (2021)
Park, J.H., et al.: Dynamic structure pruning for compressing CNNs. ar**v:2303.09736 (2023)
Russakovsky, O., Deng, J., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015)
Shen, M.Y., Yin, H.X., et al.: Structural pruning via latency-saliency knapsack. ar**v preprint ar**v:2210.06659 (2022)
Tang, Y.H., et al.: Scientific control for reliable neural network pruning. Scop. Adv. Neural. Inf. Process. Syst. 33, 10936–10947 (2020)
Wang, H., Qin, C., et al.: Neural pruning via growing regularization. ar**v preprint ar**v:2012.09243 (2020)
Wang, N.G., Liu, C.C., et al.: Deep compression of pre-trained transformer models. Adv. Neural. Inf. Process. Syst. 35, 14140–14154 (2022)
Wang, Z., Li, C.C., Wang, X.Y.: Convolutional neural network pruning with structural redundancy reduction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14913–14922 (2021)
Yang, H.C., Gui, S.P., Zhu, Y.H., Liu, J.: Automatic neural network compression by sparsity-quantization joint learning: A constrained optimization-based approach. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2178–2188 (2020)
Yu, S.X., Mazaheri, A., Jannesari, A.: Auto graph encoder-decoder for neural network pruning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6362–6372 (2021)
Zhang, M., et al.: Weighted mutual learning with diversity-driven model compression. In: Advances in Neural Information Processing Systems (2022)
Zhang, Y.F., et al.: Exploration and estimation for model compression. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 487–496 (2021)
Zheng, S., et al.: Fast-and-light stochastic ADMM. In: IJCAI, pp. 2407–2613 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wu, J., Hu, D., Zheng, Z. (2024). Network Pruning via Explicit Information Migration. In: Fang, L., Pei, J., Zhai, G., Wang, R. (eds) Artificial Intelligence. CICAI 2023. Lecture Notes in Computer Science(), vol 14474. Springer, Singapore. https://doi.org/10.1007/978-981-99-9119-8_35
Download citation
DOI: https://doi.org/10.1007/978-981-99-9119-8_35
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-9118-1
Online ISBN: 978-981-99-9119-8
eBook Packages: Computer ScienceComputer Science (R0)