SpringerLink
Log in

Multiple classifier for concatenate-designed neural network

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

This article introduces a multiple classifier method to improve the performance of concatenate-designed neural networks, such as ResNet and DenseNet, with the purpose of alleviating the pressure on the final classifier. We give the design of the classifiers, which collects the features produced between the network sets, and present the constituent layers and the activation function for the classifiers, to calculate the classification score of each classifier. We use the \(L2 \left( \sqrt{e^x}\right)\) normalization method to obtain the classifier score instead of the \(Softmax\) normalization. We also determine the conditions that can enhance convergence. As a result, the proposed classifiers are able to improve the accuracy in the experimental cases significantly and show that the method not only has better performance than the original models, but also produces faster convergence. Moreover, our classifiers are general and can be applied to all classification related concatenate-designed network models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. Generally, datasets CIFAR-10 and CIAR-100 can also reach complete convergence within 300 epochs, depending on the complexity of network architecture.

References

  1. Bagheri MA, Gao Q, Escalera S (2013) A genetic-based subspace analysis method for improving error-correcting output coding. Pattern Recognit 46(10):2830–2839

    Article  Google Scholar 

  2. Ball JE, Anderson DT, Chan CS (2017) A comprehensive survey of deep learning in remote sensing: Theories, tools and challenges for the community. ar**v: CoRR/abs/1709.00308

  3. Bridle JS (1989) Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In: Fogelman-Soulié F, Hérault J (eds) Proceedings of the NATO advanced research workshop on neurocomputing algorithms, architectures and applications, Les Arcs, France, February 27–March 3, 1989, Springer, NATO ASI Series, vol 68, pp 227–236, https://doi.org/10.1007/978-3-642-76153-9_28

  4. Byun H, Lee S (2003) A survey on pattern recognition applications of support vector machines. Int J Pattern Recognit Artif Intell 17(3):459–486

    Article  Google Scholar 

  5. Cao H, Bernard S, Heutte L, Sabourin R (2018) Dynamic voting in multi-view learning for radiomics applications. In: Bai X, Hancock ER, Ho TK, Wilson RC, Biggio B, Robles-Kelly A (eds) Structural, syntactic, and statistical pattern recognition–Joint IAPR international workshop, s+SSPR 2018, Bei**g, China, August 17–19, 2018, Proceedings, Springer, Lecture Notes in Computer Science, vol 11004, pp 32–41. https://doi.org/10.1007/978-3-319-97785-0_4

  6. Chen Y, ** X, Kang B, Feng J, Yan S (2017) Sharing residual units through collective tensor factorization in deep neural networks. ar**v:CoRR/abs/1703.02180

  7. Ciresan DC, Meier U, Gambardella LM, Schmidhuber J (2011) Convolutional neural network committees for handwritten character classification. In: 2011 International conference on document analysis and recognition, ICDAR 2011, Bei**g, China, September 18–21, 2011, IEEE Computer Society, pp 1135–1139, https://doi.org/10.1109/ICDAR.2011.229

  8. Collobert R, Sinz FH, Weston J, Bottou L (2006) Large scale transductive svms. J Mach Learn Res 7:1687–1712, http://jmlr.org/papers/v7/collobert06a.html

  9. Dietterich TG (2000) Ensemble methods in machine learning. In: Kittler J, Roli F (eds) Multiple classifier systems, first international workshop, MCS 2000, Cagliari, Italy, June 21-23, 2000, Springer, Lecture Notes in Computer Science, vol 1857, pp 1–15. https://doi.org/10.1007/3-540-45014-9_1

  10. Dietterich TG, Bakiri G (1995) Solving multiclass learning problems via error-correcting output codes. J Artif Intell Res 2:263–286

    Article  Google Scholar 

  11. Díez-Pastor J, Rodríguez JJ, García-Osorio CI, Kuncheva LI (2015) Diversity techniques improve the performance of the best imbalance learning ensembles. Inf Sci 325:98–117

    Article  MathSciNet  Google Scholar 

  12. Elleuch M, Maalej R, Kherallah M (2016) A new design based-svm of the CNN classifier architecture with dropout for offline arabic handwritten recognition. In: Connolly M (ed) International conference on computational science 2016, ICCS 2016, 6-8 June 2016, San Diego, California, USA, Elsevier, Procedia Computer Science, vol 80, pp 1712–1723. https://doi.org/10.1016/j.procs.2016.05.512

  13. Galar M, Fernández A, Tartas EB, Sola HB, Herrera F (2012) A review on ensembles for the class imbalance problem: Bagging-, boosting-, and hybrid-based approaches. IEEE Trans Syst Man Cybern Part C 42(4):463–484

    Article  Google Scholar 

  14. Gong S, Cristani M, Loy CC, Hospedales TM (2014) The re-identification challenge. In: Gong S, Cristani M, Yan S, Loy CC (eds) Person re-identification, advances in computer vision and pattern recognition, Springer, pp 1–20. https://doi.org/10.1007/978-1-4471-6296-4_1

  15. Hammann M, Kraus M, Shafaei S, Knoll AC (2020) Identity recognition in intelligent cars with behavioral data and lstm-resnet classifier. ar**v:CoRR/abs/2003.00770

  16. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, IEEE Computer Society, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  17. Hershey S, Chaudhuri S, Ellis DPW, Gemmeke JF, Jansen A, Moore RC, Plakal M, Platt D, Saurous RA, Seybold B, Slaney M, Weiss RJ, Wilson KW (2017) CNN architectures for large-scale audio classification. In: 2017 IEEE international conference on acoustics, speech and signal processing, ICASSP 2017, New Orleans, LA, USA, March 5-9, 2017, IEEE, pp 131–135. https://doi.org/10.1109/ICASSP.2017.7952132

  18. Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: 2017 IEEE conference on computer vision and pattern recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, IEEE Computer Society, pp 2261–2269. https://doi.org/10.1109/CVPR.2017.243

  19. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Bach FR, Blei DM (eds) Proceedings of the 32nd international conference on machine learning, ICML 2015, Lille, France, 6-11 July 2015, JMLR.org, JMLR Workshop and Conference Proceedings, vol 37, pp 448–456, http://proceedings.mlr.press/v37/ioffe15.html

  20. Jiang J, Trundle PR, Ren J (2010) Medical image analysis with artificial neural networks. Comput Medical Imaging Graph 34(8):617–631

    Article  Google Scholar 

  21. Kim DK, Chen T (2015) Deep neural network for real-time autonomous indoor navigation. ar**v:CoRR/abs/1511.04668

  22. Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: Bengio Y, LeCun Y (eds) 3rd international conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, conference track proceedings, arxiv:abs/1412.6980

  23. Krizhevsky A (2009) Learning multiple layers of features from tiny images. Tech. rep

  24. Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

    Article  Google Scholar 

  25. Lai K, Liu D, Chang S, Chen M (2015) Learning sample specific weights for late fusion. IEEE Trans Image Process 24(9):2772–2783

    Article  MathSciNet  Google Scholar 

  26. Lawrence S, Giles CL, Tsoi AC, Back AD (1997) Face recognition: a convolutional neural-network approach. IEEE Trans Neural Netw 8(1):98–113

    Article  Google Scholar 

  27. Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324. https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  28. Liu X, Gao J, He X, Deng L, Duh K, Wang Y (2015) Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In: Mihalcea R, Chai JY, Sarkar A (eds) NAACL HLT 2015, The 2015 conference of the north american chapter of the association for computational linguistics: human language technologies, Denver, Colorado, USA, May 31 - June 5, 2015, The Association for Computational Linguistics, pp 912–921. https://doi.org/10.3115/v1/n15-1092

  29. Mandelbaum A, Weinshall D (2017) Distance-based confidence score for neural network classifiers. ar**v:CoRR/abs/1709.09844

  30. Naranjo L, Perez CJ, Martín J, Campos-Roca Y (2017) A two-stage variable selection and classification approach for Parkinsons disease detection by using voice recording replications. Comput Methods Programs Biomed 142:147–156

    Article  Google Scholar 

  31. Nawaz W, Ahmed S, Tahir A, Khan HA (2018) Classification of breast cancer histology images using ALEXNET. In: Campilho A, Karray F, ter Haar Romeny BM (eds) Image analysis and recognition - 15th international conference, ICIAR 2018, Póvoa de Varzim, Portugal, June 27-29, 2018, Proceedings, Springer, Lecture Notes in Computer Science, vol 10882, pp 869–876. https://doi.org/10.1007/978-3-319-93000-8_99

  32. Nortje L, Kamper H (2020) Unsupervised vs. transfer learning for multimodal one-shot matching of speech and images. In: Meng H, Xu B, Zheng TF (eds) Interspeech 2020, 21st annual conference of the international speech communication association, virtual event, Shanghai, China, 25-29 October 2020, ISCA, pp 2712–2716. https://doi.org/10.21437/Interspeech.2020-0087

  33. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Köpf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S (2019) Pytorch: An imperative style, high-performance deep learning library. In: Wallach HM, Larochelle H, Beygelzimer A, d’Alché-Buc F, Fox EB, Garnett R (eds) Advances in neural information processing systems 32: annual conference on neural information processing systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp 8024–8035, https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html

  34. Qian Y, Ye M, Zhou J (2013) Hyperspectral image classification based on structured sparse logistic regression and three-dimensional wavelet texture features. IEEE Trans Geosci Remote Sens 51(4–2):2276–2291

    Article  Google Scholar 

  35. Ranjan R, Sankaranarayanan S, Castillo CD, Chellappa R (2017) An all-in-one convolutional neural network for face analysis. In: 12th IEEE international conference on automatic face & gesture recognition, FG 2017, Washington, DC, USA, May 30–June 3, 2017, IEEE Computer Society, pp 17–24. https://doi.org/10.1109/FG.2017.137

  36. Ren Z, Lee YJ (2018) Cross-domain self-supervised multi-task feature learning using synthetic imagery. In: 2018 IEEE conference on computer vision and pattern recognition, CVPR 2018, Salt Lake City, UT, USA, June 18–22, 2018, IEEE Computer Society, pp 762–771, DOIurl10.1109/CVPR.2018.00086, http://openaccess.thecvf.com/content_cvpr_2018/html/Ren_Cross-Domain_Self-Supervised_Multi-Task_CVPR_2018_paper.html

  37. Sandler M, Howard AG, Zhu M, Zhmoginov A, Chen L (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: 2018 IEEE conference on computer vision and pattern recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, IEEE Computer Society, pp 4510–4520. DOIurl10.1109/CVPR.2018.00474, http://openaccess.thecvf.com/content_cvpr_2018/html/Sandler_MobileNetV2_Inverted_Residuals_CVPR_2018_paper.html

  38. Shorten C, Khoshgoftaar TM (2019) A survey on image data augmentation for deep learning. J Big Data 6:60

    Article  Google Scholar 

  39. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: Bengio Y, LeCun Y (eds) 3rd international conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, conference track proceedings, arxiv:abs/1409.1556

  40. Sun Y, Liang D, Wang X, Tang X (2015) Deepid3: Face recognition with very deep neural networks. ar**v:CoRR/abs/1502.00873

  41. Szegedy C, Liu W, Jia Y, Sermanet P, Reed SE, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: IEEE conference on computer vision and pattern recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, IEEE Computer Society, pp 1–9. DOIurl10.1109/CVPR.2015.7298594

  42. Takemura A, Shimizu A, Hamamoto K (2010) Discrimination of breast tumors in ultrasonic images using an ensemble classifier based on the adaboost algorithm with feature selection. IEEE Trans Med Imaging 29(3):598–609

    Article  Google Scholar 

  43. Tan M, Le QV (2019) Efficientnet: Rethinking model scaling for convolutional neural networks. In: Chaudhuri K, Salakhutdinov R (eds) Proceedings of the 36th international conference on machine learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, PMLR, proceedings of machine learning research, vol 97, pp 6105–6114, http://proceedings.mlr.press/v97/tan19a.html

  44. Tian Y, Pei K, Jana S, Ray B (2018) Deeptest: automated testing of deep-neural-network-driven autonomous cars. In: Chaudron M, Crnkovic I, Chechik M, Harman M (eds) Proceedings of the 40th international conference on software engineering, ICSE 2018, Gothenburg, Sweden, May 27–June 03, 2018, ACM, pp 303–314, https://doi.org/10.1145/3180155.3180220

  45. Ueda N (2000) Optimal linear combination of neural networks for improving classification performance. IEEE Trans Pattern Anal Mach Intell 22(2):207–215

    Article  Google Scholar 

  46. Vapnik V (1999) An overview of statistical learning theory. IEEE Trans Neural Netw 10(5):988–999

    Article  Google Scholar 

  47. Wang L, Lee C, Tu Z, Lazebnik S (2015) Training deeper convolutional networks with deep supervision. ar**v:CoRR/abs/1505.02496

  48. Wei Y, Zhang Y, Huang J, Yang Q (2018) Transfer learning via learning to transfer. In: Dy JG, Krause A (eds) Proceedings of the 35th international conference on machine learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10–15, 2018, PMLR, proceedings of machine learning research, vol 80, pp 5072–5081, http://proceedings.mlr.press/v80/wei18a.html

  49. Wu Z, Shen C, van den Hengel A (2019) Wider or deeper: revisiting the resnet model for visual recognition. Pattern Recognit 90:119–133

    Article  Google Scholar 

  50. **ao L, Yan Q, Deng S (2017) Scene classification with improved alexnet model. In: Li T, Martínez-López L, Li Y (eds) 12th international conference on intelligent systems and knowledge engineering, ISKE 2017, Nan**g, China, November 24–26, 2017, IEEE, pp 1–6. https://doi.org/10.1109/ISKE.2017.8258820

  51. **e S, Girshick RB, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In: 2017 IEEE conference on computer vision and pattern recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, IEEE Computer Society, pp 5987–5995. https://doi.org/10.1109/CVPR.2017.634

  52. Xu B, Guo X, Ye Y, Cheng J (2012) An improved random forest classifier for text categorization. J Comput 7(12):2913–2920

    Google Scholar 

  53. Yu F, Wang D, Shelhamer E, Darrell T (2018) Deep layer aggregation. In: 2018 IEEE conference on computer vision and pattern recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, IEEE Computer Society, pp 2403–2412, https://doi.org/10.1109/CVPR.2018.00255, http://openaccess.thecvf.com/content_cvpr_2018/html/Yu_Deep_Layer_Aggregation_CVPR_2018_paper.html

  54. Zhang Y, Górriz JM, Dong Z (2021) Deep learning in medical image analysis. J Imaging 7(4):74

    Article  Google Scholar 

  55. Zheng X, Lin L, Liu B, **ao Y, **ong X (2020) A multi-task transfer learning method with dictionary learning. Knowl Based Syst 191:105233

    Article  Google Scholar 

  56. Zhong Z, Zheng L, Kang G, Li S, Yang Y (2020) Random erasing data augmentation. In: The Thirty-Fourth AAAI conference on artificial intelligence, AAAI 2020, the thirty-second innovative applications of artificial intelligence conference, IAAI 2020, The Tenth AAAi symposium on educational advances in artificial intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, AAAI Press, pp 13001–13008, https://aaai.org/ojs/index.php/AAAI/article/view/7000

Download references

Funding

This work was funded by The Science and Technology Development Fund, Macao SAR (File no. 0001/2018/AFJ).

Author information

Authors and Affiliations

  1. School of Applied Sciences, Macao Polytechnic Institute, Macao, China

    Ka-Hou Chan & Wei Ke

  2. Macao Polytechnic Institute, Macao, China

    Sio-Kei Im

  3. Engineering Research Centre of Applied Technology on Machine Translation and Artificial Intelligence of Ministry of Education, Macao Polytechnic Institute, Macao, China

    Ka-Hou Chan, Sio-Kei Im & Wei Ke

Authors
  1. Ka-Hou Chan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sio-Kei Im
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Ke
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ka-Hou Chan.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chan, KH., Im, SK. & Ke, W. Multiple classifier for concatenate-designed neural network. Neural Comput & Applic 34, 1359–1372 (2022). https://doi.org/10.1007/s00521-021-06462-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06462-0

Keywords

Search

Navigation