SpringerLink
Log in

A facial expression recognition algorithm incorporating SVM and explainable residual neural network

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

To address the problem that traditional convolutional neural networks cannot classify facial expression image features precisely, an interpretable face expression recognition method combining ResNet18 residual network and support vector machines (SVM) is proposed in the paper. The SVM classifier is used to enhance the matching ability of feature vectors and labels under the expression image feature space, to improve the expression recognition effect of the whole model. The class activation map** and t-distributed stochastic neighbor embedding methods are used to visualize and interpret facial expression recognition’s feature analysis and decision making under the residual neural network. The experimental results and the interpretable visualization analysis show that the algorithm structure can effectively improve the recognition ability of the network. Under the FER2013, JAFFE, and CK+ datasets, it achieved 67.65%, 84.44%, and 96.94% emotional recognition accuracy, respectively, showing a certain generalization ability and superior performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

All the data included in this study are available upon request by contacting the corresponding author.

References

  1. Shan, L., Deng, W.: Deep facial expression recognition: a survey. IEEE Trans. Affect. Comput. 13, 1195–1215 (2018)

    Google Scholar 

  2. Yan, W., et al.: A systematic review on affective computing: emotion models, databases, and recent advances. Inf. Fus. 83–84 (2022)

  3. Mehrabian, A., Russell, J.A.: An Approach to Environment Psychology. MIT (1974)

    Google Scholar 

  4. Quan, C., Yao, Q., Ren, F.: Dynamic facial expression recognition based on K-order emotional intensity model. In: IEEE International Conference on Robotics & Biomimetics IEEE (2015)

  5. Wen-qiang, G.U.O., et al.: Facial expression recognition with small data sets based by bayesian network modeling. Sci. Technol. Eng. 18(35), 179–183 (2018)

    Google Scholar 

  6. Makhmudkhujaev, F., et al.: Facial expression recognition with local prominent directional pattern. Signal Process. Image Commun. 74, 1–12 (2019)

    Article  Google Scholar 

  7. Yang, H., Ciftci U., Yin, L.: Facial expression recognition by de-expression residue learning. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) IEEE (2018)

  8. Farzaneh, A.H., Qi, X.: Facial expression recognition in the wild via deep attentive center loss. In: Workshop on Applications of Computer Vision IEEE (2021)

  9. Zhao, J., Zhou, Y., Wang, X., et al.: Facial expression recognition method based on branch-assisted learning network. Comput. Eng. Appl. 58(23), 151–160 (2022)

    Google Scholar 

  10. Tan, K., Chen, J., Wang, D.L.: Gated residual networks with dilated convolutions for supervised speech separation. In: IEEE International Conference on Acoustics IEEE (2018)

  11. He, K., Zhang, X., Ren, S., Sun, J. Deep residual learning for image recognition. CoRR, http://arxiv.org/abs/1512.03385 (2015)

  12. Mingyang, L.A.N., Yulong, L.I.U., Tao, J.I.N., et al.: An improved recognition method based on visual trajectory circle and ResnetN18 for complex power quality disturbances. Proc. CSEE 42(17), 6274–6286 (2022)

    Google Scholar 

  13. Lingmin, L.I., Mengran, H.O.U., Kun, C.H.E.N., et al.: Survey on interpretability research of deep learning. J. Comput. Appl. 42(12), 3639–3650 (2022)

    Google Scholar 

  14. Zhou, B., Khosla, A., Lapedriza, À., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. CoRR, http://arxiv.org/abs/1512.04150 (2015)

  15. Mohamed, E., Sirlantzis, K., Howells, G.: A review of visualisation-as-explanation techniques for convolutional neural networks and their evaluation. Displays 73, 102239 (2022)

    Article  Google Scholar 

  16. Wang, X., Gu, Y.: Classification of macular abnormalities using a lightweight CNN-SVM framework. Meas. Sci. Technol. 6, 33 (2022)

    Google Scholar 

  17. Anowar, F., Sadaoui, S., Selim, B.: Conceptual and empirical comparison of dimensionality reduction algorithms (PCA, KPCA, LDA, MDS, SVD, LLE, ISOMAP, LE, ICA, t-SNE). Comput. Sci. Rev. 40, 100378 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  18. Dongyu, S.H.I., et al.: Study on visualization method of electrical distance in power system using t-SNE. Electr. Power Eng. Technol. 37(02), 78–82 (2018)

    Google Scholar 

  19. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 94–101 (2010)

  20. Goodfellow, I.J., Erhan, D., Carrier, P.L., et al.: Challenges in representation learning: a report on three machine learning contests. In: Neural Information Processing: 20th International Conference. Proceedings, Part III, vol. 20, pp. 117–124. Springer, Berlin (2013)

  21. Lyons, M.J., Kamachi, M., Gyoba, J.: The Japanese female facial expression (JAFFE) database. In: The 3th International Conference on Automatic Face And Gesture Recognition, pp. 14–16 (1997)

  22. Pahikkala, T., et al.: Efficient Hold-Out for Subset of Regressors. Springer, Berlin (2010)

    Google Scholar 

  23. Si, N.-W., Zhang, W.-L., Qu, D., et al.: Representation visualization of convolutional neural networks: a survey. Acta Automatica Sinica 48(08), 1890–1920 (2022)

    Google Scholar 

  24. Xu, L.L., Zhang, S.M., Zhao, J.L.: Expression recognition algorithm for parallel convolutional neural networks. J. Image Graph. 24(02), 227–236 (2019)

    Google Scholar 

  25. Agrawal, A., Mittal, N.: Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. Vis. Comput. 36(2), 405–412 (2020)

    Article  Google Scholar 

  26. **e, S., Hu, H., Wu, Y.: Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition. Pattern Recognit. 92, 177–191 (2019)

    Article  Google Scholar 

  27. Mollahosseini, A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks. In: IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE (2016)

  28. Arriaga, Octavio, Matias Valdenegro-Toro, Paul Plöger: Real-time convolutional neural networks for emotion and gender classification. http://arxiv.org/abs/1710.07557 (2017)

  29. Zhang, P., Kong, W.W., Teng, J.B.: Facial expression recognition based on multi-scale feature attention mechanism. Comput. Eng. Appl. 58(01), 182–189 (2022)

    Google Scholar 

  30. Cheng, H.X., Wang, X., Cheng, L., et al.: Facial expression recognition model design based on CNN and LSTM. Electron. Meas. Technol. 44(17), 160–164 (2021)

    Google Scholar 

Download references

Funding

This research was received by the Shanghai “Science and Technology Innovation Action Plan” Artificial Intelligence Science and Technology Support Special Project (20511101600).

Author information

Authors and Affiliations

  1. School of Mechanical Engineering, University of Shanghai for Science and Technology, Shanghai, 200093, China

    Lipeng Ji & Shilong Wu

  2. Department of Educational Information Technology, East China Normal University, Shanghai, 200062, China

    **aoqing Gu

Authors
  1. Lipeng Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shilong Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. **aoqing Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

JL and WS contributed to the conception of the study and contributed significantly to analysis and manuscript preparation; GX provided pre-theoretical support for this research.

Corresponding author

Correspondence to Lipeng Ji.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ji, L., Wu, S. & Gu, X. A facial expression recognition algorithm incorporating SVM and explainable residual neural network. SIViP 17, 4245–4254 (2023). https://doi.org/10.1007/s11760-023-02657-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02657-1

Keywords

Search

Navigation