SpringerLink
Log in

A Novel Video Emotion Recognition System in the Wild Using a Random Forest Classifier

  • Conference paper
  • First Online:
Data Science (ICDS 2019)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1179))

Included in the following conference series:

Abstract

Emotions are expressed by humans to demonstrate their feelings in daily life. Video emotion recognition can be employed to detect various human emotions captured in videos. Recently, many researchers have been attracted to this research area and attempted to improve video emotion detection in both lab controlled and unconstrained environments. While the recognition rate of existing methods is high on lab-controlled datasets, they achieve much lower accuracy rates in a real-world uncontrolled environment. This is because of a variety of challenges present in real-world environments such as variations in illumination, head pose, and individual appearance. To address these challenges, in this paper, we propose a framework to recognize seven human emotions by extracting robust visual features from the videos captured in the wild and handle the head pose variation using a new feature extraction technique. First, sixty-eight face landmarks are extracted from different video sequences. Then, the Generalized Procrustes analysis (GPA) method is employed to normalize the extracted features. Finally, a random forest classifier is applied to recognize emotions. We have evaluated the proposed method using Acted Facial Expressions in the Wild (AFEW) dataset and obtained better accuracy than three existing video emotion recognition methods. It is noticeable that the proposed system can be applied to various contextual applications such as smart homes, healthcare, game industry and marketing in a smart city.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (Germany)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 42.79
Price includes VAT (Germany)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 53.49
Price includes VAT (Germany)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Mehrabian, A.J.: Communication without words. In: IOJT, pp. 193–200 (2008)

    Google Scholar 

  2. Li, Y., Zeng, J., Shan, S., Chen, X.: Occlusion aware facial expression recognition using CNN with attention mechanism. IEEE Trans. Image Process. 28(5), 2439–2450 (2019)

    Article  MathSciNet  Google Scholar 

  3. Xu, X., Deng, J., Coutinho, E., Wu, C., Zhao, L., Schuller, B.W.: Connecting subspace learning and extreme learning machine in speech emotion recognition. IEEE Trans. Multimedia 21(3), 795–808 (2019)

    Article  Google Scholar 

  4. Muszynski, M., et al.: Recognizing induced emotions of movie audiences from multimodal information. IEEE Trans. Affect. Comput. (2019). https://doi.org/10.1109/TAFFC.2019.2902091

    Article  Google Scholar 

  5. Tsiourti, C., Weiss, A., Vincze, M.: Multimodal integration of emotional signals from voice, body, and context: effects of (in) congruence on emotion recognition and attitudes towards robots. Int. J. Soc. Robot. 11, 555–573 (2019)

    Article  Google Scholar 

  6. Manfredonia, J., Bangerter, A., Manyakov, N., Ness, S., et al.: Automatic recognition of posed facial expression of emotion in individuals with autism spectrum disorder. J. Autism Dev. Disord. 49(1), 279–293 (2019)

    Article  Google Scholar 

  7. Ekman, P., Friesen, W.V., Ellsworth, P.: Emotion in the Human Face: Guide-Lines for Research and an Integration of Findings. Pergamon, Berlin (1972)

    Google Scholar 

  8. Samadiani, N., et al.: A review on automatic facial expression recognition systems assisted by multimodal sensor data. Sensors 19(8), 1863–1891 (2019)

    Article  Google Scholar 

  9. Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31, 39–58 (2009)

    Article  Google Scholar 

  10. Yan, H., Ang, M.H., Poo, A.N.: A survey on perception methods for human–robot interaction in social robots. Int. J. Soc. Robot. 6, 85–119 (2014)

    Article  Google Scholar 

  11. Yan, H., Lu, J., Zhou, X.: Prototype-based discriminative feature learning for kinship verification. IEEE Trans. Cybern. 45, 2535–2545 (2015)

    Article  Google Scholar 

  12. Yan, H.: Transfer subspace learning for cross-dataset facial expression recognition. Neurocomputing 208, 165–173 (2016)

    Article  Google Scholar 

  13. Wu, Z., **amixiding, R., Sajjanhar, A., Chen, J., Wen, Q.: Image appearance-based facial expression recognition. Int. J. Image Graph. 18(2), 1850012 (2018)

    Article  MathSciNet  Google Scholar 

  14. Dahua, L., Zhe, W., Qiang, G., Yu, S., **ao, Y., Chuhan, W.: Facial expression recognition based on Electroencephalogram and facial landmark localization. Technol. Health Care 27, 373–387 (2019)

    Article  Google Scholar 

  15. Owusu, E., Zhan, Y., Mao, Q.R.: A neural-AdaBoost based facial expression recognition system. Expert Syst. Appl. 41(7), 3383–3390 (2014)

    Article  Google Scholar 

  16. **e, S., Hu, H., Wu, Y.: Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition. Pattern Recogn. 92, 177–191 (2019)

    Article  Google Scholar 

  17. Mahendhiran, P.D., Kannimuthu, S.: Deep learning techniques for polarity classification in multimodal sentiment analysis. Int. J. Image Graph. 17(3), 883–910 (2018)

    Google Scholar 

  18. Dhall, A., Goecke, R., Lucey, S., Gedeon, T.: Static facial expression analysis in tough conditions: data, evaluation protocol and benchmark. In: Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain, pp. 2106–2112, November 2011

    Google Scholar 

  19. Dhall, A., Goecke, R., Lucey, S., Gedeon, T.: Collecting large, richly annotated facial expression databases from movies. IEEE Multimed. 19, 34–41 (2012)

    Article  Google Scholar 

  20. Munir, A., Hussain, A., Khan, S.A., Nadeem, M., Arshid, S.: Illumination invariant facial expression recognition using selected merged binary patterns for real world images. Optik 158, 1016–1025 (2018)

    Article  Google Scholar 

  21. Liu, M., Wang, R., Li, S., Shan, S., Huang, Z., Chen, X.: Combining multiple kernel methods on Riemannian manifold for emotional expression recognition in the wild. In: Proceedings of the 16th International Conference on Multimodal Interaction, Istanbul, Turkey, pp. 494–501 (2014)

    Google Scholar 

  22. Liu, L., Gui, W., Zhang, L., Chen, J.: Real-time pose invariant spontaneous smile detection using conditional random regression forests. Optik 182, 647–657 (2018)

    Article  Google Scholar 

  23. Fan, X., Tjahjadi, T.: A dynamic framework based on local Zernike moment and motion history image for facial expression recognition. Pattern Recogn. 64, 399–406 (2017)

    Article  Google Scholar 

  24. Zhao, J., Mao, X., Zhang, J.: Learning deep facial expression features from image and optical flow sequences using 3D CNN. Vis. Comput. Int. J. Comput. Graph. 34(10), 1461–1475 (2018)

    Google Scholar 

  25. Roza, V.C.C., Postolache, O.A.: Citizen emotion analysis in smart city. In: 7th International Conference on Information, Intelligence, Systems & Applications (IISA), Chalkidiki, pp. 1–6 (2016)

    Google Scholar 

  26. Muhammad, G., Alsulaiman, M., Amin, S.U., Ghoneim, A., Alhamid, M.F.: A facial-expression monitoring system for improved healthcare in smart cities. IEEE Access 5, 10871–10881 (2017)

    Article  Google Scholar 

  27. Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867–1874 (2014)

    Google Scholar 

  28. Gower, J.C.: Generalized procrustes analysis. Psychometrika 40, 33–51 (1975)

    Article  MathSciNet  Google Scholar 

  29. Leutenegger, S., Chli, M., Siegwart, R.Y.: BRISK: Binary robust invariant scalable keypoints. In: IEEE International Conference on Computer Vision, pp. 2548–2555 (2011)

    Google Scholar 

  30. Berge, J., Bekker, P.: The isotropic scaling problem in Generalized Procrustes Analysis. Comput. Stat. Data Anal. 16(2), 201–204 (1993)

    Article  Google Scholar 

  31. Wiering, M.A., Hasselt, H.: Ensemble algorithms in reinforcement learning. IEEE Trans. Syst. Man Cybern. Part B Cybern. 38(4), 930–936 (2008)

    Article  Google Scholar 

  32. Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)

    Article  Google Scholar 

  33. Raczko, E., Zagajewski, B.: Comparison of support vector machine, random forest and neural network classifiers for tree species classification on airborne hyperspectral APEX images. Eur. J. Remote Sens. 50(1), 144–154 (2017)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by Australian Research Council (ARC) Grant (No. DE140100387).

Author information

Authors and Affiliations

  1. School of Information Technology, Deakin University, Burwood, Australia

    Najmeh Samadiani, Guangyan Huang & Wei Luo

  2. Data61, CSIRO, Sandy Bay, Hobart, TAS, 7005, Australia

    Yanfeng Shu

  3. School of Architecture and Built Environment, Deakin University, Geelong, Australia

    Rui Wang & Tuba Kocaturk

Authors
  1. Najmeh Samadiani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guangyan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yanfeng Shu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tuba Kocaturk
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guangyan Huang .

Editor information

Editors and Affiliations

  1. Swinburne University of Technology, Melbourne, VIC, Australia

    **g He

  2. University of Illinois at Chicago, Chicago, USA

    Philip S. Yu

  3. College of Information Science and Technology, University of Nebraska at Omaha, Omaha, NE, USA

    Yong Shi

  4. Research Institute of Extenics and Innovation Methods, Guangdong University of Technology, Guangzhou, China

    **ngsen Li

  5. Ningbo University, Ningbo, China

    Zhijun **e

  6. Deakin University, Burwood, VIC, Australia

    Guangyan Huang

  7. Department of Computer Science and Technology, Nan**g University of Science and Technology, Nan**g, China

    Jie Cao

  8. Nan**g University of Posts and Telecommunications, Nan**g, China

    Fu **ao

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Samadiani, N., Huang, G., Luo, W., Shu, Y., Wang, R., Kocaturk, T. (2020). A Novel Video Emotion Recognition System in the Wild Using a Random Forest Classifier. In: He, J., et al. Data Science. ICDS 2019. Communications in Computer and Information Science, vol 1179. Springer, Singapore. https://doi.org/10.1007/978-981-15-2810-1_27

Download citation

  • DOI: https://doi.org/10.1007/978-981-15-2810-1_27

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-15-2809-5

  • Online ISBN: 978-981-15-2810-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation