Abstract
This paper delves into the critical importance of understanding emotions from a person’s perspective, and the potential for machines to improve human interaction by possessing this ability. While existing research on emotion recognition in computer vision has mainly focused on analyzing facial expressions and categorizing them into six basic emotions, it is important to recognize that contextual factors also play a crucial role in emotion perception. Emotions are not just limited to facial expressions but also include body language, the pitch of voice, and other nonverbal cues. We then trained a convolutional neural network model on this vast dataset and demonstrated the importance of incorporating context to recognize rich information about emotional states in images. Our model surpasses previous benchmarks and confirms the value of contextual information in emotion recognition. We have used the Emotions in Context (EMOTIC) [1] and Body Language Dataset (BoLD) [2] datasets for recognizing emotions by taking their contextual information into account. By incorporating contextual factors, machines can enhance human interaction by accurately recognizing emotional states in various situations. Based on the experiments, we recognized that the emotions of engagement (93.62%), confidence (92.41%), and excitement (95.93%) were predicted accurately. In contrast, the emotions of yearning, disapproval, and pain had low classification accuracy, with less than 40% accuracy. Lastly, this paper highlights the importance of understanding emotions beyond just facial expressions and provides a benchmark for emotion recognition in a contextual setting.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Kosti, R., Alvarez, J. M., Recasens, A., Lapedriza, A.: EMOTIC: emotions in context dataset. In: Computer Vision and Pattern Recognition (2017). https://doi.org/10.1109/cvprw.2017.285
Luo, Y., Ye, J., Adams, R.B., Li, J., Newman, M.G., Wang, J.Z.: ARBEE: towards automated recognition of bodily expression of emotion in the wild. Int. J. Comput. Vision 128, 1–25 (2018). https://doi.org/10.1007/s11263-019-01215-y
Kosti, R., Alvarez, J.M., Recasens, A., Lapedriza, A.: Context-based emotion recognition using emotic dataset. IEEE Trans. Pattern Anal. Mach. Intell. 42(11), 2755–2766 (2019)
Zhang, M., Liang, Y., Ma, H.: Context-aware affective graph reasoning for emotion recognition. In: 2019 IEEE International Conference on Multimedia and Expo (ICME), pp. 151–156. IEEE (2019)
Lee, J., Kim, S., Kim, S., Park, J., Sohn, K.: Context-aware emotion recognition networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10143–10152 (2019)
Mittal, T., Bera, A., Manocha, D.: Multimodal and context-aware emotion perception model with multiplicative fusion. IEEE Multimedia 28, 67–75 (2021)
Mittal, T., Guhan, P., Bhattacharya, U., Chandra, R., Bera, A., Manocha, D.: Emoticon: context-aware multimodal emotion recognition using frege’s principle. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14234–14243 (2020)
Hoang, M., Kim, S., Yang, H., Lee, G.: Context-aware emotion recognition based on visual relationship detection. IEEE Access 9, 90465–90474 (2021). https://doi.org/10.1109/access.2021.3091169
Goyal, A., Kumar, N., Guha, T., Narayanan, S.S.: A multimodal mixture- of-experts model for dynamic emotion prediction in movies. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 2822–2826). IEEE (2016)
Liu, S., Gao, P., Li, Y., Fu, W., Ding, W.: Multi-modal fusion network with complementarity and importance for emotion recognition. Inf. Sci. 619, 679–694 (2023)
Gupta, S., Kumar, P., Tekchandani, R.K.: Facial emotion recognition based real- time learner engagement detection system in online learning context using deep learning models. Multimed Tools Appl 82, 11365–11394 (2023). https://doi.org/10.1007/s11042-022-13558-9
Chaudhari, A., Bhatt, C., Krishna, A., Mazzeo, P.L.: ViTFER: facial emotion recognition with vision transformers. Appl. Syst. Innovation 5, 80 (2022). https://doi.org/10.3390/asi5040080
Chaudhari, A., Bhatt, C., Krishna, A., Travieso, C.M.: Facial emotion recognition with inter-modality-attention-transformer-based self-supervised learning. Electronics 12, 288 (2023). https://doi.org/10.3390/electronics12020288
Kothadiya, D., Chaudhari, A., Macwan, R., Patel, K., Bhatt, C.: The convergence of deep learning and computer vision: smart city applications and research challenges. In: Proceedings of the 3rd International Conference on Integrated Intelligent Computing Communication &Amp; Security (ICIIC 2021) (2021). https://doi.org/10.2991/ahis.k.210913.003
Zhao, J., Mao, X., Chen, L.: Speech emotion recognition using deep 1D & 2D CNN LSTM networks. Biomed. Signal Process. Control 47, 312–323 (2019). https://doi.org/10.1016/j.bspc.2018.08.035
Ye, M., Qian, H., Guangyuan, L.: CNN-LSTM facial expression recognition method fused with two-layer attention mechanism. Comput. Intell. Neurosci. 2022, 1–9 (2022). https://doi.org/10.1155/2022/7450637
Gao, Y., Li, B., Wang, N., Zhu, T.: Speech emotion recognition using local and global features. In: Zeng, Y., He, Y., Kotaleski, J.H., Martone, M., Xu, B., Peng, H., Luo, Q. (eds.) BI 2017. LNCS (LNAI), vol. 10654, pp. 3–13. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-70772-3_1
Milton, A.H., Roy, S.S., Selvi, S.T.: SVM scheme for speech emotion recognition using MFCC feature Int. J. Comput. Appl. (2013).https://doi.org/10.5120/11872-7667
Huang, Z., Dong, M., Dong, M., Zhan, Y.: Speech Emotion Recognition Using CNN. ACM Multimedia (2014).https://doi.org/10.1145/2647868.2654984
Lim, W., Jang, D., Lee, T.: Speech emotion recognition using convolutional and Recurrent Neural Networks. Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (2016). https://doi.org/10.1109/apsipa.2016.7820699
Kalliatakis, G., Ehsan, S., Leonardis, A., Fasli, M., McDonald-Maier, K.D.: Exploring object-centric and scene-centric CNN features and their complementarity for human rights violations recognition in images. IEEE Access 7, 10045–10056 (2019). https://doi.org/10.1109/access.2019.2891745
Sun, G., et al.: Deep fusion of localized spectral features and multi-scale spatial features for effective classification of hyperspectral images. Int. J. Appl. Earth Obs. Geoinf. 91, 102157 (2020). https://doi.org/10.1016/j.jag.2020.102157
Lu, M., Du, G., Li, Z.: Multimode gesture recognition algorithm based on convolutional long short-term memory network. Comput. Intell. Neurosci. 2022, 1 (2022). https://doi.org/10.1155/2022/4068414
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chaudhari, A., Bhatt, C., Krishna, A., Corchado, J.M. (2023). CERDL: Contextual Emotion Recognition Analysis Using Deep Learning. In: Novais, P., et al. Ambient Intelligence – Software and Applications – 14th International Symposium on Ambient Intelligence. ISAmI 2023. Lecture Notes in Networks and Systems, vol 770. Springer, Cham. https://doi.org/10.1007/978-3-031-43461-7_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-43461-7_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43460-0
Online ISBN: 978-3-031-43461-7
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)