SpringerLink
Log in

Emotion Recognition from Multimodal Data: a machine learning approach combining classical and hybrid deep architectures

  • Original Article
  • Published:
Research on Biomedical Engineering Aims and scope Submit manuscript

Abstract

Purpose

The expression of emotions is essential in human relationships. However, the aging process associated with some pathologies such as Alzheimer’s Disease and other dementias can affect our ability to express emotions.

Methods

In this context, we propose a method for automatic recognition of emotions from multimodal data. We based this approach on Artificial Intelligence algorithms, as part of the development of a human–machine interface to support the personalization of therapy for elderly people with dementia. From this tool, emotional feedback can modulate the therapy. By doing this we hope to improve the therapeutic results. In this work, the performance of the proposed architectures was evaluated regarding to their ability to recognize emotions in physiological and speech signals and in images of facial expressions.

Results

In the context of physiological and speech signals, we achieved promising results with the use of Random Forest. We found an accuracy of up to 99% in classifying emotions from physiological signals and almost 80% with speech signals. In the images assessment, we found more than 82% of accuracy when adopting a hybrid architecture.

Conclusion

The good results in the test stage are encouraging and point to the possibility of adopting the method in the analysis of emotions in multimodal data. These findings are even more interesting due to the large amount and variety of emotions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

Not applicable.

References

  • Abdullah SMSA, Ameen SYA, Sadeeq MA, Zeebaree S. Multimodal emotion recognition using deep learning. J Appl Sci Technol Trends. 2021;2(02):52–8.

    Article  Google Scholar 

  • Alarcao SM, Fonseca MJ. Emotions recognition using eeg signals: A survey. IEEE Trans Affect Comput. 2017;10(3):374–93.

    Article  Google Scholar 

  • Andrade MK, Santana MA, Moreno G, Oliveira I, Santos J, Rodrigues MCA, dos Santos WP: An EEG Brain-Computer Interface to Classify Motor Imagery Signals, 83–98. Springer, Singapore (2020)

  • Aranha RV, Silva LS, Chaim ML, Nunes FDLDS.: Using Affective Computing to Automatically Adapt Serious Games for Rehabilitation. In: 2017 IEEE 30th International Symposium on Computer-Based Medical Systems (CBMS), 55–60 (2017). https://doi.org/10.1109/CBMS.2017.89

  • Arroyo-Palacios J, Slater M. Dancing with Physio: A Mobile Game with Physiologically Aware Virtual Humans. IEEE Trans Affect Comput. 2016;7(4):326–36. https://doi.org/10.1109/TAFFC.2015.2472013.

    Article  Google Scholar 

  • Artstein R, Poesio M. Inter-coder agreement for computational linguistics. Comput Linguist. 2008;34(4):555–96. https://doi.org/10.1162/coli.07-034-R2.

    Article  Google Scholar 

  • Ayata D, Yaslan Y, Kamasak ME. Emotion recognition from multimodal physiological signals for emotion aware healthcare systems. J Med Biol Eng. 2020;40:149–57.

    Article  Google Scholar 

  • Barbosa VAF, Santana MA, Andrade MKS, Lima RCF, Santos WP. Deep-wavelet neural networks for breast cancer early diagnosis using mammary termographies. In: Das H, Pradhan C, Dey N, editors. Deep Learning for Data Analytics: Foundations, Biomedical Applications, and Challenges. 1st ed. London: Academic Press; 2020.

    Google Scholar 

  • Barnes DE, Yaffe K. The projected effect of risk factor reduction on alzheimer’s disease prevalence. The Lancet Neurology. 2011;10(9):819–28.

    Article  Google Scholar 

  • Behere R, Arasappa R, Jagannathan A, Varambally S, Venkatasubramanian G, Thirthalli J, Subbakrishna D, Nagendra H, Gangadhar B. Effect of yoga therapy on facial emotion recognition deficits, symptoms and functioning in patients with schizophrenia. Acta Psychiatr Scand. 2011;123(2):147–53.

    Article  Google Scholar 

  • Blackburn R, Bradshaw T. Music therapy for service users with dementia: A critical review of the literature. J Psychiatr Ment Health Nurs. 2014;21(10):879–88.

    Article  Google Scholar 

  • Blagus R, Lusa L. Smote for high-dimensional class-imbalanced data. BMC Bioinformatics. 2013;14(1):106.

    Article  Google Scholar 

  • Bota PJ, Wang C, Fred AL, Da Silva HP. A review, current challenges, and future possibilities on emotion recognition using machine learning and physiological signals. IEEE Access. 2019;7:140990–1020.

    Article  Google Scholar 

  • Breiman L. Random Forests. Machine Learning. 2001;45(1):5–32.

    Article  MATH  Google Scholar 

  • Brotons M, Marti P. Music therapy with alzheimer’s patients and their family caregivers: a pilot project. J Music Ther. 2003;40(2):138–50.

    Article  Google Scholar 

  • Calvo RA, D’Mello S. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Trans Affect Comput. 2010;1(1):18–37.

    Article  Google Scholar 

  • Cambria E, Das D, Bandyopadhyay S, Feraco A.: Affective computing and sentiment analysis. A practical guide to sentiment analysis, 1–10 (2017)

  • Castro CB, Costa L, Dias CB, Chen J, Loo R, Sohrabi HR, Brown BM, Martins RN. Multi-domain interventions for dementia prevention–a systematic review. Alzheimer’s & Dementia. 2021;17:058289.

    Article  Google Scholar 

  • Chaturvedi V, Kaur AB, Varshney V, Garg A, Chhabra GS, Kumar M.: Music mood and human emotion recognition based on physiological signals: a systematic review. Multimed Syst, 1–24 (2021)

  • Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. Smote: synthetic minority over-sampling technique. J Artif Intell Res. 2002;16:321–57.

    Article  MATH  Google Scholar 

  • Cobos FJM, Rodríguez MDMM. A review of psychological intervention in alzheimer s disease. Int J Psychol Psychol Therapy. 2012;12(3):373–88.

    Google Scholar 

  • Cortes C, Vapnik V. Support-Vector Networks. Machine Learning. 1995;20(3):273–97.

    Article  MATH  Google Scholar 

  • Cruz T, Cruz T, Santos W. Detection and classification of lesions in mammographies using neural networks and morphological wavelets. IEEE Lat Am Trans. 2018;16(3):926–32.

    Article  Google Scholar 

  • da Saúde OM (2021) Ageing and health. Available in: https://www.who.int/news−room/fact−sheets/detail/ageing−and−health

  • da Silva CAS, Krohling RA: Classificação de grandes bases de dados utilizando algoritmo de máquina de aprendizado extremo. In: Simpósio Brasileiro de Pesquisa Operacional - SBPO (2016)

  • de Oliveira E, Jaques PA. Classificação de emoções básicas através de imagens capturadas por webcam. Revista Brasileira De Computação Aplicada. 2013;5(2):40–54.

    Article  Google Scholar 

  • de Oliveira APS, Santana MA, Andrade MKS, Gomes JC, Rodrigues MC, dos Santos WP. Early diagnosis of parkinson’s disease using eeg, machine learning and partial directed coherence. Res Biomed Eng. 2020;36(3):311–31.

    Article  Google Scholar 

  • de Santana MA, de Lima CL, Torcate AS, Fonseca FS, dos Santos WP. Affective computing in the context of music therapy: a systematic review. Res Soc Dev. 2021;10(15):392101522844–392101522844.

    Article  Google Scholar 

  • De Silva LC, Miyasato TM, Nakatsu R: Facial emotion recognition using multi-modal information. In: Proceedings of ICICS, 1997 International Conference on Information, Communications and Signal Processing. Theme: Trends in Information Systems Engineering and Wireless Multimedia Communications (Cat., 1, 397–401 (1997). IEEE

  • Delmastro F, Martino FD, Dolciotti C.: Physiological Impact of VibroAcoustic Therapy on Stress and Emotions through Wearable Sensors. In: 2018 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), 621–626 (2018). https://doi.org/10.1109/PERCOMW.2018.8480170

  • Deng L. The mnist database of handwritten digit images for machine learning research. IEEE Signal Process Mag. 2012;29(6):141–2.

    Article  Google Scholar 

  • Deshpande A, Kumar M. Artificial Intelligence for Big Data: Complete Guide to Automating Big Data Solutions Using Artificial Intelligence Techniques. United Kingdom: Packt Publishing; 2018.

    Google Scholar 

  • Dhuheir M, Albaseer A, Baccour E, Erbad A, Abdallah M, Hamdi M: Emotion recognition for healthcare surveillance systems using neural networks: A survey. In: 2021 International Wireless Communications and Mobile Computing (IWCMC), 681–687 (2021). IEEE

  • Doma V, Pirouz M. A comparative analysis of machine learning methods for emotion recognition using eeg and peripheral physiological signals. Journal of Big Data. 2020;7(1):1–21.

    Article  Google Scholar 

  • Dorneles SODSO, Barbosa DNF, Barbosa JLV: Sensibilidade ao contexto na identificação de estados afetivos aplicados à educação: um mapeamento sistemático. Renote 18(1) (2020)

  • Eaton JW, Bateman D, Hauberg S, Wehbring R.: Gnu octave version 4.0. 0 manual: a high-level interactive language for numerical computations. 2015. URL http://www.gnu.org/software/octave/doc/interpreter 8, 13 (2015)

  • Espinola CW, Gomes JC, Pereira JMS, dos Santos WP. Detection of major depressive disorder using vocal acoustic analysis and machine learning—an exploratory study. Res Biomed Eng. 2021a;37(1):53–64.

    Article  Google Scholar 

  • Espinola CW, Gomes JC, Pereira JMS, dos Santos WP. Vocal acoustic analysis and machine learning for the identification of schizophrenia. Res Biomed Eng. 2021b;37(1):33–46.

    Article  Google Scholar 

  • Ferreira CD, Torro-Alves N.: Reconhecimento de emoções faciais no envelhecimento: uma revisão sistemática. Universitas Psychologica 15(5) (2016)

  • de Freitas Barbosa VA, Gomes JC, de Santana MA, Jeniffer EdA, de Souza RG, de Souza RE, dos Santos WP.: Heg.IA: An intelligent system to support diagnosis of Covid-19 based on blood tests. Res Biomed Eng 1–18 (2021)

  • García-Casal JA, Goñi-Imizcoz M, Perea-Bartolomé M, Soto-Pérez F, Smith SJ, Calvo-Simal S, Franco-Martín M. The efficacy of emotion recognition rehabilitation for people with alzheimer’s disease. J Alzheimer’s Disease. 2017;57(3):937–51.

    Article  Google Scholar 

  • Gomes JC, Barbosa VADF, Santana MA, Bandeira J, Valença MJS, de Souza RE, Ismael AM, dos Santos WP: IKONOS: An intelligent tool to support diagnosis of covid-19 by texture analysis of x-ray images. Res Biomed Eng 1–14 (2020)

  • Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hamner F, Li R, Wang X, Athanasakis D, Shawe-Taylor J, Milakov M, Park J, Ionescu R, Popescu M, Grozea C, Bergstra J, **e J, Romaszko L, Xu B, Chuang Z, Bengio Y.: Challenges in representation learning: A report on three machine learning contests. In: International Conference on Neural Information Processing, 117–124 (2013). Springer

  • Guess H.: Alzheimer’s disease and the impact of music therapy a systematic literature review (2017)

  • Guetin S, Charras K, Berard A, Arbus C, Berthelon P, Blanc F, Blayac J-P, Bonte F, Bouceffa J-P, Clement S, Ducourneau G, Gzil F, Laeng N, Lecourt E, Ledoux S, Platel H, Thomas-Anterion C, Touchon J, Vrait F-X, Leger J-M. An overview of the use of music therapy in the context of alzheimer’s disease: a report of a french expert group. Dementia. 2013;12(5):619–34.

    Article  Google Scholar 

  • Gupta V, Chopda MD, Pachori RB. Cross-subject emotion recognition using flexible analytic wavelet transform from eeg signals. IEEE Sens J. 2018;19(6):2266–74.

    Article  Google Scholar 

  • Haan MN, Wallace R. Can dementia be prevented? brain aging in a population-based context. Annu Rev Public Health. 2004;25:1–24.

    Article  Google Scholar 

  • Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology. 1982;143(1):29–36.

    Article  Google Scholar 

  • Harms MB, Martin A, Wallace GL. Facial emotion recognition in autism spectrum disorders: a review of behavioral and neuroimaging studies. Neuropsychol Rev. 2010;20(3):290–322.

    Article  Google Scholar 

  • Hasnul MA, Aziz NAA, Alelyani S, Mohana M, Aziz AA. Electrocardiogram-based emotion recognition systems and their applications in healthcare—a review. Sensors. 2021;21(15):5015.

    Article  Google Scholar 

  • Huang G-B, Zhu Q-Y, Siew C-K: Extreme learning machine: a new learning scheme of feedforward neural networks. In: 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541), 2: 985–9902 (2004). https://doi.org/10.1109/IJCNN.2004.1380068

  • Issa D, Demirci MF, Yazici A. Speech emotion recognition with deep convolutional neural networks. Biomed Signal Process Control. 2020;59:101894.

    Article  Google Scholar 

  • Izard CE. Human Emotions. New York: Springer; 1977.

    Book  Google Scholar 

  • Izard CE. The Psychology of Emotions. New York, USA: Springer; 1991.

    Book  Google Scholar 

  • Jackins V, Vimal S, Kaliappan M, Lee MY. Ai-based smart prediction of clinical disease using random forest classifier and naive bayes. J Supercomput. 2021;77(5):5198–219.

    Article  Google Scholar 

  • Jung Y, Hu J. A K-fold averaging cross-validation procedure. J Nonparametric Stat. 2015;27(2):167–79.

    Article  MathSciNet  MATH  Google Scholar 

  • Khalili Z, Moradi MH: Emotion recognition system using brain and peripheral signals: using correlation dimension to improve the results of eeg. In: 2009 International Joint Conference on Neural Networks, 1571–1575 (2009). IEEE

  • Kim J, André E. Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal Mach Intell. 2008;30(12):2067–83.

    Article  Google Scholar 

  • Kohler CG, Turner TH, Bilker WB, Brensinger CM, Siegel SJ, Kanes SJ, Gur RE, Gur RC. Facial emotion recognition in schizophrenia: intensity effects and error pattern. Am J Psychiatry. 2003;160(10):1768–74.

    Article  Google Scholar 

  • Kołakowska A, Szwoch W, Szwoch M. A review of emotion recognition methods based on data acquired via smartphone sensors. Sensors. 2020;20(21):6367.

    Article  Google Scholar 

  • Kusuma GP, Jonathan APL, Lim A. Emotion recognition on fer-2013 face images using fine-tuned vgg-16. Adv Sci Technol Eng Syst J. 2020;5(6):315–22.

    Article  Google Scholar 

  • Lawrence K, Campbell R, Skuse D. Age, gender, and puberty influence the development of facial emotion recognition. Front Psychol. 2015;6:761.

    Article  Google Scholar 

  • LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE. 1998;86(11):2278–324.

    Article  Google Scholar 

  • Leggieri M, Thaut MH, Fornazzari L, Schweizer TA, Barfett J, Munoz DG, Fischer CE. Music intervention approaches for alzheimer’s disease: A review of the literature. Front Neurosci. 2019;13:132.

    Article  Google Scholar 

  • Lenze SN, Pautsch J, Luby J. Parent–child interaction therapy emotion development: A novel treatment for depression in preschool children. Depress Anxiety. 2011;28(2):153–9.

    Article  Google Scholar 

  • Lin Y-P, Wang C-H, Wu T-L, Jeng S-K, Chen J-H.: Eeg-based emotion recognition in music listening: A comparison of schemes for multiclass support vector machine. In: 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, 489–492 (2009). IEEE

  • Livingston G, Sommerlad A, Orgeta V, Costafreda SG, Huntley J, Ames D, Ballard C, Banerjee S, Burns A, Cohen-Mansfield J, Cooper C, Fox N, Gitlin LN, Howard R, Kales HC, Larson EB, Ritchie K, Rockwood K, Sampson EL, Samus Q, Schneider LS, Selbaek G, Teri L, Mukadam N. Dementia prevention, intervention, and care. The Lancet. 2017;390(10113):2673–734.

    Article  Google Scholar 

  • Livingstone SR, Russo FA. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PLoS ONE. 2018;13(5):0196391.

    Article  Google Scholar 

  • Luna-Jiménez C, Griol D, Callejas Z, Kleinlein R, Montero JM, Fernández-Martínez F. Multimodal emotion recognition on ravdess dataset using transfer learning. Sensors. 2021;21(22):7665.

    Article  Google Scholar 

  • Marcos S, García Peñalvo FJ, Vázquez Ingelmo A.: Emotional ai in healthcare: A pilot architecture proposal to merge emotion recognition tools. In: Ninth International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM’21), 342–349 (2021)

  • Marinoiu E, Zanfir M, Olaru V, Sminchisescu C.: 3d human sensing, action and emotion recognition in robot assisted therapy of children with autism. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2158–2167 (2018)

  • Matziorinis AM, Koelsch S. The promise of music therapy for alzheimer’s disease: A review. Ann N Y Acad Sci. 2022;1516(1):11–7.

    Article  Google Scholar 

  • McIntosh LG, Mannava S, Camalier CR, Folley BS, Albritton A, Konrad PE, Charles D, Park S, Neimat JS. Emotion recognition in early parkinson’s disease patients undergoing deep brain stimulation or dopaminergic therapy: a comparison to healthy participants. Front Aging Neurosci. 2015;6:349.

    Article  Google Scholar 

  • Organização Mundial da Saúde, O.M.S (218) Ageing. Available in: https://www.who.int/health−topics/ageingtab=tab1

  • Muyuan W, Naiyao Z, Hancheng Z: User-adaptive music emotion recognition. In: Proceedings 7th International Conference on Signal Processing, 2004. Proceedings. ICSP’04. 2004. 2 1352–1355 (2004). IEEE

  • Ng H-W, Nguyen VD, Vonikakis V, Winkler S: Deep learning for emotion recognition on small datasets using transfer learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, 443–449 (2015)

  • Olanrewaju O, Clare L, Barnes L, Brayne C. A multimodal approach to dementia prevention: a report from the cambridge institute of public health. Alzheimer’s & Dementia: Transl Res Clin Interv. 2015;1(3):151–6.

    Google Scholar 

  • Pal M. Random forest classifier for remote sensing classification. Int J Remote Sens. 2005;26(1):217–22.

    Article  Google Scholar 

  • Picard RW: Affective Computing. MIT press, ??? (2000)

  • Platt J.: Fast training of support vector machines using sequential minimal optimization. In: Schoelkopf, B., Burges, C., Smola, A. (eds.) Advances in Kernel Methods - Support Vector Learning. MIT Press, ??? (1998). http://research.microsoft.com/texttildelowjplatt/smo.html

  • Pujol FA, Mora H, Martínez A.: Emotion recognition to improve ehealthcare systems in smart cities. In: Research & Innovation Forum 2019: Technology, Innovation, Education, and Their Social Impact 1, 245–254 (2019). Springer

  • Reyes BN, Segal SC, Moulson MC. An investigation of the effect of race-based social categorization on adults’ recognition of emotion. PLoS ONE. 2018;13(2):0192418.

    Article  Google Scholar 

  • Russell JA. A Circumplex Model of Affect. J Pers Soc Psychol. 1980;39(6):1161–78. https://doi.org/10.1037/h0077714.

    Article  Google Scholar 

  • Saganowski S, Dutkowiak A, Dziadek A, Dziezyc M, Komoszy˙nska J, Michalska W, Polak A, Ujma M, Kazienko P.: Emotion recognition using wearables: A systematic literature review-work-in-progress. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), 1–6 (2020). IEEE

  • Santana MA, Pereira JMS, Silva FLD, Lima NMD, Sousa FND, Arruda GMSD, Lima RDCFD, Silva WWAD, Santos WPD. Breast cancer diagnosis based on mammary thermography and extreme learning machines. Res Biomed Eng. 2018;34:45–53.

    Article  Google Scholar 

  • Santana MA, Gomes JC, de Souza GM, Suarez A, Torcate AS, Fonseca FS, Moreno GMM, dos Santos WP. Reconhecimento automático de emoçoes a partir de sinais multimodais e inteligência artificial. Anais Do IV Simpósio De Inovação Em Engenharia Biomédica-SABIO. 2020a;2020:43.

    Google Scholar 

  • Santana MA, Pereira JMS, Lima RCF, Santos WP. Breast lesions classification in frontal thermographic images using intelligent systems and moments of haralick and zernike. In: dos Santos WP, de Santana MA, da Silva WWA, editors. Understanding a Cancer Diagnosis. 1st ed. New York: Nova Science; 2020b. p. 65–80.

    Google Scholar 

  • dos Santos WP, de Assis FM, de Souza RE, dos Santos Filho PB.: Evaluation of alzheimer’s disease by analysis of mr images using objective dialectical classifiers as an alternative to adc maps. In: 2008a 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 5506–5509 (2008a). IEEE

  • dos Santos WP, de Souza RE, Santos Filho PB, Neto FBL, de Assis FM.: A dialectical approach for classification of DW-MR Alzheimer’s images. In: 2008b IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), 1728–1735 (2008b). IEEE

  • dos Santos WP, De Assis FM, De Souza RE, Mendes PB, de Souza Monteiro HS, Alves HD.: A dialectical method to classify alzheimer’s magnetic resonance images. In: dos Santos, W.P. (ed.) Evolutionary Computation 473. IntechOpen, ??? (2009a)

  • dos Santos WP, de Assis FM, de Souza RE, dos Santos Filho PB.: Dialectical classification of MR images for the evaluation of Alzheimer’s disease. In: Naik, G.R. (ed.) Recent Advances in Biomedical Engineering. IntechOpen, ??? (2009b)

  • Saxena A, Khanna A, Gupta D. Emotion recognition and detection methods: A comprehensive survey. J Artif Intell Syst. 2020;2(1):53–79.

    Google Scholar 

  • Schipor O, Pentiuc S, Schipor M. The utilization of feedback and emotion recognition in computer based speech therapy system. Elektronika Ir Elektrotechnika. 2011;109(3):101–4.

    Article  Google Scholar 

  • Schuller B, Rigoll G, Lang M.: Hidden markov model-based speech emotion recognition. In: 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)., 2, 1 (2003). IEEE

  • Shafqat S. Alzheimer disease therapeutics: perspectives from the develo** world. J Alzheimer’s Disease. 2008;15(2):285–7.

    Article  Google Scholar 

  • Shu Y, Wang S.: Emotion recognition through integrating eeg and peripheral signals. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2871–2875 (2017). IEEE

  • Silva WWA, Santana MA, Silva Filho AG, Lima SML, Santos WP. Morphological extreme learning machines applied to the detection and classification of mammary lesions. In: Gandhi TK, Bhattacharyya S, De S, Konar D, Dey S, editors. Advanced Machine Vision Paradigms for Medical Image Analysis. London: Elsevier; 2020.

    Google Scholar 

  • Silva IR, Silva GS, de Souza RG, dos Santos WP, Fagundes RADA.: Model based on deep feature extraction for diagnosis of alzheimer’s disease. In: 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–7 (2019). IEEE

  • Sinha N.: Affective computing and emotion-sensing technology for emotion recognition in mood disorders. Enhanced Telemedicine and e-Health: Advanced IoT Enabled Soft Computing Framework, 337–360 (2021)

  • Soleymani M, Lichtenauer J, Pun T, Pantic M. A multimodal database for affect recognition and implicit tagging. IEEE Trans Affect Comput. 2011;3(1):42–55.

    Article  Google Scholar 

  • Sörensen S, Duberstein P, Gill D, Pinquart M. Dementia care: mental health effects, intervention strategies, and clinical implications. The Lancet Neurology. 2006;5(11):961–73.

    Article  Google Scholar 

  • Sourina O, Liu Y, Nguyen MK. Real-time eeg-based emotion recognition for music therapy. J Multimodal User Interfaces. 2012;5(1):27–35.

    Article  Google Scholar 

  • de Souza RG, dos Santos Lucas e Silva G, dos Santos WP, de Lima ME, Initiative, A.D.N 2021 Computer-aided diagnosis of Alzheimer’s disease by MRI analysis and evolutionary computing. Res Biomed Eng 37, 455–483

  • Van der Steen JT, Smaling HJ, Van der Wouden JC, Bruinsma MS, Scholten RJ, Vink AC.: Music-based therapeutic interventions for people with dementia. Cochrane Database of Systematic Reviews (7) (2018)

  • Vijayakumar S, Flynn R, Murray N.: A comparative study of machine learning techniques for emotion recognition from peripheral physiological signals. In: 2020 31st Irish Signals and Systems Conference (ISSC), 1–6 (2020). IEEE

  • Wei W, Jia Q, Feng Y, Chen G.: Emotion recognition based on weighted fusion strategy of multichannel physiological signals. Comput Intell Neurosci 2018 (2018)

  • Wiem MBH, Lachiri Z.: Emotion classification in arousal valence model using mahnob-hci database. Int J Adv Comput Sci App 8(3) (2017)

  • Witten IH, Frank E. Data Mining: Pratical Machine Learning Tools and Technique. San Francisco, CA, USA: Morgan Kaufmann Publishers; 2005.

    MATH  Google Scholar 

  • Yang Y-H, Lin Y-C, Chen H.: Personalized music emotion recognition. In: Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 748–749 (2009)

  • Zeng J, Roussis PC, Mohammed AS, Maraveas C, Fatemi SA, Armaghani DJ, Asteris PG.: Prediction of peak particle velocity caused by blasting through the combinations of boosted-chaid and svm models with various kernels. Applied Sciences 11(8) (2021). 10.3390/ app11083705

  • Zhang X-X, Tian Y, Wang Z-T, Ma Y-H, Tan L, Yu J-T. The epidemiology of alzheimer’s disease modifiable risk factors and prevention. J Prev Alzheimer’s Disease. 2021;8:313–21.

    Google Scholar 

Download references

Funding

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior—Brasil (CAPES)—Finance Code 001. The authors are also grateful to the Fundação de Amparo à Ciência e Tecnologia do Estado de Pernambuco, FACEPE, Brazil, under the code IBPG-0013–1.03/20, and to Conselho Nacional de Desenvolvimento Científico e Tecnológico, CNPq, Brazil, for the partial financial support of this research.

Author information

Authors and Affiliations

  1. Núcleo de Engenharia da Computação, Escola Politécnica da Universidade de Pernambuco, Recife, PE, Brazil

    Maíra Araújo de Santana, Flávio Secco Fonseca, Arianne Sarmento Torcate & Wellington Pinheiro dos Santos

  2. Departamento de Engenharia Biomédica, Universidade Federal de Pernambuco, Recife, PE, Brazil

    Wellington Pinheiro dos Santos

Authors
  1. Maíra Araújo de Santana
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Flávio Secco Fonseca
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Arianne Sarmento Torcate
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wellington Pinheiro dos Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

M. S.: Conceptualization, methodology, investigation, writing.

F. F.: Conceptualization, methodology, investigation, writing.

A. T.: Conceptualization, methodology, investigation, writing.

W. P.: Conceptualization, methodology, investigation, writing, scientific supervision.

Corresponding author

Correspondence to Wellington Pinheiro dos Santos.

Ethics declarations

Conflict of interest

The authors declare no conflicts of interest.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

de Santana, M.A., Fonseca, F.S., Torcate, A.S. et al. Emotion Recognition from Multimodal Data: a machine learning approach combining classical and hybrid deep architectures. Res. Biomed. Eng. 39, 613–638 (2023). https://doi.org/10.1007/s42600-023-00293-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42600-023-00293-9

Keywords

Search

Navigation