SpringerLink
Log in

A deep perceptual framework for affective video tagging through multiband EEG signals modeling

  • S.I.: Knowledge services: challenges and future prospects
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Nowadays, multimedia content, such as photographs and movies, is ingrained in every aspect of human lives and has become a vital component of their entertainment. Multimedia content, such as videos or movie clips, is typically created with the intent to evoke certain feelings or emotions in viewers. Thus, by examining the viewer’s cognitive state while watching such content, its affectiveness can be evaluated. Considering the emotional aspect of videos, in this paper, a deep learning-based paradigm for affective tagging of video clips is proposed, in which participants’ irrational EEG responses are used to examine how people perceive videos. The information behind different brain regions, frequency waves, and connections among them play an important role in understanding a human’s cognitive state. Thus, here a contribution is made toward the effective modeling of EEG signals through two different representations, i.e., spatial feature matrix and combined power spectral density maps. The proposed feature representations highlight the spatial features of EEG signals and are therefore used to train a convolution neural network model for implicit tagging of two categories of videos in the Arousal domain, i.e., “Low Arousal” and “High Arousal.” The arousal emotional space represents the excitement level of the viewer; thus, this domain is selected to analyze the viewer’s engagement while watching video clips. The proposed model is developed using the EEG data taken from publicly available datasets “AMIGOS” and “DREAMER.” The model is tested using two different approaches, i.e., single-subject classification and multi-subject classification, and an average accuracy of 90%-95% and 90%-93% is achieved, respectively. The simulations presented in this paper show the pioneering applicability of the proposed framework for the development of brain–computer interface (BCI) devices for affective tagging of videos.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Thailand)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data and material availability

AMIGOS, DREAMER.

Code availability

Custom code.

References

  1. Caviedes JE (2012) The evolution of video processing technology and its main drivers. Proc IEEE 100(4):872–877. https://doi.org/10.1109/JPROC.2011.2182072

    Article  Google Scholar 

  2. Pouyanfar S, Yang Y, Chen SC, Shyu ML, Iyengar SS (2018) Multimedia Big Data Analytics. ACM Comput Surv 51(1):1–34. https://doi.org/10.1145/3150226

    Article  Google Scholar 

  3. Pereira F, Ascenso J, Brites C, Fonseca P, Pinho P, Baltazar J (2007) Evolution and Challenges in Multimedia Representation Technologies. In: M.S. Pereira (Ed) A Portrait of State-of-the-Art Research at the Technical University of Lisbon. Springer, Dordrecht, 2007. 275–294. https://doi.org/10.1007/978-1-4020-5690-1

  4. Scherp A, Mezaris V (2014) Survey on modeling and indexing events. Multimedia Tools Appl 70:7–23. https://doi.org/10.1007/s11042-013-1427-7

    Article  Google Scholar 

  5. Baveye Y, Chamaret C, Dellandréa E, Chen L (2018) Affective video content analysis: a multidisciplinary insight. IEEE Trans Affect Comput 9(4):396–409. https://doi.org/10.1109/TAFFC.2017.2661284

    Article  Google Scholar 

  6. Hanjalic A, Xu L (2005) Affective video content representation and modeling. IEEE Trans Multimed 7(1):143–154. https://doi.org/10.1109/TMM.2004.840618

    Article  Google Scholar 

  7. Assabumrungrat R, Sangnark S, Charoenpattarawut T, Polpakdee W, Sudhawiyangkul T, Boonchieng E, Wilaiprasitporn T (2022) ubiquitous affective computing: a review. IEEE Sens J 22:1867–1881. https://doi.org/10.1109/jsen.2021.3138269

    Article  Google Scholar 

  8. Wang D, Zhao X (2022) Affective video recommender systems: a survey. Front Neurosci 16:984404. https://doi.org/10.3389/fnins.2022.984404

    Article  Google Scholar 

  9. Slaney M (2011) Web-scale multimedia analysis: does content matter? IEEE Multimedia 18(2):12–15. https://doi.org/10.1109/mmul.2011.34

    Article  Google Scholar 

  10. Dimitrova N, Zhang HJ, Shahraray B, Sezan I, Huang T, Zakhor A (2002) Applications of video-content analysis and retrieval. IEEE Multimed 9(3):42–55. https://doi.org/10.1109/MMUL.2002.1022858

    Article  Google Scholar 

  11. Smith MA, Chen T (2005) 9.1: image and video indexing and retrieval. In: Bovik AL (ed) In: communications, networking and multimedia, handbook of image and video processing, 2nd edn. Academic Press, New York. https://doi.org/10.1016/B978-012119792-6/50121-2

  12. Müller V, Boden MA (2008) Mind as machine: a history of cognitive science 2 vols. Mind Mach 18:121–125. https://doi.org/10.1007/s11023-008-9091-9

    Article  Google Scholar 

  13. Hassanien AE, Azar A (2014) Brain computer interfaces: current trends and applications, intelligent systems reference library, vol 74. Springer, Cham

    Google Scholar 

  14. Ghaemmaghami P (2017) Information retrieval from neurophysiological signals. Ph.D. Thesis. University of Trento. Canada

  15. Zabcikova M, Koudelkova Z, Jasek R, Lorenzo Navarro JJ (2022) Recent advances and current trends in brain-computer interface research and their applications. Int J Dev Neurosci 82:107–123. https://doi.org/10.1002/jdn.10166

    Article  Google Scholar 

  16. Alarcao SM, Fonseca MJ (2018) Emotions recognition using EEG signals: a survey. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2017.2714671

    Article  Google Scholar 

  17. Yang X, Yan J, Wang W, Li S, Hu B (2022) Lin J (2022) Brain-inspired models for visual object recognition: an overview. Artif Intell Rev 55:5263–5311. https://doi.org/10.1007/s10462-021-10130-z

    Article  Google Scholar 

  18. Sharma S, Dubey AK, Ranjan P, Rocha A (2023) Neural correlates of affective content: application to perceptual tagging of video. Neural Comput & Applic 35:7925–7941. https://doi.org/10.1007/s00521-021-06591-6

    Article  Google Scholar 

  19. Sharma S, Dubey AK, Ranjan P (2022) Affective video tagging framework using human attention modelling through EEG signals. International Journal of Intelligent Information Technologies (IJIIT) 18(1):1–18. https://doi.org/10.4018/IJIIT.306968

    Article  Google Scholar 

  20. Gawali BW, Rao S, Abhang P, Rokade P, Mehrotra SC (2012) Classification of EEG signals for different emotional states. In: Fourth international conference on advances in recent technologies in communication and computing (ARTCom2012), pp 177–181. https://doi.org/10.1049/cp.2012.2521

  21. Li J, Zhang Z, He H (2018) Hierarchical convolutional neural networks for EEG-based emotion recognition. Cogn Comput 10:368–380. https://doi.org/10.1007/s12559-017-9533-x

    Article  Google Scholar 

  22. Hiyoshi-Taniguchi K, Kawasaki M, Yokota T, Bakardjian H, Fukuyama H, Cichocki A, Vialatte FB (2015) EEG correlates of voice and face emotional judgments in the human brain. Cogn Comput 7:11–19. https://doi.org/10.1007/s12559-013-9225-0

    Article  Google Scholar 

  23. Frydenlund A, Rudzicz F (2015) Emotional affect estimation using video and EEG data in deep neural networks. In: Barbosa D, Milios E (eds) Advances in artificial intelligence. Canadian AI 2015. Lecture notes in computer science, vol 9091. Springer, Cham. https://doi.org/10.1007/978-3-319-18356-5_24

  24. Hee Lin Wang (2006) Loong-fah cheong: affective understanding in film. IEEE Trans Circuits Syst Video Technol 16:689–704. https://doi.org/10.1109/tcsvt.2006.873781

    Article  Google Scholar 

  25. Soleymani M, Pantic M (2013) Multimedia implicit tagging using EEG signals. In: 2013 IEEE international conference on multimedia and expo (ICME), San Jose, CA, USA, 2013, pp 1–6. https://doi.org/10.1109/ICME.2013.6607623

  26. Koelstra S, Muhl C, Patras I (2009) EEG analysis for implicit tagging of video data. In: 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops. ACII 2009. IEEE, Amsterdam, Netherlands. pp 1–6 https://doi.org/10.1109/acii.2009.5349482.

  27. Garg D, Verma GK, Singh AK (2023) A review of deep learning based methods for affect analysis using physiological signals. Multimed Tools Appl 82:26089–26134. https://doi.org/10.1007/s11042-023-14354-9

    Article  Google Scholar 

  28. Vecchiato G, Cherubino P, Maglione AG, Ezquierro MT, Marinozzi F, Bini F, Trettel A, Babiloni F (2014) How to measure cerebral correlates of emotions in marketing relevant tasks. Cogn Comput 6:856–871. https://doi.org/10.1007/s12559-014-9304-x

    Article  Google Scholar 

  29. Kumar S, Riddoch MJ, Humphreys G (2013) Mu rhythm desynchronization reveals motoric influences of hand action on object recognition. Front Hum Neurosci 7:66. https://doi.org/10.3389/fnhum.2013.00066

    Article  Google Scholar 

  30. Sharma S, Mishra A, Kumar S, Ranjan P, Ujlayan A (2018) Analysis of action oriented effects on perceptual process of object recognition using physiological responses. In: Tiwary, U. (ed) Intelligent Human Computer Interaction. IHCI 2018. Lecture Notes in Computer Science. 46–58. doi: https://doi.org/10.1007/978-3-030-04021-5_5.

  31. Padfield N, Zabalza J, Zhao H, Vargas VM, Ren J (2019) EEG-based brain-computer interfaces using motor-imagery: techniques and challenges. Sensors. https://doi.org/10.3390/s19061423

    Article  Google Scholar 

  32. Kumar JS, Bhuvaneswari P (2012) Analysis of electroencephalography (EEG) signals and its categorization–a study. Procedia Eng 38:2525–2536. https://doi.org/10.1016/j.proeng.2012.06.298

    Article  Google Scholar 

  33. Bigdely-Shamlo N, Vankov A, Ramirez RR, Makeig S (2008) Brain activity-based image classification from rapid serial visual presentation. IEEE Trans Neural Syst Rehabil Eng 16(5):432–441. https://doi.org/10.1109/TNSRE.2008.2003381

    Article  Google Scholar 

  34. Wang J, Pohlmeyer E, Hanna B, Jiang YG, Sajda,P, Chang SF (2009) Brain state decoding for rapid image retrieval. In: Proceedings of the 17th ACM international conference on multimedia, pp 945–954. ACM, New York. https://doi.org/10.1145/1631272.1631463

  35. Huang Y, Erdogmus D, Pavel M, Mathan S, Hild KE (2011) A framework for rapid visual image search using single-trial brain evoked responses. Neurocomputing 74(12):2041–2051. https://doi.org/10.1016/j.neucom.2010.12.025

    Article  Google Scholar 

  36. Lees S, Dayan N, Cecotti H, McCullagh P, Maguire L, Lotte F, Coyle D (2018) A review of rapid serial visual presentation-based brain- computer interfaces. J Neural Eng 15(2):021001. https://doi.org/10.1088/1741-2552/aa9817

    Article  Google Scholar 

  37. Kapoor A, Shenoy P (2008) Combining brain computer interfaces with vision for object categorization. In: 2008 IEEE conference on computer vision and pattern recognition, pp 1–8. https://doi.org/10.1109/CVPR.2008.4587618

  38. Mohedano E, Healy G, McGuinness K, Giró-i-Nieto X, O’Connor NE, Smeaton AF (2014) Object segmentation in images using EEG signals. In: Proceedings of the 22Nd ACM international conference on multimedia, pp 417–426. ACM, New York. https://doi.org/10.1145/2647868.2654896

  39. Mohedano E, McGuinness K, Healy G, O’Connor NE, Smeaton AF, Salvador A, Porta S, Nieto XG (2015) Exploring EEG for object detection and retrieval. In: Proceedings of the 5th ACM on international conference on multimedia retrieval, pp 591–594. ACM, New York. https://doi.org/10.1145/2671188.2749368

  40. Healy G, Smeaton AF (2011) Optimising the number of channels in EEG-augmented image search. In: Proceedings of the 25th BCS conference on human–computer interaction, pp 157–162. British Computer Society, Swinton

  41. Tauscher JP, Mustafa M, Magnor M (2017) Comparative analysis of three different modalities for perception of artifacts in videos. ACM Trans Appl Percept. https://doi.org/10.1145/3129289

    Article  Google Scholar 

  42. Mutasim AK, Tipu RS, Bashar MR, Amin MA (2017) Video category classification using wireless EEG. In: Zeng Y, He Y, Kotaleski JH, Martone M, Xu B, Peng H, Luo Q (eds) Brain informatics. Lecture notes in computer science, vol 10654. Springer, Cham, pp 39–48. https://doi.org/10.1007/978-3-319-70772-3_4

  43. Soleymani M, Lichtenauer J, Pun T, Pantic M (2012) A multimodal database for affect recognition and implicit tagging. IEEE Trans Affect Comput 3(1):42–55. https://doi.org/10.1109/T-AFFC.2011.25

    Article  Google Scholar 

  44. Wang S, Zhu Y, Wu G, Ji Q (2014) Hybrid video emotional tagging using users’ EEG and video content. Multimed Tools Appl 72:1257–1283. https://doi.org/10.1007/s11042-013-1450-8

    Article  Google Scholar 

  45. Martínez-Rodrigo A, García-Martínez B, Huerta Á, Alcaraz R (2021) Detection of negative stress through spectral features of electroencephalographic recordings and a convolutional neural network. Sensors 21:3050. https://doi.org/10.3390/s21093050

    Article  Google Scholar 

  46. Mishra A, Ranjan P, Ujlayan A (2020) Empirical analysis of deep learning networks for affective video tagging. Multimed Tools Appl 79:18611–18626. https://doi.org/10.1007/s11042-020-08714-y

    Article  Google Scholar 

  47. Jang S, Moon S-E, Lee J-S (2018) Eeg-based video identification using graph signal modeling and graph convolutional neural network. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) Calgary, AB, Canada, 2018, pp. 3066–3070. https://doi.org/10.1109/icassp.2018.8462207.

  48. Correa JAM, Abadi MK, Sebe N, Patras I (2018) AMIGOS: a dataset for affect, personality and mood research on individuals and groups. IEEE Trans Affect Comput 12(2):479–493. https://doi.org/10.1109/TAFFC.2018.2884461

    Article  Google Scholar 

  49. Katsigiannis S, Ramzan N (2018) DREAMER: a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J Biomed Health Inform 22:98–107. https://doi.org/10.1109/jbhi.2017.2688239

    Article  Google Scholar 

  50. Koelstra S, Mühl C, Soleymani M, Jong-Seok L, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2012) Deap: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3(1):18–31

    Article  Google Scholar 

  51. Abadi MK, Subramanian R, Kia SM, Avesani P, Patras I, Sebe N (2015) DECAF: MEG-based multimodal database for decoding affective physiological responses. IEEE Trans Affect Comput 6(3):209–222. https://doi.org/10.1109/TAFFC.2015.2392932

    Article  Google Scholar 

  52. Mallat S (1989) A theory for multiresolution signal decomposition: the wavelet representation. IEEE Trans Pattern Anal Mach Intell 11(7):674–693. https://doi.org/10.1109/34.192463

    Article  MATH  Google Scholar 

  53. Kehtarnavaz N (2008) Chapter 7: frequency domain processing. In: Kehtarnavaz N (ed) Digital signal processing system design, 2nd edn. Academic Press, London, pp 175–196. https://doi.org/10.1016/B978-0-12-374490-6.00007-6

  54. Vivas EL, García-González A, Figueroa I, Fuentes RQ (2013) Discrete wavelet transform and ANFIS classifier for brain-machine interface based on EEG. In: 2013 6th international conference on human system interactions (HSI), pp 137–144. https://doi.org/10.1109/HSI.2013.6577814

  55. Subasi A (2007) EEG signal classification using wavelet feature extraction and a mixture of expert model. Expert Syst Appl 32:1084–1093. https://doi.org/10.1016/j.eswa.2006.02.005

    Article  Google Scholar 

  56. Welch P (1967) The use of fast Fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Trans Audio Electroacoust 15(2):70–73. https://doi.org/10.1109/TAU.1967.1161901

    Article  Google Scholar 

  57. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. ar**v preprint ar**v:1409.

  58. Asghar MA, Fawad Khan MJ, Amin Y, Akram A (2020) EEG-based Emotion Recognition for Multi-Channel Fast Empirical Mode Decomposition using VGG-16. International Conference on Engineering and Emerging Technologies (ICEET). Lahore, Pakistan, 2020, pp. 1–7. https://doi.org/10.1109/iceet48479.2020.9048217

Download references

Acknowledgements

The authors are grateful to the database creators for granting us access to the dataset.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and Affiliations

  1. Amity School of Engineering and Technology, Amity University Uttar Pradesh, Noida, 201313, UP, India

    Shanu Sharma & Ashwani Kumar Dubey

  2. Department of Computer Science and Engineering, ABES Engineering College, Ghaziabad, 201009, India

    Shanu Sharma

  3. School of Computer Science (SoCS), Internet of Things - Center of Excellence, University of Petroleum and Energy Studies (UPES), Dehradun, 248007, India

    Priya Ranjan

  4. ISEG, University of Lisbon, Rua Do Quelhas, N86, 1200-781, Lisbon, Portugal

    Alvaro Rocha

Authors
  1. Shanu Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ashwani Kumar Dubey
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Priya Ranjan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alvaro Rocha
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

SS: Conceptualization, Methodology, Software, Data Curation, Validation, Writing- Original Draft Preparation. AKD: Conceptualization, Methodology, Supervision, Reviewing, and Editing. PR: Conceptualization, Supervision, Reviewing, and Editing. AR: Reviewing and Editing.

Corresponding author

Correspondence to Ashwani Kumar Dubey.

Ethics declarations

Conflicts of interest

Authors declare that they have no conflict of interest.

Ethical approval

The article includes approaches used on datasets AMIGOS and DREAMER that are available to the public. According to the dataset description, participants gave the developers their written approval before participating.

Animal Rights

There are no animal studies conducted by any of the authors in this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sharma, S., Dubey, A.K., Ranjan, P. et al. A deep perceptual framework for affective video tagging through multiband EEG signals modeling. Neural Comput & Applic (2023). https://doi.org/10.1007/s00521-023-09086-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00521-023-09086-8

Keywords

Search

Navigation