Abstract
Geometric and textural features have been separately used in the literature for facial emotion classification. In this paper, we theoretically and empirically study the capture of facial expressions through geometric and texture-based features, and demonstrate that a simple concatenation of these features can lead to significant improvement in facial emotion classification. We also propose the use of the Directed Acyclic Graph SVM (DAGSVM) for facial emotion classification using the concatenated feature, by analyzing DAGSVM structures. We perform experiments using the well-known extended Cohn-Kanade (CK+), the MUG facial expression (MUG) and the Japanese Female Facial Expression (JAFFE) databases to evaluate the integration of geometric and textural features, and the use of DAGSVM for facial emotion classification. The said integration is found to be effective and DAGSVM is found to be computationally efficient in facial emotion classification.
Similar content being viewed by others
References
Aifanti N, Papachristou C, Delopoulos A (2010) The mug facial expression database. In: Proceedings of the international workshop on image analysis for multimedia interactive services, pp 1–4
Agarwal S, Mukherjee DP (2017) Facial expression recognition through adaptive learning of local motion descriptor. Multimedia Tools and Applications 76(1):1073–1099
Bartlett MS, Littlewort G, Fasel I, Movellan JR (2003) Real time face detection and facial expression recognition: development and applications to human computer interaction. In: Proceedings of the IEEE conference on computer vision and pattern recognition - workshop, pp 53–53
Bashyal S, Venayagamoorthy GK (2008) Recognition of facial expressions using Gabor wavelets and learning vector quantization. Eng Appl Artif Intell 21:1056–1064
Bazrafkan S, Nedelcu T, Filipczuk P, Corcoran P (2017) Deep learning for facial expression recognition: A step closer to a smartphone that knows your moods. In: Proceedings of the IEEE international conference on consumer electronics (ICCE), pp 217–220. https://www.hindawi.com/journals/ahci/2014/408953/
Bickel S, Scheffer T (2004) Multi-view clustering. In: IEEE International Conference on Data Mining, pp 19–26
Blum A, Mitchell T (1998) Combining labeled and unlabeled data with co-training. In: International conference on computational learning theory, pp 92–100
Buciu I, Pitas I (2004) Application of non-negative and local non negative matrix factorization to facial expression recognition. In: Proceedings of the international conference on pattern recognition (ICPR), pp 288–291
Busso C, Bulut M, Lee C-C, Kazemzadeh A, Mower E, Kim S, Chang JN, Lee S, Narayanan SS (2008) Iemocap: interactive emotional dyadic motion capture database. Lang Resour Eval 42(4):335
Chaudhuri K, Kakade S, Livescu K, Sridharan K (2009) Multi-view clustering via canonical correlation analysis. In: International Conference on Machine Learning, pp 129–136
Cootes TF, Taylor CJ, Cooper DH, Graham J (1995) Active shape models - their training and application. Comp Vision Image Underst (CVIU) 61(1):38–59
Cootes TF, Edwards GJ, Taylor CJ (2001) Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(6):681–685
Cohen I, Sebe N, Garg A, Chen LS, Huang TS (2003) Facial expression recognition from video sequences: temporal and static modeling. Comput Vis Image Underst 91:160–187
Cruz A, Bhanu B, Yang S (2011) A psychologically-inspired match-score fusion mode for video-based facial expression recognition. In: Proceedings of the international conference on affective computing and intelligent interaction, pp 341–350
Dahmane M, Meunier J (2011) Continuous emotion recognition using Gabor energy filters. In: Proceedings of the international conference on affective computing and intelligent interaction, pp 351–358
Datta S, Sen D, Balasubramanian R (2016) Integrating geometric and textural features for facial emotion classification using SVM frameworks. In: Proceedings of the international conference on computer vision and image processing, pp 619–628
Dhall A, Goecke R, Joshi J, Wagner M, Gedeon T (2013) Emotion recognition in the wild challenge 2013. In: Proceedings of the ACM on international conference on multimodal interaction, pp 509–516
Ding C, Choi J, Tao D, Davis LS (2016) Multi-directional multi-level dual-cross patterns for robust face recognition. IEEE Trans Pattern Anal Mach Intell 38(3):518–531
Douglas-Cowie E, Cowie R, Cox C, Amier N, Heylen DK (2008) The sensitive artificial listner: an induction technique for generating emotionally coloured conversation. In: Proceedings of the LREC workshop on corpora for research on emotion and affect, no WP 08-02, pp 1–4
Farquhar J, Hardoon D, Meng H, Shawe-taylor JS, Szedmák S (2006) Two view learning: Svm-2k, theory and practice. In: Weiss Y, Schölkopf B, Platt JC (eds) Advances in neural information processing systems 18, pp 355–362
Glodek M, Tschechne S, Layher G, Schels M, Brosch T, Scherer S, Kächele M., Schmidt M, Neumann H, Palm G, Schwenker F (2011) Multiple classifier systems for the classification of audio-visual emotional states. In: Proceedings of the international conference on affective computing and intelligent interaction, pp 359–368
Gritti T, Shan C, Jeanne V, Braspenning R (2008) Local features based facial expression recognition with face registration errors. In: Proceedings of the IEEE international conference on automatic face gesture recognition, pp 1–8
Gu W, **ang C, Venkatesh YV, Huang D, Lin H (2012) Facial expression recognition using radial encoding of local Gabor features and classifier synthesis. Pattern Recogn 45(1):80–91
Huang KC, Huang SY, Kuo YH (2010) Emotion recognition based on a novel triangular facial feature extraction method. In: Proceedings of the international joint conference on neural networks (IJCNN), pp 1–6
Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings of the IEEE international conference on automatic face and gesture recognition, pp 46–53
Knill O (2009) Probability and stochastic processes with applications. Overseas Press, New Delhi
Krebel U (1999) Pairwise classification and support vector machine, ser. Advances in Kernel Methods: support vector learning. MIT Press, Cambridge, pp 255–268
Kumar A, Rai P, DauméIII H (2011) Co-regularized spectral clustering with multiple kernels. In: International Conference on Neural Information Processing Systems, pp 1413–1421
Jeni LA, Girard JM, Cohn JF, Torre FDL (2013) Continuous AU intensity estimation using localized, sparse facial feature space. In: Proceedings of the IEEE international conference and workshops on automatic face and gesture recognition (FG), pp 1–7
Jiang B, Valstar MF, Pantic M (2011) Action unit detection using sparse appearance descriptors in space-time video volumes. In: Face and Gesture, pp 314–321
Jiang B, Valstar M, Martinez B, Pantic M (2014) A dynamic appearance descriptor approach to facial actions temporal modeling. IEEE Transactions on Cybernetics 44(2):161–174
Kotsia I, Pitas I (2007) Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans Image Process 16(1):172–187
Lyons MJ, Budynek J, Akamatsu S (1999) Automatic classification of single facial images. IEEE Transactions on Pattern Analysis and Machine Intelligence 21:1357–1362
Liu P, Yin L (2015) Spontaneous facial expression analysis based on temperature changes and head motions. In: Proceedings of the IEEE international conference and workshops on automatic face and gesture recognition (FG), vol 1, pp 1–6
Liu Y, Nie L, Han L, Zhang L, Rosenblum D (2015) Action2activity: Recognizing complex activities from sensor data. In: International Joint Conference on Artificial Intelligence, pp 1617–1623
Liu Y, Nie L, Liu L, Rosenblum DS (2016) From action to activity: Sensor-based activity recognition. Neurocomputing 181:108–115
Liu Y, Zhang L, Nie L, Yan Y, Rosenblum D (2016) Fortune teller predicting your career path. In: AAAI Conference on Artificial Intelligence, pp 201–207
Liu W, Yang X, Tao D, Cheng J, Tang Y (2018) Multiview dimension reduction via hessian multiset canonical correlations. Information Fusion 41:119–128
Long F, Wu T, Movellan JR, Bartlett MS, Littlewort G (2012) Learning spatiotemporal features by using independent component analysis with application to facial expression recognition. Neurocomputing 93:126–132
Lyons MJ, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with Gabor wavelets. In: Proceedings of the IEEE international conference on automatic face and gesture recognition, pp 200–205
Lucey S, Ashraf AB, Cohn J (2007) Face Recognition Germany: Pro Literatur Verlag, ch. Investigating spontaneous facial action recognition through AAM representations of the face, pp 395–406
Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kande dataset (ck+): A complete facial expression dataset for action unit and emotion-specified expression. In: Proceedings of the IEEE computer vision and pattern recognition - workshops, pp 94–101
Luo Y, Liu T, Tao D, Xu C (2015) Multiview matrix completion for multilabel image classification. IEEE Trans Image Processing 24(8):2355–2368
Mariooryad S, Busso C (2013) Feature and model level compensation of lexical content for facial emotion recognition. In: Proceedings of the IEEE international conference and workshops on automatic face and gesture recognition (FG), pp 1–6
Mariooryad S (2016) Facial expression recognition in the presence of speech using blind lexical compensation. IEEE Trans Affect Comput 7(4):346–359
McKeown G, Valstar MF, Cowie R, Pantic M (2010) The semaine corpus of emotionally coloured character interactions. In: Proceedings of the IEEE international conference on multimedia and expo, pp 1079–1084
Meng H, Romera-paredes B, Bianchi-Berthouze N (2011) Emotion recognition by two view SVM_2k classifier on dynamic facial expression features. In: Face and gesture, pp 854–859
Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: Proceedings of the IEEE winter conference on applications of computer vision (WACV), pp 1–10
Milborrow S, Nicolls F (2014) Active shape models with SIFT descriptors and MARS. In: Proceedings of the international conference on computer vision theory and applications (VISAPP), vol 2, pp 380–387
Nicolle J, Rapp V, Bailly K, Prevost L, Chetouani M (2012) Robust continuous prediction of human emotions using multiscale dynamic cues. In: Proceedings of the ACM international conference on multimodal interaction, pp 501–508
Niese R, Al-Hamadi A, Farag A, Neumann H, Michaelis B (2012) Facial expression recognition based on geometric and optical flow features in colour image sequences. IET Comput Vis 6(2):79–89
Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24:971–987
Pantic M, Rothkrantz JM (2004) Facial action recognition for facial expression analysis from static face images. IEEE Transactions on Systems, Man and Cybernetics Part B 34(3):1449–1461
Pantic M, Valstar MF, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: Proceedings of the IEEE international conference on multimedia and expo, Amsterdam, The Netherlands, pp 317–321
Pantic M, Patras I (2005) Detecting facial actions and their temporal segments in nearly frontal-view face image sequences. In: Proceedings of the IEEE international conference on systems, man and cybernetics, pp 3358–3363
Platt JC, Cristianini N, Taylor JS (1999) Large margin dags for multiclass classification. Advances in Neural Information Processing Systems (NIPS), pp 547–553
Quadrianto N, Lampert C (2011) Learning multi-view neighborhood preserving projections. In: International Conference on International Conference on Machine Learning, pp 425–432
Rudovic O, Pavlovic V, Pantic M (2012) Multi-output laplacian dynamic ordinal regression for facial expression recognition and intensity estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2634–2641
Saeed A, Al-Hamadi A, Niese R, Elzobi M (2012) Effective geometric features for human emotion recognition. In: Proceedings of the IEEE international conference on signal processing (ICSP), pp 623–627
Saeed A, Al-Hamadi A, Niese R, Elzobi M (2014) Frame-based facial expression recognition using geometrical features. Advances in Human-Computer Interaction, vol 2014
Senechal T, Rapp V, Prevost L (2011) Facial feature tracking for emotional dynamic analysis. In: Proceedings of the international conference on advanced concepts for intelligent vision systems, pp 495–506
Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816
Sariyanidi E, Gunes H, Gokmen M, Cavallaro A (2013) Local zernike moment representations for facial affect recognition. In: Proceedings of the British machine vision conference, p 108.1108.13
Savran A, Cao H, Shah M, Nenkova A, Verma R (2012) Combining video, audio and lexical indicators of affect in spontaneous conversation via particle filtering. In: Proceedings of the ACM international conference on multimodal interaction, pp 485–492
Shojaeilangari S, Yau WY, Nandakumar K, Li J, Teoh EK (2015) Robust representation and recognition of facial emotions using extreme sparse learning. IEEE Trans Image Process 24:2140–2152
Sikka K, Wu T, Susskind J, Bartlett M (2012) Exploring bag of words architectures in the facial expression domain. In: Proceedings of the European conference on computer vision - workshops and demonstrations, pp 250–259
Sindhwani V, Niyogi P, Belkin M (2005) A co-regularization approach to semi-supervised learning with multiple views. ICML Workshop on Learning with Multiple Views
Tian YI, Kanade T, Cohn JF (2001) Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(2):97–115
Tao H, Huang TS (1998) Connected vibrations: a modal analysis approach for non-rigid motion tracking. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 735–740
Valstar MF, Pantic M (2012) Fully automatic recognition of the temporal phases of facial actions. EEE Trans Syst Man Cybern B (Cybernetics) 42(1):28–43
van der Schalk J, Hawk ST, Fischer AH, Doosje B (2011) Moving faces, looking places: Validation of the amsterdam dynamic facial expression set (ADFES). Emotion 11(4):907–920
Vapnik V (1998) Statistical Learning Theory. Wiley, New York
Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Proceedings of the international conference on computer vision and pattern recognition (CVPR), pp 511–518
Wu T, Bartlett MS, Movellan JR (2010) Facial expression recognition using Gabor motion energy filters. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition - workshops, pp 42–47
Yang P, Liu Q, Metaxas D (2008) Similarity features for facial event analysis. In: Proceedings of the European conference on computer vision, pp 685–696
Yang S, Bhanu B (2011) Facial expression recognition using emotion avatar image. In: Face and gesture, pp 866–871
Yang P (2011) Dynamic soft encoded patterns for facial event analysis. Comput Vis Image Underst 115(3):456–465
Yang X, Liu W, Tao D, Cheng J (2017) Canonical correlation analysis networks for two-view image recognition. Inf Sci 385-386:338–352
Yang S, Ramanan D (2015) Multi-scale recognition with dag-cnns. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 1215–1223
Zen G, Porzi L, Sangineto E, Ricci E, Sebe N (2016) Learning personalized models for facial expression analysis and gesture recognition. IEEE Transactions on Multimedia 18(4):775–788
Zhang Z, Lyons M, Schuster M, Akamatsu S (1998) Comparison between geometry-based and Gabor-wavelets-based facial expression recognition using multi-layer perceptron. In: Proceedings of the IEEE international conference on automatic face and gesture recognition, pp 454–459
Zhang S, Zhao X, Lei B (2012) Facial expression recognition based on local binary patterns and local fisher discriminant analysis. WSEAS Transactions on Signal Processing 8:21–31
Zhang J, Huang X, Yang L, Nie L (2016) Bridge the semantic gap between pop music acoustic feature and emotion: Build an interpretable model. Neurocomputing 208:333–341
Zhang T, Zheng W, Cui Z, Zong Y, Yan J, Yan K (2016) A deep neural network-driven feature learning method for multi-view facial expression recognition. IEEE Transactions on Multimedia 18(12):2528–2536
Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(6):915–928
Zhao G, Pietikinen M (2009) Boosted multi-resolution spatiotemporal descriptors for facial expression recognition. Pattern Recogn Lett 30(12):1117–1127
Zhu Y, la Torre FD, Cohn JF, Zhang YJ (2011) Dynamic cascades with bidirectional bootstrap** for action unit detection in spontaneous facial behavior. IEEE Trans Affect Comput 2(2):79–91
Acknowledgment
One of the authors, Debashis Sen, would like to thank Mr. Ashish Verma, a Ph.D. student at IIT Kharagpur, for his help in completing this article. The authors would also like to thank the anonymous reviewers, as their contribution has been pivotal in bringing this article to its best version.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Sen, D., Datta, S. & Balasubramanian, R. Facial emotion classification using concatenated geometric and textural features. Multimed Tools Appl 78, 10287–10323 (2019). https://doi.org/10.1007/s11042-018-6537-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-018-6537-9