SpringerLink
Log in

Visual sentiment analysis using data-augmented deep transfer learning techniques

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

The use of visual content to express emotions on social media platforms has become increasingly popular. Visual sentiment analysis can be used to understand the sentiment conveyed by the users using images. Compared to text, visual sentiment analysis is a challenging task since images are a more condensed form of data, have ambiguity and do not have explicit textual clues. Recently, a few studies used deep transfer learning techniques for visual sentiment analysis but the results reported can be significantly improved. In this research paper, we introduce a novel architecture that combines data augmentation and transfer learning. Our approach involves feature fusion of the pretrained VGG16 and MobileNetV1 models, followed by fine-tuning using an SVM classifier using augmented training data. For evaluation, we used two image datasets. To augment these datasets, we apply various techniques. The proposed model is also compared with three other transfer techniques, as well as four machine learning models (excluding SVM). VGG16+MobileNetV1-SVM has the best accuracy of 96% and recall of 99% for both datasets. Compared to other studies that also employed the same dataset, the proposed model produced the best results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Availability of data and materials

No dataset generated in this study.

References

  1. Yadav, A., Vishwakarma, D.K.: Sentiment analysis using deep learning architectures: a review. Artif. Intell. Rev. 53(6), 4335–4385 (2020)

    Article  Google Scholar 

  2. McDuff, D., El Kaliouby, R., Cohn, J.F., Picard, R.W.: Predicting ad liking and purchase intent: large-scale analysis of facial responses to ads. IEEE Trans. Affect. Comput. 6(3), 223–235 (2014)

    Article  Google Scholar 

  3. Cambria, E., Livingstone, A., Hussain, A.: The hourglass of emotions. In: Hawley, T.S., Hawley, R.G. (eds.) Cognitive Behavioural Systems, pp. 144–157. Springer, Berlin (2012)

    Chapter  Google Scholar 

  4. Ain, Q.T., et al.: Sentiment analysis using deep learning techniques: a review. Int. J. Adv. Comput. Sci. Appl. 8(6), 424 (2017)

    Google Scholar 

  5. Campos, V., Jou, B., Giro-i Nieto, X.: From pixels to sentiment: fine-tuning cnns for visual sentiment prediction. Image Vis. Comput. 65, 15–22 (2017)

    Article  Google Scholar 

  6. Sun, M., Yang, J., Wang, K., Shen, H.: Discovering affective regions in deep convolutional neural networks for visual sentiment prediction. In: 2016 IEEE International Conference on Multimedia and Expo (ICME) (2016)

  7. Yang, J., et al.: Visual sentiment prediction based on automatic discovery of affective regions. IEEE Trans. Multimedia 20(9), 2513–2525 (2018)

    Article  Google Scholar 

  8. Hassan, S.Z., et al.: Visual sentiment analysis from disaster images in social media. Sensors 22(10), 3628 (2022)

    Article  Google Scholar 

  9. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. ar**v preprint ar**v:1409.1556 (2014)

  10. Boser, B.E., Guyon, I.M., Vapnik, V.N.: A training algorithm for optimal margin classifiers. In: Proceedings of the Fifth Annual Workshop on Computational learning theory (1992)

  11. Tan, M., Le, Q.: Efficientnet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114 (2019)

  12. Howard, A.G. et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. ar**v preprint ar**v:1704.04861 (2017)

  13. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A.: Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)

  14. Chen, T., Guestrin, C:. Xgboost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794 (2016)

  15. **, C., De-Lin, L., Fen-**ang, M. :An improved id3 decision tree algorithm. In: 2009 4th International Conference on Computer Science, Education, pp. 127–130 (2009)

  16. Breiman, L.: Random forests. Mach Learn 45(1), 5–32 (2001)

    Article  Google Scholar 

  17. Golub, T.R., et al.: Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 286(5439), 531–537 (1999)

    Article  Google Scholar 

  18. Islam, J., Zhang, Y.: Visual sentiment analysis for social images using transfer learning approach. In: 2016 IEEE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Computing and Communications (SustainCom) (BDCloud-SocialCom-SustainCom) (2016)

  19. Wang, Y., Li, B.: Sentiment analysis for social media images. In: 2015 IEEE International Conference on Data Mining Workshop (ICDMW) (2015)

  20. Giancristofaro, G.T., Panangadan, A.: Predicting sentiment toward transportation in social media using visual and textual features. In: 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC) (2016)

  21. Ahsan, U., De Choudhury, M., Essa, I.: Towards using visual attributes to infer image sentiment of social events. In: 2017 International Joint Conference on Neural Networks (IJCNN) (2017)

  22. Cai, G., Lv, G.: Heterogeneous transfer with deep latent correlation for sentiment analysis. In: 2017 10th International Symposium on Computational Intelligence and Design (ISCID), vol. 2, pp. 252–256 (2017)

  23. Liu, W., Qiu, J.-L., Zheng, W.-L., Lu, B.-L.: Multimodal emotion recognition using deep canonical correlation analysis. ar**v preprint ar**v:1908.05349 (2019)

  24. Chen, S., Yang, J., Feng, J., Gu, Y.: Image sentiment analysis using supervised collective matrix factorization. In: 2017 12th IEEE Conference on Industrial Electronics and Applications (ICIEA) (2017)

  25. Poria, S., Peng, H., Hussain, A., Howard, N., Cambria, E.: Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis. Neurocomputing 261, 217–230 (2017)

    Article  Google Scholar 

  26. Yazdavar, A.H., et al.: Multimodal mental health analysis in social media. PLoS ONE 15(4), e0226248 (2020)

    Article  Google Scholar 

  27. Zhang, L., Wang, S., Liu, B.: Deep learning for sentiment analysis: a survey. Wiley Interdiscipl. Rev. Data Min. Knowl. Discov. 8(4), e1253 (2018)

    Article  Google Scholar 

  28. Fengjiao, W., Aono, M.: Visual Sentiment Prediction by Merging Hand-craft and CNN Features (2018)

  29. You, Q., **, H., Luo, J.: Visual Sentiment Analysis by Attending on Local Image Regions (2017)

  30. Sharma, R., Le Tan, N., Sadat, F.: Multimodal sentiment analysis using deep learning. In: 2018 5th International Conference on Advanced Informatics: Concept Theory and Applications (ICAICTA), pp. 1475–1478 (2018)

  31. Szegedy, C. et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)

  32. Chandrasekaran, G., Antoanela, N., Andrei, G., Monica, C., Hemanth, J.: Visual sentiment analysis using deep learning models with social media data. Appl. Sci. 12(3), 1030 (2022)

    Article  Google Scholar 

  33. Sowmyayani, S., Rani, P.: Salient object based visual sentiment analysis by combining deep features and handcrafted features. Multimedia Tools Appl. 81(6), 7941–7955 (2022)

    Article  Google Scholar 

  34. Mittal, N., Sharma, D., Joshi, M.L.: Image sentiment analysis using deep learning. In: 2018 IEEE/WIC/ACM International Conference on Web Intelligence (WI) (2018)

  35. Yang, L., Song, Q., Wang, Z., Jiang, M.: Parsing r-cnn for instance-level human analysis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019)

  36. Wang, C., Yang, J., **e, L., Yuan, J.: Kervolutional neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019)

  37. Zhang, K., Zhu, Y., Zhang, W., Zhu, Y.: Cross-modal image sentiment analysis via deep correlation of textual semantic. Knowl.-Based Syst. 216, 106803 (2021)

    Article  Google Scholar 

  38. Zhu, T., et al.: Multimodal sentiment analysis with image-text interaction network. IEEE Trans. Multimedia 25, 3375–3385 (2022)

    Article  Google Scholar 

  39. Poria, S., et al.: Context-dependent sentiment analysis in user-generated videos. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vol. 1: Long Papers) (2017)

  40. Huang, C.-C., Wu, Y.-L., Tang, C.-Y.: Human face sentiment classification using synthetic sentiment images with deep convolutional neural networks. In: 2019 International Conference on Machine Learning and Cybernetics (ICMLC) (2019)

  41. Hassan, S.Z., et al.: Visual sentiment analysis from disaster images in social media. ar**v preprint ar**v:2009.03051 (2020)

  42. Zisad, S.N., Chowdhury, E., Hossain, M.S., Islam, R.U., Andersson, K.: An integrated deep learning and belief rule-based expert system for visual sentiment analysis under uncertainty. Algorithms 14(7), 213 (2021)

    Article  Google Scholar 

  43. Wu, L., Zhang, H., Deng, S., Shi, G., Liu, X.: Discovering sentimental interaction via graph convolutional network for visual sentiment prediction. Appl. Sci. 11(4), 1404 (2021)

    Article  Google Scholar 

  44. Wu, L., Zhang, H., Shi, G., Deng, S.: Weakly supervised interaction discovery network for image sentiment analysis. Tech. Rep, EasyChair (2021)

  45. Li, Z., et al.: Visual sentiment analysis based on image caption and adjective-noun-pair description. Soft Comput. (2021). https://doi.org/10.1007/s00500-021-06530-6

    Article  Google Scholar 

  46. You, Q., Luo, J., **, H., Yang, J.: Robust image sentiment analysis using progressively trained and domain transferred deep networks (2015)

  47. Vadicamo, L., et al.: Cross-media learning for image sentiment analysis in the wild. In: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 308–317 (2017)

  48. Fawaz, A., Ali, M. B., Adan, M., Mujtaba, M., Wali, A:. A deep learning framework for efficient high-fidelity speech synthesis: Styletts. iKSP J. Comput. Sci. Eng. 1(1) (2021)

  49. Malik, Y.S., Tamoor, M., Naseer, A., Wali, A., Khan, A.: Applying an adaptive OTSU-based initialization algorithm to optimize active contour models for skin lesion segmentation. J. X-Ray Sci. Technol. 30(6), 1169–1184 (2022)

    Google Scholar 

  50. Wali, A., Ahmad, M., Naseer, A., Tamoor, M., Gilani, S.: Stynmedgan: medical images augmentation using a new GAN model for improved diagnosis of diseases. J. Intell. Fuzzy Syst. 44, 1–18 (2023)

    Google Scholar 

  51. Wali, A., Saeed, M.: m-calp-yet another way of generating handwritten data through evolution for pattern recognition. Biosystems 175, 24–29 (2019)

    Article  Google Scholar 

  52. Xu, Y., Wali, A.: Handwritten pattern recognition using birds-flocking inspired data augmentation technique. IEEE Access 11, 71426–71434 (2023)

    Article  Google Scholar 

  53. Hamza, H.M., Wali, A.: Pakistan sign language recognition: leveraging deep learning models with limited dataset. Mach. Vis. Appl. 34(5), 71 (2023)

    Article  Google Scholar 

  54. Shahzad, A., Wali, A.: Computerization of off-topic essay detection: a possibility? Educ. Inf. Technol. 27(4), 5737–5747 (2022)

    Article  Google Scholar 

  55. Alzubaidi, L., et al.: Novel transfer learning approach for medical imaging with limited labeled data. Cancers 13(7), 1590 (2021)

    Article  Google Scholar 

  56. What is color histogram. https://www.igi-global.com/dictionary/color-histogram/4519

  57. Siersdorfer, S., Minack, E., Deng, F., Hare, J.: Analyzing and predicting sentiment of images on the social web. In: Proceedings of the 18th ACM International Conference on Multimedia (2010)

  58. Borth, D., Chen, T., Ji, R., Chang, S.-F.: Sentibank: large-scale ontology and classifiers for detecting sentiment and emotions in visual content. In: Proceedings of the 21st ACM International Conference on Multimedia (2013)

  59. Yuan, J., Mcdonough, S., You, Q., Luo, J.: Sentribute: image sentiment analysis from a mid-level perspective. In: Proceedings of the Second International Workshop on Issues of Sentiment Discovery and Opinion Mining (2013)

  60. Jiang, Z., Zaheer, W., Wali, A., Gilani, S.: Visual sentiment analysis using data-augmented deep transfer learning techniques. Multimedia Tools Appl. 83, 17233–17249 (2023)

    Article  Google Scholar 

Download references

Funding

This project was funded by (1) Program of Teaching Reform in higher education: Teaching Reform and Practical Research on the core curriculum in Communication Engineering Major Based on the Concept of Emerging Engineering Education-A Case Study of the Principles of Communications Course and (2) Guangdong Provincial Department of Education Youth Talent Project: 2021KQNCX125.

Author information

Authors and Affiliations

  1. School of Engineering, Guangzhou College of Technology and Business, Guangzhou, 510850, Guangdong, China

    Haoran Hong

  2. Department of Computer Science, National University of Computer and Emerging Sciences, Lahore, Pakistan

    Waneeza Zaheer & Aamir Wali

Authors
  1. Haoran Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Waneeza Zaheer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aamir Wali
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed equally.

Corresponding author

Correspondence to Aamir Wali.

Ethics declarations

Conflict of interest

The authors of this paper affirm that they do not have any significant financial or non-financial interests to disclose that could be perceived as influencing the work presented in this manuscript. Furthermore, there are no personal relationships or affiliations that could have influenced the outcomes of this study. No funding was received to support the execution of this research. The authors assert that there are no conflicts of interest to declare that are pertinent to the content of this article.

Ethical standards

We, the authors of this research, hereby certify that the author list provided is accurate. Furthermore, we confirm that this research has not been previously published and is not currently under consideration for publication elsewhere. The Corresponding Author assumes full responsibility for the submission on behalf of all co-authors. We declare that there is no Conflict of interest associated with this research. Additionally, this study did not involve any human participants and/or animals.

Additional information

Communicated by S. Vrochidis.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hong, H., Zaheer, W. & Wali, A. Visual sentiment analysis using data-augmented deep transfer learning techniques. Multimedia Systems 30, 103 (2024). https://doi.org/10.1007/s00530-024-01308-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00530-024-01308-w

Keywords

Search

Navigation