SpringerLink
Log in

Feature Fusion Based Parallel Graph Convolutional Neural Network for Image Annotation

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Graph neural networks’ application in automatic image annotation is becoming more mature. However, there are still several problems. First, the feature data of the original image obtained by the feature extraction algorithm, such as color features and gradient features, all have the problem of slight intra-class variance and significant inter-class variance. Second, merely utilize the graph convolution neural networks to construct samples or labeled graphs, limiting multimodality’s fusion and expansion. This paper uses a parallel graph convolution network based on feature fusion for automatic image annotation. By fusing the sample features, the inherent defects of the features extracted by a single model are reduced, and the annotation performance under the condition of semi-supervised learning is improved. Experiments on three benchmark image annotation datasets show that this method is superior to the existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Chen M, Zheng A, Weinberger KQ (2013) Fast image tagging. In Proceedings of the 30th international conference on international conference on machine learning - Volume 28, ICML’13, page III-1274-III-1282. JMLR.org,

  2. Chen T, Xu M, Hui X, Wu H, Lin L (2019) Learning semantic-specific graph representation for multi-label image recognition. In 2019 IEEE/CVF International conference on computer vision (ICCV), pages 522–531,

  3. Chen Z-M, Wei X-S, Wang P, Guo Y (2019) Multi-label image recognition with graph convolutional networks. In 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 5172–5181,

  4. Dai Y, Gieseke F, Oehmcke S, Wu Y, Barnard K (2021) Attentional feature fusion. In 2021 IEEE winter conference on applications of computer vision (WACV), pages 3559–3568,

  5. Feng D, Zhongcheng W, Zhang J, Ren T (2021) Multi-scale spatial temporal graph neural network for skeleton-based action recognition. IEEE Access 9:58256–58265

    Article  Google Scholar 

  6. Gibert D, Mateu C, Planes J (2020) The rise of machine learning for detection and classification of malware: research developments, trends and challenges. J Netw Comput Appl 153:102526

    Article  Google Scholar 

  7. Gunes H, Piccardi M (2005) Affect recognition from face and body: early fusion vs. late fusion. In 2005 IEEE international conference on systems, man and cybernetics, volume 4, pages 3437–3443 Vol. 4,

  8. Huang Z, Shen X, Tian X, Li H, Huang J, Hua X-S (2020) Spatio-temporal inception graph convolutional networks for skeleton-based action recognition, page 2122–2130. Association for Computing Machinery, New York, NY, USA

  9. Kalayeh MM, Idrees H, Shah M (2014) Nmf-knn: image annotation using weighted multi-view non-negative matrix factorization. In 2014 IEEE conference on computer vision and pattern recognition, pages 184–191,

  10. Ke X, Zou J, Niu Y (2019) End-to-end automatic image annotation based on deep cnn and multi-label data augmentation. IEEE Trans Multimed 21(8):2093–2106

    Article  Google Scholar 

  11. Kipf T, Welling M (2017) Semi-supervised classification with graph convolutional networks. Ar**v:1609.02907

  12. Li C, Qin X, **aodong X, Yang D, Wei G (2020) Scalable graph convolutional networks with fast localized spectral filter for directed graphs. IEEE Access 8:105634–105644

    Article  Google Scholar 

  13. Li C, Qin X, **aodong X, Yang D, Wei G (2020) Scalable graph convolutional networks with fast localized spectral filter for directed graphs. IEEE Access 8:105634–105644

    Article  Google Scholar 

  14. Li X, Shen B, Liu B-D, Zhang Y-J (2018) Ranking-preserving low-rank factorization for image annotation with missing labels. IEEE Trans Multimed 20(5):1169–1178

    Article  Google Scholar 

  15. Li X, Shen B, Liu B-D, Zhang Y-J (2018) Ranking-preserving low-rank factorization for image annotation with missing labels. IEEE Trans Multimed 20(5):1169–1178

    Article  Google Scholar 

  16. Lin T-Y, Dollár P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In 2017 IEEE conference on computer vision and pattern recognition (CVPR), pages 936–944,

  17. Liu WB, Zou ZY, **ng WW (2017) Feature fusion methods in pattern classification. J Bei**g Univ of Posts Telecommun

  18. Liu W, Ma X, Zhou Y, Tao D, Cheng J (2019) \(p\) -laplacian regularization for scene recognition. IEEE Transact Cybernet 49(8):2927–2940

    Article  Google Scholar 

  19. Liu W, Ma X, Zhou Y, Tao D, Cheng J (2019) \(p\) -laplacian regularization for scene recognition. IEEE Transact Cybernet 49(8):2927–2940

    Article  Google Scholar 

  20. Ma Y, Hao J, Yang Y, Li H, ** J, Chen G (2019) Spectral-based graph convolutional network for directed graphs. Ar**v:1907.08990

  21. Niu Y, Zhiwu L, Wen J-R, **ang T, Chang S-F (2019) Multi-modal multi-scale deep learning for large-scale image annotation. IEEE Trans Image Process 28:1720–1731

    Article  MathSciNet  Google Scholar 

  22. Ravindraiah R, Chandra Mohan Reddy S (2018) Exudates detection in diabetic retinopathy images using possibilistic c-means clustering algorithm with induced spatial constraint. pages 455–463,

  23. Shao Q, Liu B-D (2019) Laplacian eigenmaps regularized feature map** for image annotation. In 2019 IEEE international conference on systems, man and cybernetics (SMC), pages 3901–3906,

  24. Shao Q, Wang M, Li J, Liu W, Zhang K, Liu B (2021) Semi-supervised image annotation with parallel graph convolutional networks. In 2021 40th Chinese control conference (CCC), pages 7415–7420,

  25. Snoek CGM, Worring M, Smeulders AWM (2005) Early versus late fusion in semantic video analysis. In MULTIMEDIA ’05

  26. Song H, Wang P, Yun J, Li W, Xue B, Gang W (2020) A weighted topic model learned from local semantic space for automatic image annotation. IEEE Access 8:76411–76422

    Article  Google Scholar 

  27. Tang P, Jiang M, ** for multi-label image annotation. Multimed Tools Appl 78(10):13149–13168

    Article  Google Scholar 

  28. Jiahao X, Tian H, Wang Z, Wang Y, Kang W, Chen F (2021) Joint input and output space learning for multi-label image classification. IEEE Trans Multimed 23:1696–1707

    Article  Google Scholar 

  29. Yang J, Yang J, Zhang D, feng Lu J (2003) Feature fusion: parallel strategy vs. serial strategy. Pattern Recogn 36(6):1369–1381

    Article  MATH  Google Scholar 

  30. Yang J, Wang L (2019) Feature fusion and enhancement for single shot multibox detector. In 2019 Chinese automation congress (CAC), pages 2766–2770

  31. Zeng Y, Li Y, Liu J, Ma J, Liu Z (2021) Pri-pgd: forging privacy-preserving graph towards spectral-based graph neural network. In 2021 IEEE global communications conference (GLOBECOM), pages 01–06,

  32. Zhang Y, Jia W, Cai Z, Philip SY (2020) Multi-view multi-label learning with sparse feature selection for image annotation. IEEE Trans Multimed 22(11):2844–2857

    Article  Google Scholar 

  33. Zhu P, Tan Y, Zhang L, Wang Y, Mei J, Liu H, Mengfan W (2020) Deep learning for multilabel remote sensing image annotation with dual-level semantic concepts. IEEE Trans Geosci Remote Sens 58:4047–4060

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Control Science and Engineering, China University of Petroleum (East China), Qingdao, China

    Mengke Wang, Yan Liu, Weifeng Liu & Baodi Liu

Authors
  1. Mengke Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Weifeng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Baodi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Baodi Liu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, M., Liu, Y., Liu, W. et al. Feature Fusion Based Parallel Graph Convolutional Neural Network for Image Annotation. Neural Process Lett 55, 6153–6164 (2023). https://doi.org/10.1007/s11063-022-11131-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-022-11131-x

Keywords

Search

Navigation