SpringerLink
Log in

An intensity-based deep approach to mitigate step-imbalance problem under extreme paucity of images from rare classes

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this manuscript, a deep learning-based solution to the step-imbalance problem has been investigated for multi-class image annotation task where the number of training images from some of the rare classes is extremely low. Step-imbalance is a complex sub-problem of the popular class-imbalance problems where there is a steep (stair-like) disparity amongst the sample frequencies for the majority, medium and minority classes. Contrasting to the classical solutions to class- imbalance and long-tailed distribution problems, here there is a huge gap in the sample frequencies between majority and minority classes thus forming a staircase function in the overall class frequency distribution. Moreover, the pro- posed methodology focuses on a robust solution by operating under an extreme scarcity of labeled images from the minority classes (for example, below 20 images per class). Due to this, the existing neural solutions based on cost-sensitisation or generative oversampling are ineffective as they rely on the availability of sufficient minority examples in mitigating the effect of a relative and skewed ‘class-imbalance ratio’ measure. This situation is prevalent in the real-life appli- cations of computer vision, remote sensing and allied domains having severe scarcity of minority examples along with a wide gap between the major-minor class frequencies. To work under a scarce environment, an intensity-based split- ting technique has been explored to automatically extract synthetic samples for oversampling the images from the minority classes devoid of any training. In par- allel, a siamese network-based undersampling technique has been investigated for selective fusion of non-contrasting images from the majority classes. In overall, a 6–19% improvement over the existing approaches in terms of precision, recall and F1-score has been observed for the proposed technique while experimenting with CIFAR-10, Natural Images and ASCD datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Brazil)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Algorithm 1
Algorithm 2
Fig. 7
Algorithm 3
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Data Availability

The datasets are publicly available.

References

  1. Wang M, Hua X (2011) Active learning in multimedia annotation and retrieval: A survey. ACM Trans Intell Syst Technol (TIST) 2(2):1–21

    Article  Google Scholar 

  2. Mottaghi, R, Chen X, Liu X, Cho NG, Lee SW, Fidler S, Urtasun R, Yuille A (2014) The role of context for object detection and semantic segmentation in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 891–898

  3. Chen H, Wang Y, Guo T, Xu C, Deng Y, Liu Z, Ma S, Xu C, Gao W (2021) Pre-trained image processing transformer. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12299–12310

  4. Krizhevsky A, Sutskever I, Hinton G (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

    Article  Google Scholar 

  5. Shao L, Zhu F, Li X (2014) Transfer learning for visual categorization: A survey. IEEE Trans Neural Networks Learn Syst 26(5):1019–1034

    Article  MathSciNet  Google Scholar 

  6. Guo Y, Liu Y, Georgiou T, Lew M (2018) A review of semantic segmentation using deep neural networks. Intl J Multimed Inform Retriev 7:87–93

    Article  Google Scholar 

  7. Kotsiantis S, Sotiris B, Zaharakis I, Pintelas P (2007) Supervised machine learning: A review of classification techniques. Emerg Artif Intell Appl Comp Eng 160(1):3–24

    Google Scholar 

  8. Das S, Datta S, Chaudhuri B (2018) Handling data irregularities in classification: Foundations, trends, and future challenges. Pattern Recogn 81:674–693

    Article  Google Scholar 

  9. Johnson J, Khoshgoftaar T (2019) Survey on deep learning with class imbalance. J Big Data 6(1):1–54

    Article  Google Scholar 

  10. Wang L, Zhang L, Qi X, Yi Z (2022) Deep attention-based imbalanced image classification. IEEE Trans Neural Networks Learn Syst 33(8):3320–3330

    Article  MathSciNet  Google Scholar 

  11. Buda M, Maki A, Mazurowski M (2018) A systematic study of the class imbalance problem in convolutional neural networks. Neural Netw 106:249–259

    Article  Google Scholar 

  12. Yang Lu, Jiang He, Song Qing (1837) Jun Guo 2022 A survey on long-tailed visual recognition. Intl J Comp Vision 130(7):1872

    Google Scholar 

  13. Masko D, Hensman P (2015) The impact of imbalanced training data for convolutional neural networks

  14. Lee H, Park M, Kim J (2016) Plankton classification on imbalanced large scale database via convolutional neural networks with transfer learning. In: 2016 IEEE international conference on image processing (ICIP), pp 3713–3717. IEEE

  15. Chawla N, Bowyer K, Hall L, Kegelmeyer W (2002) Smote: synthetic minority over-sampling technique. J Artif Intell Res 16:321–357

    Article  Google Scholar 

  16. Roy S, Haut J, Paoletti M, Dubey S, Plaza A (2022) Generative adversarial minor- ity oversampling for spectral–spatial hyperspectral image classification. IEEE Transactions Geoscience Remote Sensing 60:1–15

    Google Scholar 

  17. Cui Y, Jia M, Lin T, Song Y, Belongie S (2019) Class-Balanced Loss Based on Effective Number of Samples. IEEE Computer Society, Los Alamitos, CA, USA

    Book  Google Scholar 

  18. Li B, Han Z, Li H, Fu H, Zhang C (2022) Trustworthy Long-Tailed Classification. IEEE Computer Society, Los Alamitos, CA, USA

    Book  Google Scholar 

  19. Wu J, Song L, Zhang Q, Yang M, Yuan J (2021) Forestdet: Large-vocabulary long-tailed object detection and instance segmentation. In: IEEE Transactions on Multimedia, pp 3693–3705. IEEE

  20. Suh MK, Seo SW (2023) Long-tailed recognition by mutual information maximization between latent features and ground-truth labels. International Conference on Machine Learning 32770–32782

  21. Liu B, Li H, Kang H, Hua G, Vasconcelos N (2021) Gistnet: a geometric structure transfer network for long-tailed recognition. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 8209–8218

  22. Buda M, Maki A, Mazurowski MA (2018) A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks (106):248–259. Elsevier

  23. Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images. Toronto, ON, Canada

  24. Roy P, Ghosh S, Bhattacharya S, Pal U (2018) Effects of degradations on deep neural network architectures ar**v preprint ar**v:1807.10108

  25. Kalita I, Roy M (2020) Deep neural network-based heterogeneous domain adaptation using ensemble decision making in land cover classification. IEEE Trans Artif Intell 1(2):167–180

    Article  Google Scholar 

  26. Wang S, Liu W, Wu J, Cao L, Meng Q, Kennedy PJ (2016) Training deep neural networks on imbalanced data sets. In: 2016 international joint conference on neural networks (IJCNN), 4368–4374. IEEE

  27. Lin T, Goyal P, Girshick R, He K (2020) Focal loss for dense object detection. IEEE Trans Pattern Anal Mach Intell 42(2):318–327

    Article  Google Scholar 

  28. Ding W, Huang DY, Chen Z, Yu X, Lin W (2017) Facial action recognition using very deep networks for highly imbalanced class distribution. In: 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp 1368–1372. IEEE

  29. Wang H, Cui Z, Chen Y, Avidan M, Abdallah A, Kronzer A (2018) Predicting hospital readmission via cost-sensitive deep learning. IEEE/ACM Trans Comput Biol Bioinf 15(6):1968–1978

    Article  Google Scholar 

  30. Khan S, Hayat M, Bennamoun M, Sohel F, Togneri R (2018) Cost-sensitive learning of deep feature representations from imbalanced data. IEEE Trans Neural Netw Learn Syst 29(8):3573–3587

    Article  Google Scholar 

  31. Pouyanfar S, Tao Y, Mohan A, Tian H, Kaseb AS, Gauen K, Dailey R, Aghajanzadeh S, Lu YH, Chen SC (2018) Dynamic sampling in convolutional neural networks for imbalanced data classification. In: 2018 IEEE conference on multimedia information processing and retrieval (MIPR), pp 112–117

  32. Huang C, Li Y, Loy CC, Tang X (2016) Learning deep representation for imbalanced classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5375–5384

  33. Dong Q, Gong S, Zhu X (2018) Imbalanced deep learning by minority class incremental rectification. IEEE Trans Pattern Anal Mach Intell 41(6):1367–1381

    Article  Google Scholar 

  34. Zhao L, Shang Z, Tan J, Zhou M, Zhang M, Gu D, Zhang T, Tang Y (2022) Siamese networks with an online reweighted example for imbalanced data learning. Pattern Recogn 132:108947

  35. Kaelbling LP, Littman ML, Moore AW (1996) Reinforcement learning: A survey. J Artif Intell Res 237–285

  36. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT press

  37. Zhang Y, Shuai L, Ren Y, Chen H (2018) Image classification with category centers in class imbalance situation. In: 2018 33rd youth academic annual conference of Chinese association of automation (YAC), pp 359–363. IEEE

  38. Ando S, Huang CY (2017) Deep over-sampling framework for classifying imbalanced data. Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2017, Skopje, Macedonia, Proceedings, Part I 10. Springer, pp 770–785

  39. Wang Y, Yao Q, Kwok J, Ni L (2019) Few-shot learning: A survey. CoRR

  40. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 16: 321–357

  41. Khairy M, Mahmoud TM, Abd-El-Hafeez T (2024) The effect of rebalancing techniques on the classification performance in cyberbullying datasets. Neural Comp Appl (36):1049–1065. Springer

  42. Eliwa EHI, El Koshiry AM, Abd El-Hafeez T, Farghaly HM (2023) Utilizing convolutional neural networks to classify monkeypox skin lesions. Sci Rep 13. Nature Publishing Group, UK, London

  43. Chicco D (2021) Siamese Neural Networks: An Overview. Springer, US, New York, NY

    Google Scholar 

  44. Parisot S (2022) Esperan¸ca, Pedro, M, McDonagh, Steven, Madarasz, Tamas, J, Yang, Li Y, and Henguo Z. Long-tail Recognition via Compositional Knowledge Transfer. IEEE Computer Society, Los Alamitos, CA, USA

    Google Scholar 

Download references

Funding

This work is supported with the project grant (with No. TDP/DR- ISHTI CPS/L2M/SL/2023/0007) from IITI DRISHTI CPS Foundation under the aegis of National Mission on Interdisciplinary Cyber-Physical System (NMICPS), Department of Science and Technology, Government of India.

Author information

Authors and Affiliations

  1. Indian Institute of Information Technology Design and Manufacturing Kurnool, Kurnool, Andhra Pradesh, 518008, India

    Vishnu Meher Vemulapalli, Shounak Chakraborty & Sathya Babu Korra

Authors
  1. Vishnu Meher Vemulapalli
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shounak Chakraborty
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sathya Babu Korra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shounak Chakraborty.

Ethics declarations

Ethical Approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 91 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vemulapalli, V.M., Chakraborty, S. & Korra, S.B. An intensity-based deep approach to mitigate step-imbalance problem under extreme paucity of images from rare classes. Multimed Tools Appl (2024). https://doi.org/10.1007/s11042-024-19303-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11042-024-19303-8

Keywords

Search

Navigation