SpringerLink
Log in

An improved semantic segmentation with region proposal network for cardiac defect interpretation

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Detecting cardiac abnormalities between 14 and 28 weeks of gestation with an apical four-chamber view is a difficult undertaking. Several unfavorable factors can prevent such detection, such as the fetal heart’s relatively small size, unclear appearances in anatomical structures (e.g., shadows), and incomplete tissue boundaries. Cardiac defects without segmentation are not always straightforward to detect, so using only segmentation cannot produce defect interpretation. This paper proposes an improved semantic segmentation approach that uses a region proposal network for septal defect detection and combines two processes: contour segmentation with U-Net architecture and defect detection with Faster-RCNN architecture. The model is trained using 764 ultrasound images that include three abnormal conditions (i.e., atrial septal defect, ventricular septal defect, and atrioventricular septal defect) and normal conditions from an apical four-chamber view. The proposed model produces a satisfactory mean intersection over union, mean average precision, and dice similarity component metrics of about 75%, 87.80%, and 96.37%, respectively. Furthermore, the proposed model has also been validated on 71 unseen images in normal conditions and produces 100% sensitivity, which means that all normal conditions without septal defects can be detected effectively. The developed model has the potential to identify the fetal heart in normal and pathological settings accurately. The developed deep learning model's practical use in identifying congenital heart disorders has substantial future promise.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Thailand)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Availability of data and material

All data considered for this study are available at https://github.com/ISySRGg/U-FRcnns/Image Data.

References

  1. Garcia-Canadilla P, Sanchez-Martinez S, Crispi F, Bijnens B (2020) Machine learning in fetal cardiology: what to expect. Fetal Diagn Ther 47(5):363–372. https://doi.org/10.1159/000505021

    Article  Google Scholar 

  2. Lopez KN, Morris SA, Sexson Tejtel SK, Espaillat A, Salemi JL (2020) US mortality attributable to congenital heart disease across the lifespan from 1999 through 2017 exposes persistent racial/ethnic disparities. Circulation 142(12):1132–1147

    Article  Google Scholar 

  3. Pace ND, Oster ME, Forestieri NE, Enright D, Knight J, Meyer RE (2018) Sociodemographic factors and survival of infants with congenital heart defects. Pediatrics 142(3)

  4. Han B, Tang Y, Qu X, Deng C, Wang X, Li J (2021) Comparison of the 1-year survival rate in infants with congenital heart disease diagnosed by prenatal and postnatal ultrasound: a retrospective study. Medicine 100(4)

  5. Espinoza J (2019) Fetal MRI and prenatal diagnosis of congenital heart defects. Lancet (London, England) 393(10181):1574–1576. https://doi.org/10.1016/S0140-6736(18)32853-8

    Article  Google Scholar 

  6. Kaluva KC, Shanthi C, Thittai AK, Krishnamurthi G (2018) CardioNet: identification of fetal cardiac standard planes from 2D Ultrasound data

  7. Rawat V, Jain A, Shrimali V (2018) Automated techniques for the interpretation of fetal abnormalities: a review. Appl Bionics Biomech 2018

  8. Sundaresan V, Bridge CP, Ioannou C, Noble JA (2017) Automated characterization of the fetal heart in ultrasound images using fully convolutional neural networks. In: 2017 IEEE 14th international symposium on biomedical imaging (ISBI 2017), pp 671–674. https://doi.org/10.1109/ISBI.2017.7950609

  9. Harangi B (2018) Skin lesion classification with ensembles of deep convolutional neural networks. J Biomed Inform 86:25–32. https://doi.org/10.1016/j.jbi.2018.08.006

    Article  Google Scholar 

  10. Yu L, Guo Y, Wang Y, Yu J, Chen P (2016) Segmentation of fetal left ventricle in echocardiographic sequences based on dynamic convolutional neural networks. IEEE Trans Biomed Eng 64(8):1886–1895. https://doi.org/10.1109/TBME.2016.2628401

    Article  Google Scholar 

  11. Ha VK, Ren J-C, Xu X-Y, Zhao S, **e G, Masero V, Hussain A (2019) Deep learning based single image super-resolution: a survey. Int J Autom Comput 16(4):413–426. https://doi.org/10.1007/s11633-019-1183-x

    Article  Google Scholar 

  12. Patel N, Narasimhan E, Kennedy A (2017) Fetal cardiac US: techniques and normal anatomy correlated with adult CT and MR imaging. Radiographics 37(4):1290–1303

    Article  Google Scholar 

  13. **a Z, Wang X, Wang C, Zhang C (2018) Subpixel-based accurate and fast dynamic tumor image recognition. J Med Imaging Health Inform 8(5):925–931. https://doi.org/10.1166/jmihi.2018.2390

    Article  Google Scholar 

  14. **a Z, Wang X, Zhou W, Li R, Wang C, Zhang C (2019) Color medical image lossless watermarking using chaotic system and accurate quaternion polar harmonic transforms. Signal Process 157:108–118. https://doi.org/10.1016/j.sigpro.2018.11.011

    Article  Google Scholar 

  15. Nurmaini S, Rachmatullah MN, Sapitri AI, Darmawahyuni A, Tutuko B, Firdaus F, Partan RU, Bernolian N (2021) Deep learning-based computer-aided fetal echocardiography: application to heart standard view segmentation for congenital heart defects detection. Sensors 21(23):8007

    Article  Google Scholar 

  16. Yang R, Yu Y (2021) Artificial convolutional neural network in object detection and semantic segmentation for medical imaging analysis. Front Oncol 11:573

    Google Scholar 

  17. Rachmatullah MN, Nurmaini S, Sapitri AI, Darmawahyuni A, Tutuko B, Firdaus F (2021) Convolutional neural network for semantic segmentation of fetal echocardiography based on four-chamber view. Bull Electr Eng Inform 10(4):1987–1996

    Article  Google Scholar 

  18. Ma P, Li Q, Li J (2022) Application of artificial intelligence in cardiovascular imaging. J Healthc Eng

  19. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, pp 234–241.https://doi.org/10.1007/978-3-319-24574-4_28

  20. Hunter LE (2018) Screening views of the fetal heart. In: Fetal cardiology. Springer, Berlin, pp 9–20. https://doi.org/10.1007/978-3-319-77461-9_2

  21. Rueda S, Fathima S, Knight CL, Yaqub M, Papageorghiou AT, Rahmatullah B, Foi A, Maggioni M, Pepe A, Tohka J, Stebbing RV, McManigle JE, Ciurte A, Bresson X, Cuadra MB, Sun C, Ponomarev GV, Gelfand MS, Kazanov MD, Noble JA (2014) Evaluation and comparison of current fetal ultrasound image segmentation methods for biometric measurements: A grand challenge. IEEE Trans Med Imaging 33(4):797–813. https://doi.org/10.1109/TMI.2013.2276943

    Article  Google Scholar 

  22. Gao Y, Noble JA (2017) Detection and characterization of the fetal heartbeat in free-hand ultrasound sweeps with weakly-supervised two-streams convolutional networks. In: International conference on medical image computing and computer-assisted intervention, pp 305–313. https://doi.org/10.1007/978-3-319-66185-8_35

  23. Tajbakhsh N, Shin JY, Gurudu SR, Hurst RT, Kendall CB, Gotway MB, Liang J (2016) Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE Trans Med Imaging 35(5):1299–1312. https://doi.org/10.1109/TMI.2016.2535302

    Article  Google Scholar 

  24. **e HN, Wang N, He M, Zhang LH, Cai HM, **an JB, Lin MF, Zheng J, Yang YZ (2020) Using deep-learning algorithms to classify fetal brain ultrasound images as normal or abnormal. Ultrasound Obstet Gynecol 56(4):579–587. https://doi.org/10.1002/uog.21967

    Article  Google Scholar 

  25. Komatsu M, Sakai A, Komatsu R, Matsuoka R, Yasutomi S, Shozu K, Dozen A, Machino H, Hidaka H, Arakaki T et al (2021) Detection of cardiac structural abnormalities in fetal ultrasound videos using deep learning. Appl Sci 11(1):371. https://doi.org/10.3390/app11010371

    Article  Google Scholar 

  26. Al-Bander B, Alzahrani T, Alzahrani S, Williams BM, Zheng Y (2020) Improving fetal head contour detection by object localisation with deep learning. In: Communications in computer and information science, 1065 CCIS, pp 142–150. https://doi.org/10.1007/978-3-030-39343-4_12

  27. Liu K, Ye Z, Guo H, Cao D, Chen L, Wang FY (2021) FISS GAN: A generative adversarial network for foggy image semantic segmentation. IEEE/CAA J Automatica Sinica 8(8):1428–1439

    Article  Google Scholar 

  28. Sobhaninia Z, Rafiei S, Emami A, Karimi N, Najarian K, Samavi S, Soroushmehr SMR (2019) Fetal ultrasound image segmentation for measuring biometric parameters using multi-task deep learning. In: 2019 41st Annual international conference of the ieee engineering in medicine and biology society (EMBC), pp 6545–6548. ar**v:1909.00273

  29. Senouf O, Vedula S, Zurakhov G, Bronstein A, Zibulevsky M, Michailovich O, Adam D, Blondheim D (2018) High frame-rate cardiac ultrasound imaging with deep learning. In: International conference on medical image computing and computer-assisted intervention, pp 126–134. https://doi.org/10.1007/978-3-030-00928-1_15

  30. Wang C, Pedrycz W, Li Z, Zhou M (2020) Residual-driven fuzzy C-means clustering for image segmentation. IEEE/CAA J Automatica Sinica 8(4):876–889. https://doi.org/10.1109/JAS.2020.1003420

    Article  MathSciNet  Google Scholar 

  31. Wang C, Pedrycz W, Yang J, Zhou M, Li Z (2020) Wavelet frame-based fuzzy C-means clustering for segmenting images on graphs. IEEE Trans Cybern 50(9):3938–3949. https://doi.org/10.1109/TCYB.2019.2921779

    Article  Google Scholar 

  32. Zheng Y, Barbu A, Georgescu B, Scheuering M, Comaniciu D (2008) Four-chamber heart modeling and automatic segmentation for 3-D cardiac CT volumes using marginal space learning and steerable features. IEEE Trans Med Imaging 27(11):1668–1681. https://doi.org/10.1109/TMI.2008.2004421

    Article  Google Scholar 

  33. Nakphu N, Dewi DEO, Rizqie MQ, Supriyanto E, Kho DCC, Kadiman S, Rittipravat P et al (2014) Apical four-chamber echocardiography segmentation using Marker-controlled Watershed segmentation. In: 2014 IEEE Conference on biomedical engineering and sciences (IECBES), pp 644–647. https://doi.org/10.1109/IECBES.2014.7047583

  34. Cao Y, McNeillie P, Syeda-Mahmood T (2014) Segmentation of anatomical structures in four-chamber view echocardiogram images. In: 2014 22nd International conference on pattern recognition, pp 568–573. https://doi.org/10.1109/ICPR.2014.108.

  35. Syeda-Mahmood T, Wang Q, McNeillie P, Beymer D, Compas C (2014) Discriminating normal and abnormal left ventricular shapes in four-chamber view 2D echocardiography. In: 2014 IEEE 11th International symposium on biomedical imaging (ISBI), pp 401–404. https://doi.org/10.1109/ISBI.2014.6867893

  36. Kang HC, Kim B, Lee J, Shin J, Shin Y-G (2015) Accurate four-chamber segmentation using gradient-assisted localized active contour model. J Med Imaging Health Inform 5(1):126–137

    Article  Google Scholar 

  37. Zheng Y, Comaniciu D (2014) Marginal space learning for medical image analysis: efficient detection and segmentation of anatomical structures. In: Marginal space learning for medical image analysis: efficient detection and segmentation of anatomical structures, vol 9781493906, Issue Mdl. https://doi.org/10.1007/978-1-4939-0600-0

  38. Jafari MH, Girgis H, Liao Z, Behnami D, Abdi A, Vaseli H et al (2018) A unified framework integrating recurrent fully-convolutional networks and optical flow for segmentation of the left ventricle in echocardiography data. In: Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, Cham, pp. 29–37. https://doi.org/10.1007/978-3-030-00889-5

  39. Jafari MH, Girgis H, Abdi AH, Liao Z, Pesteie M, Rohling R, Gin K, Tsang T, Abolmaesumi P (2019) Semi-supervised learning for cardiac left ventricle segmentation using conditional deep generative models as Prior University of British Columbia, Vancouver, Canada. Vancouver General Hospital, Vancouver, Canada. In: 2019 IEEE 16th International symposium on biomedical imaging (ISBI 2019), Isbi, pp 649–652. https://doi.org/10.1109/ISBI.2019.8759292

  40. Alsharqi M, Woodward WJ, Mumith JA, Markham DC, Upton R, Leeson P (2018) Artificial intelligence and echocardiography. Echo Res Pract 5(4):R115–R125. https://doi.org/10.1530/ERP-18-0056

    Article  Google Scholar 

  41. Arafati A, Morisawa D, Avendi MR, Amini MR, Assadi RA, Jafarkhani H, Kheradvar A (2020) Generalizable fully automated multi-label segmentation of four-chamber view echocardiograms based on deep convolutional adversarial networks. J R Soc Interface 17(169):20200267. https://doi.org/10.1098/rsif.2020.0267

    Article  Google Scholar 

  42. Painchaud N, Skandarani Y, Judge T, Bernard O, Lalande A, Jodoin P-M (2020) Cardiac segmentation with strong anatomical guarantees. IEEE Trans Med Imaging. https://doi.org/10.1109/TMI.2020.3003240

    Article  Google Scholar 

  43. Leclerc S, Smistad E, Pedrosa J, Ostvik A, Cervenansky F, Espinosa F, Espeland T, Berg E, Jodoin PM, Grenier T, Lartizien C, Dhooge J, Lovstakken L, Bernard O (2019) Deep learning for segmentation using an open large-scale dataset in 2D echocardiography. IEEE Trans Med Imaging 38(9):2198–2210. https://doi.org/10.1109/TMI.2019.2900516

    Article  Google Scholar 

  44. Yang M, **ao X, Liu Z, Sun L, Guo W, Cui L, Sun D, Zhang P, Yang G (2020) Deep RetinaNet for dynamic left ventricle detection in multiview echocardiography classification. Sci Program

  45. Dong J, Liu S, Wang T (2019) ARVBNet: real-time detection of anatomical structures in fetal ultrasound cardiac four-chamber planes. In: Machine learning and medical engineering for cardiovascular health and intravascular imaging and computer assisted stenting. Springer, Berlin, pp 130–137. https://doi.org/10.1007/978-3-030-33327-0_16

  46. Zhou SK, Park JH, Georgescu B, Comaniciu D, Simopoulos C, Otsuki J (2006) Image-based multiclass boosting and echocardiographic view classification. In: 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06), vol 2, pp 1559–1565. https://doi.org/10.1109/CVPR.2006.146

  47. Park JH, Zhou SK, Simopoulos C, Otsuki J, Comaniciu D (2007) Automatic cardiac view classification of echocardiogram. In: 2007 IEEE 11th international conference on computer vision, pp 1–8. https://doi.org/10.1109/ICCV.2007.4408867

  48. Balaji GN, Subashini TS, Chidambaram N (2015) Automatic classification of cardiac views in echocardiogram using histogram and statistical features. Procedia Comput Sci 46:1569–1576. https://doi.org/10.1016/j.procs.2015.02.084

    Article  Google Scholar 

  49. Lili W, Zhongliang F, Pan T (2016) Four-chamber plane detection in cardiac ultrasound images based on improved imbalanced AdaBoost algorithm. In: 2016 IEEE International conference on cloud computing and big data analysis (ICCCBDA), pp 299–303. https://doi.org/10.1109/ICCCBDA.2016.7529574

  50. Khamis H, Zurakhov G, Azar V, Raz A, Friedman Z, Adam D (2017) Automatic apical view classification of echocardiograms using a discriminative learning dictionary. Med Image Anal 36:15–21. https://doi.org/10.1016/j.media.2016.10.007

    Article  Google Scholar 

  51. Lin Z, Le MH, Ni D, Chen S, Li S, Wang T, Lei B (2018) Quality assessment of fetal head ultrasound images based on faster R-CNN. In: Simulation, image processing, and ultrasound systems for assisted diagnosis and navigation. Springer, Berlin, pp 38–46. https://doi.org/10.1007/978-3-030-01045-4_5

  52. Zhou Z, Siddiquee M, Tajbakhsh N, Liang J (2018) UNet++: a nested U-Net architecture for medical image segmentation. In: Deep learning in medical image analysis and multimodal learning for clinical decision support: 4th international workshop, DLMIA 2018, and 8th international workshop, ML-CDS 2018, held in conjunction with MICCAI 2018, Granada, Spain, S..., vol 11045, pp 3–11. https://doi.org/10.1007/978-3-030-00889-5_1

  53. Milletari F, Navab N, Ahmadi SA (2016) V-Net: fully convolutional neural networks for volumetric medical image segmentation. In: Proceedings—2016 4th international conference on 3D vision, 3DV 2016, pp 565–571. https://doi.org/10.1109/3DV.2016.79

  54. Xu L, Liu M, Shen Z, Wang H, Liu X, Wang X, Wang S, Li T, Yu S, Hou M, Guo J, Zhang J, He Y (2020) DW-Net: a cascaded convolutional neural network for apical four-chamber view segmentation in fetal echocardiography. Comput Med Imaging Graph 80:101690. https://doi.org/10.1016/j.compmedimag.2019.101690

    Article  Google Scholar 

  55. Ravishankar H, Prabhu SM, Vaidya V, Singhal N (2016) Hybrid approach for automatic segmentation of fetal abdomen from ultrasound images using deep learning. In: 2016 IEEE 13th international symposium on biomedical imaging (ISBI), pp 779–782. https://doi.org/10.1109/ISBI.2016.7493382

  56. Yap MH, Goyal M, Osman FM, Martí R, Denton E, Juette A, Zwiggelaar R (2019) Breast ultrasound lesions recognition: end-to-end deep learning approaches. J Med Imaging (Bellingham, Wash.), 6(1):011007. https://doi.org/10.1117/1.JMI.6.1.011007

  57. Kopelowitz E, Engelhard G (2019) Lung nodules detection and segmentation using 3D mask-RCNN. Ar**v Preprint ar**v:1907.07676

  58. Anantharaman R, Velazquez M, Lee Y (2018) Utilizing mask R-CNN for detection and segmentation of oral diseases. In: 2018 IEEE international conference on bioinformatics and biomedicine (BIBM), pp 2197–2204. https://doi.org/10.1109/BIBM.2018.8621112

  59. Kowal M, Żejmo M, Skobel M, Korbicz J, Monczak R (2020) Cell Nuclei Segmentation in cytological images using convolutional neural network and seeded watershed algorithm. J Digit Imaging 33(1):231–242. https://doi.org/10.1007/s10278-019-00200-8

    Article  Google Scholar 

Download references

Acknowledgements

We thank the Intelligent System Research Group (ISysRG), Faculty of Computer Science, Universitas Sriwijaya, Indonesia.

Funding

This work was supported by the Ministry of Research and Technology, Indonesia, through Applied Research, under Grant 096/SP2H/LT/DRPM/2021 and Professional Grant 2022 from Universitas Sriwijaya, Indonesia. This work is also funded by Institute for Basic Science (IBS) under grant No. IBS-R029-C2-001.

Author information

Authors and Affiliations

  1. Intelligent System Research Group, Faculty of Computer Science, Universitas Sriwijaya, Palembang, South Sumatera, 30139, Indonesia

    Siti Nurmaini, Muhammad Naufal Rachmatullah, Annisa Darmawahyuni, Ade Iriani Sapitri, Firdaus Firdaus & Bambang Tutuko

  2. Data Science Group, Institute for Basic Science (IBS), Daejeon, 34126, Republic of Korea

    Bayu Adhi Tama

Authors
  1. Siti Nurmaini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bayu Adhi Tama
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Muhammad Naufal Rachmatullah
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Annisa Darmawahyuni
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ade Iriani Sapitri
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Firdaus Firdaus
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Bambang Tutuko
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization was contributed by Siti Nurmaini, Ade Iriani Sapitri, and Bayu Adhi Tama; Methodology was contributed by Ade Iriani Sapitri and Muhammad Naufal Rachmatullah; Formal analysis and investigation were contributed by Siti Nurmaini; Writing—original draft preparation was contributed by Siti Nurmaini; Writing—review and editing was contributed by Bayu Adhi Tama and Annisa Darmawahyuni; Funding acquisition was contributed by Siti Nurmaini; Resources were contributed by Firdaus Firdaus and Bambang Tutuko.

Corresponding author

Correspondence to Siti Nurmaini.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Consent to participate

Not applicable.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nurmaini, S., Tama, B.A., Rachmatullah, M.N. et al. An improved semantic segmentation with region proposal network for cardiac defect interpretation. Neural Comput & Applic 34, 13937–13950 (2022). https://doi.org/10.1007/s00521-022-07217-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07217-1

Keywords

Search

Navigation