SpringerLink
Log in

Weakly supervised pavement crack semantic segmentation based on multi-scale object localization and incremental annotation refinement

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Automatic and accurate pavement crack detection is essential for cost-effective road maintenance. Deep convolutional neural networks (DCNNs) are widely used in recent methods for pavement crack segmentation. Although DCNNs can segment pavement cracks with great accuracy, the requirement for huge pixel-level labels is demanding. In this article, we propose a novel weakly supervised framework for pavement crack segmentation based on multi-scale object localization and incremental annotation refinement. A trained pavement crack classification network is used to produce initial annotations using multi-scale class activation map** strategy. Then, a new segmentation network (U2-Net) with triplet attention (TA) module and multiple loss functions is trained using initial annotations. The TA module is developed to emphasize important features and ignore unimportant features, whereas multiple loss functions are employed to assist crack segmentation for a clean and full mask. Moreover, incremental annotation refinement (IAR) is proposed for iteratively optimizing the segmentation network and refining segmentation masks. Comparative experiments on DeepCrack and Crack500 datasets demonstrate that the proposed framework bridges the performance gap between weakly and fully supervised pavement crack segmentation methods, outperforms existing weakly supervised pavement crack segmentation methods, and achieves state-of-the-art performance while reducing human labeling efforts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Canada)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Algorithm 2
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Zhong Q u, Cao C, Liu L, Zhou Dong-Yang (2021) A deeply supervised convolutional neural network for pavement crack detection with multiscale feature fusion. IEEE Trans Neural Netw Learning Syst:1–10

  2. Protopapadakis E, Voulodimos A, Doulamis A, Doulamis N, Stathaki T (2019) Automatic crack detection for tunnel inspection using deep learning and heuristic image post-processing. Appl Intell 49 (7):2793–2806

    Article  Google Scholar 

  3. Liu C, Zhu C, **a X, Zhao J, Haihui Long. (2022) Ffedn: feature fusion encoder decoder network for crack detection

  4. Dai Z, Yi J, Zhang Y, Bo Z, He L (2020) Fast and accurate cable detection using cnn. Appl Intell 50(12):4688–4707

    Article  Google Scholar 

  5. Daipeng Y, Peng B, Al-Huda Z, Malik A, Zhai D (2022) An overview of edge and object contour detection. Neurocomputing

  6. **a H, Ma M, Li H, Song S (2022) Mc-net: multi-scale context-attention network for medical ct image segmentation. Appl Intell 52(2):1508–1519

    Article  Google Scholar 

  7. Zhang J, Liu Y, Guo C, Zhan J (2022) Optimized segmentation with image inpainting for semantic map** in dynamic scenes. Appl Intell:1–16

  8. Liu M, Yan X, Wang C, Wang K (2021) Segmentation mask-guided person image generation. Appl Intell 51(2):1161–1176

    Article  Google Scholar 

  9. Ma M, **a H, Tan Y, Li H, Song S (2022) Ht-net: hierarchical context-attention transformer network for medical ct image segmentation. Appl Intell:1–14

  10. Li J, Mei X, Prokhorov D, Tao D (2017) Deep neural network for structural prediction and lane detection in traffic scene. IEEE Transa Neural Netw Learning Syst 28(3):690–703

    Article  Google Scholar 

  11. Zhong Q, Chen W, Wang S-Y, Yi T-M, Liu L (2021) A crack detection algorithm for concrete pavement based on attention mechanism and multi-features fusion. IEEE Trans Intell Transp Syst:1–10

  12. Guo J-M, Markoni H, Lee J-D (2021) Barnet: boundary aware refinement network for crack detection. IEEE Trans Intell Transp Syst:1–16

  13. Cheng JCP, Wang M (2018) Automated detection of sewer pipe defects in closed-circuit television images using deep learning techniques. Autom Constr 95:155–171

    Article  Google Scholar 

  14. Yang X u, Wei S, Bao Y, Li H (2019) Automatic seismic damage identification of reinforced concrete columns from images by a region-based deep convolutional neural network. Struct Control Health Monit 26(3):e2313

    Article  Google Scholar 

  15. Tang W, Huang S, Zhao Q, Li R, Huangfu L (2021) An iteratively optimized patch label inference network for automatic pavement distress detection. IEEE Trans Intell Transp Syst:1–10

  16. Gopalakrishnan K, Khaitan SK, Choudhary A, Agrawal A (2017) Deep convolutional neural networks with transfer learning for computer vision-based data-driven pavement distress detection. Construct Build Mater 157:322–330

    Article  Google Scholar 

  17. Shi Y, Cui L, Qi Z, Meng F, Chen Z (2016) Automatic road crack detection using random structured forests. IEEE Trans Intell Transp Syst 17(12):3434–3445

    Article  Google Scholar 

  18. Yang F, Zhang L, Sijia Y u, Prokhorov D, Mei X, Ling H (2020) Feature pyramid and hierarchical boosting network for pavement crack detection. IEEE Trans Intell Transp Syst 21(4):1525–1535

    Article  Google Scholar 

  19. Li H, Song D, Liu Y u, Li B (2019) Automatic pavement crack detection by multi-scale image fusion. IEEE Trans Intell Transp Syst 20(6):2025–2036

    Article  Google Scholar 

  20. Bo P, Al-Huda Z, **e Z, ** W (2020) Multi-scale region composition of hierarchical image segmentation. Multimed Tools Appl:1–23

  21. Dai J, He K, Sun J (2015) Boxsup: exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In: Proceedings of the IEEE international conference on computer vision, pp 1635–1643

  22. Di L, Dai J, Jia J, He K, Jian Sun. (2016) Scribblesup: scribble-supervised convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3159–3167

  23. Al-Huda Z, Zhai D, Yang Y, Algburi RNA (2021) Optimal scale of hierarchical image segmentation with scribbles guidance for weakly supervised semantic segmentation. Int J Pattern Recognit Artif Intell 35 (10):2154026

    Article  Google Scholar 

  24. Kolesnikov A, Lampert CH (2016) Seed, expand and constrain: three principles for weakly-supervised image segmentation. In: European conference on computer vision. Springer, pp 695–711

  25. Al-Huda Z, Bo P, Yang Y, Algburi RNA (2020) Object scale selection of hierarchical image segmentation with deep seeds. IET Image Process, (8)

  26. Huang Z, Wang X, Wang J, Liu W, Wang J (2018) Weakly-supervised semantic segmentation network with deep seeded region growing. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7014–7023

  27. Al-Huda Z, Bo P, Yang Y, Muqeet A (2019) Object scale selection of hierarchical image segmentation using reliable regions. In: 2019 IEEE 14th international conference on intelligent systems and knowledge engineering (ISKE). IEEE, pp 1081–1088

  28. Al-Huda Z, Bo P, Yang Y, Algburi RNA, Ahmad M, Khurshid F, Moghalles K (2021) Weakly supervised semantic segmentation by iteratively refining optimal segmentation with deep cues guidance. Neural Comput Applic:1–26

  29. Dong Z, Wang J, Bo C, Wang D, Wang X (2020) Patch-based weakly supervised semantic segmentation network for crack detection. Construct Build Mater 258:120291

    Article  Google Scholar 

  30. Qin X, Zhang Z, Huang C, Dehghan M, Zaiane OR, Jagersand M (2020) U2-net: going deeper with nested u-structure for salient object detection. Pattern Recogn 106:107404

    Article  Google Scholar 

  31. Misra D, Nalamada T, Arasanipalai AU, Hou Q (2021) Rotate to attend: convolutional triplet attention module. In: 2021 IEEE winter conference on applications of computer vision (WACV), pp 3138–3147

  32. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  33. Deepcrack (2019) A deep hierarchical feature learning architecture for crack segmentation. Neurocomputing 338:139–153

    Article  Google Scholar 

  34. Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2018) Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans Pattern Anal Mach Intell 40(4):834–848

    Article  Google Scholar 

  35. Liu Z, Cao Y, Wang Y, Wang W (2019) Computer vision-based concrete crack detection using u-net fully convolutional networks. Autom Constr 104:129–139

    Article  Google Scholar 

  36. Wang M, Cheng JCP (2020) A unified convolutional neural network integrated with conditional random field for pipe defect segmentation. Comput-Aided Civil Infrastruc Eng 35(2):162–177

    Article  Google Scholar 

  37. Li D, Cong A, Guo S (2019) Sewer damage detection from imbalanced cctv inspection data using deep convolutional neural networks with hierarchical classification. Autom Constr 101:199–208

    Article  Google Scholar 

  38. Chen Liang-Chieh, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV), pp 801–818

  39. Oliveira H, Correia PL (2009) Automatic road crack segmentation using entropy and image dynamic thresholding. In: 2009 17th European signal processing conference. IEEE, pp 622–626

  40. Inoue Y, Nagayoshi H (2021) Crack detection as a weakly-supervised problem: towards achieving less annotation-intensive crack detectors. In: 2020 25th international conference on pattern recognition (ICPR). IEEE, pp 65–72

  41. Griffiths D, Boehm J (2018) Rapid object detection systems, utilising deep learning and unmanned aerial systems (uas) for civil engineering applications. Int Archives Photogrammetry, Remote Sensing Spatial Inf Sci-ISPRS Archives 42:391–398. International society for photogrammetry and remote sensing (ISPRS)

    Article  Google Scholar 

  42. Pizer SM, Johnston RE, Ericksen JP, Yankaskas BC, Muller KE (1990) Contrast-limited adaptive histogram equalization: speed and effectiveness. In: 1990 proceedings of the first conference on visualization in biomedical computing, pp 337–345

  43. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2921–2929

  44. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  45. Bo W, Yuan C, Li B, Ding X, Li Z, Ying W, Weiming H (2021) Multi-scale low-discriminative feature reactivation for weakly supervised object localization. IEEE Trans Image Process 30:6050–6065

    Article  Google Scholar 

  46. Jie H, Li S, Albanie S, Sun G, Enhua W (2020) Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell 42(8):2011–2023

    Article  Google Scholar 

  47. Pereira S, Pinto A, Amorim J, Ribeiro A, Alves V, Silva CA (2019) Adaptive feature recombination and recalibration for semantic segmentation with fully convolutional networks. IEEE Trans Med Imaging 38(12):2914–2925

    Article  Google Scholar 

  48. Wang Z, Simoncelli EP, Bovik AC (2003) Multiscale structural similarity for image quality assessment. Thrity-Seventh Asilomar Conf Signals Syst Comput, 2003 2:1398–1402. https://doi.org/10.1109/ACSSC.2003.1292216

  49. Aggarwal G, Jain S (2019) Road crack detection and segmentation for autonomous driving. In: 2019 international conference on communication and electronics systems (ICCES), pp 198–202

  50. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells WM, Frangi AF (eds) Medical image computing and computer-assisted intervention – MICCAI 2015. Springer International Publishing, pp 234–241, Cham

  51. Song W, Jia G, Jia D, Zhu H (2019) Automatic pavement crack detection and classification using multiscale feature attention network. IEEE Access 7:171001–171012

    Article  Google Scholar 

  52. Song W, Jia G, Zhu H, Di J, Gao L (2020) Automated pavement crack damage detection using deep multiscale convolutional features. J Adv Transp:2020

  53. Zou Q, Zhang Z, Li Q, Qi X, Wang Q, Wang S (2019) Deepcrack: learning hierarchical convolutional features for crack detection. IEEE Trans Image Process 28(3):1498–1512

    Article  MathSciNet  Google Scholar 

  54. Kolesnikov A, Lampert CH (2016) Seed, expand and constrain: three principles for weakly-supervised image segmentation. In: Leibe B, Matas J, Sebe N, Welling M (eds) Computer vision – ECCV 2016. Springer International Publishing

  55. Huang Z, Wang X, Wang J, Liu W, Wang J (2018) Weakly-supervised semantic segmentation network with deep seeded region growing. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 7014–7023

  56. Ahn J, Kwak S (2018) Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 4981–4990

Download references

Acknowledgements

This work was supported by the Natural Science Foundation of Sichuan, China (No. 2022NSFSC0502), the National Science Foundation of China (No. 61772435, 42075142) and Fundamental Research Funds for the Central Universities (No. 2682021ZTPY069).

Author information

Authors and Affiliations

  1. School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, 610031, Sichuan, China

    Zaid Al-Huda, Bo Peng & Tianrui Li

  2. Manufacturing Industry Chains Collaboration and Information Support Technology Key Laboratory of Sichuan Province, Chengdu, 610031, Sichuan, China

    Zaid Al-Huda & Bo Peng

  3. School of Mechanical Engineering, Southwest Jiaotong University, Chengdu, 610031, Sichuan, China

    Riyadh Nazar Ali Algburi

  4. College of Mathematics and Statistics, Shenzhen University, Shenzhen, China

    Saghir Alfasly

Authors
  1. Zaid Al-Huda
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bo Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Riyadh Nazar Ali Algburi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Saghir Alfasly
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tianrui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bo Peng.

Ethics declarations

Conflict of Interests

No conflict of interest exits in this manuscript

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Al-Huda, Z., Peng, B., Algburi, R.N.A. et al. Weakly supervised pavement crack semantic segmentation based on multi-scale object localization and incremental annotation refinement. Appl Intell 53, 14527–14546 (2023). https://doi.org/10.1007/s10489-022-04212-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04212-w

Keywords

Search

Navigation