SpringerLink
Log in

An enhanced crack segmentation method using implicit classification and inference rules in steel bridge

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In order to further improve the accuracy of crack segmentation of steel box girder bridges in complex backgrounds, a crack detection method based on information fusion combining implicit feature classification and masked image inference is proposed. Firstly, the crack segmentation model is employed to obtain the preliminary crack detection results from object recognition perspective. Then, the crack segmentation effect is enhanced with classification and inference rules for cracks learned from the training image set. In the novel masked inference module, the morphological correlation of cracks between different patches is automatically learned to provide the crack distribution probability. Meanwhile, the classification rules for cracks and distractors at the patch scale are captured and applied to suppress disturbances in classification module. Through iterative detection, inference and fusion of crack information at pixel scale and patch scale, the integrated segmentation network obtains the final crack identification results. Results from actual bridge images show that the crack detect effect of the enhanced crack detection integrated algorithm is significantly better than many classic single segmentation networks such as Deeplabv3Plus and Transformer. The average IoU, precision and recall of the steel bridge images in complex backgrounds reached 77.66%, 85.76% and 89.33%, respectively. The interference of objects such as handwriting, calibration rulers and welds can be effectively reduced.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Spain)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Data availability

The authors do not have permission to share data.

References

  1. Li, J., Zhang, Q., Bao, Y., et al.: An equivalent structural stress-based fatigue evaluation framework for rib-to-deck welded joints in orthotropic steel deck. Eng. Struct. 196, 109304 (2019). https://doi.org/10.1016/j.engstruct.2019.109304

    Article  Google Scholar 

  2. Ya, S., Yamada, K., Ishikawa, T.: Fatigue evaluation of rib-to-deck welded joints of orthotropic steel bridge deck. J. Bridge Eng. 16(4), 492–499 (2011). https://doi.org/10.1061/(ASCE)BE.1943-5592.0000181

    Article  Google Scholar 

  3. Yoon, H.: Enabling smart city resilience: post-disaster response and structural health monitoring. University of Illinois at Urbana-Champaign, Urbana (2016). https://doi.org/10.14359/51686286

    Book  Google Scholar 

  4. Gupta, P., Dixit, M.: Image-based crack detection approaches: a comprehensive survey. Multimed. Tools Appl. 81(28), 40181–40229 (2022). https://doi.org/10.1007/s11042-022-13152-z

    Article  Google Scholar 

  5. Peng, L., Chao, W., Shuangmiao, L., et al.: Research on crack detection method of airport runway based on twice-threshold segmentation. In: 2015 Fifth International Conference on Instrumentation and Measurement, Computer, Communication and Control (IMCCC), Qinhuangdao, China, pp. 1716–1720 (2015). https://doi.org/10.1109/IMCCC.2015.364

  6. Salman, M., Mathavan, S., Kamal, K., et al.: Pavement crack detection using the Gabor filter. In: 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013). The Hague, The Netherlands, pp. 2039–2044 (2013). https://doi.org/10.1109/ITSC.2013.6728529

  7. Wang, K.C.P., Li, Q., Gong, W.: Wavelet-based pavement distress image edge detection with a trous algorithm. Transp. Res. Rec. 2024(1), 73–81 (2007). https://doi.org/10.3141/2024-09

    Article  Google Scholar 

  8. Zhao, H., Qin, G., Wang, X.: Improvement of canny algorithm based on pavement edge detection. In: 2010 3rd International Congress on Image and Signal Processing. Yantai, China, pp. 964–967 (2010). https://doi.org/10.1109/CISP.2010.5646923

  9. Pauly, L., Hogg, D., Fuentes, R., et al.: Deeper networks for pavement crack detection. In: Proceedings of the 34th ISARC. Taipei, Taiwan, pp. 479–485 (2017). https://doi.org/10.22260/ISARC2017/0066

  10. Vaswani, A., Shazeer, N., Parmar, N.: Attention is all you need. In: NIPS, 30 (2017). https://doi.org/10.5040/9781350101272.00000005

  11. Dosovitskiy, A., Beyer, L., Kolesnikov, A.: An image is worth 16x16 words: transformers for image recognition at scale. ar**v preprint https://arxiv.org/abs/2010.11929. https://doi.org/10.1109/TIP.2022.3148867 (2020)

  12. Guo, J. M., Markoni, H.: Transformer based refinement network for accurate crack detection. In: 2021 International Conference on System Science and Engineering (ICSSE). Ho Chi Minh City, Vietnam, pp. 442–446 (2021). https://doi.org/10.1109/ICSSE52999.2021.9538477

  13. Ji, G. P., Fan, D. P., Xu, P., et al.: SAM struggles in concealed scenes-empirical study on" Segment Anything". ar**v preprint https://arxiv.org/abs/2304.06022. https://doi.org/10.2307/j.ctvh8r4kj.4 (2023)

  14. Zhang, L., Yang, F., Zhang, Y. D.: Road crack detection using deep convolutional neural network. In: 2016 IEEE International Conference on Image Processing (ICIP). Phoenix, AZ, pp. 3708–3712 (2016). https://doi.org/10.1109/ICIP.2016.7533052

  15. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017). https://doi.org/10.1145/3065386

    Article  Google Scholar 

  16. Simonyan, K., Zisserman, A: Very deep convolutional networks for large-scale image recognition. ar**v preprint ar**v: 1409.1556 (2014). https://doi.org/10.1109/SLT.2016.7846307

  17. Szegedy, C., Liu, W., Jia, Y.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA, pp. 1–9 (2015). https://doi.org/10.1109/CVPR.2015.7298594

  18. Inoue, Y., Nagayoshi, H.: Deployment conscious automatic surface crack detection. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV). Waikoloa, USA, pp. 686–694 (2019). https://doi.org/10.1109/WACV.2019.00078

  19. Fan, Z., Wu, Y., Lu, J.: Automatic pavement crack detection based on structured prediction with the convolutional neural network. ar**v preprint https://arxiv.org/abs/1802.02208. https://doi.org/10.29007/h4k6 (2018)

  20. Cheng, J., **ong, W., Chen, W.: Pixel-level crack detection using U-net. In: TENCON 2018–2018 IEEE region 10 conference. Jeju, Korea (South), pp. 0462–0466 (2018). https://doi.org/10.1109/ICSIP55141.2022.9886386

  21. Liu, Y., Yao, J., Lu, X.: DeepCrack: a deep hierarchical feature learning architecture for crack segmentation. Neurocomputing 338, 139–153 (2019). https://doi.org/10.1016/j.neucom.2019.01.036

    Article  Google Scholar 

  22. Zou, Q., Zhang, Z., Li, Q.: Deepcrack: learning hierarchical convolutional features for crack detection. IEEE Trans. Image Process. 28(3), 1498–1512 (2018). https://doi.org/10.1109/TIP.2018.2878966

    Article  MathSciNet  Google Scholar 

  23. Jenkins, M. D., Carr, T. A., Iglesias, M. I.: A deep convolutional neural network for semantic pixel-wise segmentation of road and pavement surface cracks. In: 2018 26th European Signal Processing Conference (EUSIPCO). Rome, Italy, pp. 2120–2124 (2018). https://doi.org/10.23919/EUSIPCO.2018.8553280

  24. König, J., Jenkins, M. D., Barrie, P.: A convolutional neural network for pavement surface crack segmentation using residual connections and attention gating. In: 2019 IEEE international conference on image processing (ICIP). Taipei, Taiwan, pp. 1460–1464 (2019). https://doi.org/10.1109/ICIP.2019.8803060

  25. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, pp. 3431–3440 (2015). https://doi.org/10.1109/ACCESS.2020.3045280

  26. Liu, D., Cui, Y., Tan, W., et al.: SG-Net: Spatial granularity network for one-stage video instance segmentation. CoRR, https://arxiv.org/abs/2103.10284 (2021). https://doi.org/10.48550/ar**v.2103.10284

  27. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, pp. 234–241 (2015). https://doi.org/10.1007/s12559-022-10038-y

  28. König, J., Jenkins, M. D., Barrie, P.: Segmentation of surface cracks based on a fully convolutional neural network and gated scale pooling. In: 2019 27th European Signal Processing Conference (EUSIPCO). A Coruna, Spain, pp. 1–5 (2019). https://doi.org/10.23919/EUSIPCO.2019.8902341

  29. Wu, Z., Lu, T., Zhang, Y.: Crack detecting by recursive attention U-Net[C]. In: 2020 3rd International Conference on Robotics, Control and Automation Engineering (RCAE). Chongqing, China, pp. 103–107 (2020). https://doi.org/10.1109/RCAE51546.2020.9294343

  30. Augustauskas, R., Lipnickas, A.: Improved pixel-level pavement-defect segmentation using a deep autoencoder. Sensors 20(9), 2557 (2020). https://doi.org/10.3390/s20092557

    Article  Google Scholar 

  31. Guo, J.M., Markoni, H., Lee, J.D.: BARNet: boundary aware refinement network for crack detection. IEEE Trans. Intell. Transp. Syst. 23(7), 7343–7358 (2021). https://doi.org/10.1109/TITS.2021.3069135

    Article  Google Scholar 

  32. Gao, Z., Peng, B., Li, T.: Generative adversarial networks for road crack image segmentation. In: 2019 International Joint Conference on Neural Networks (IJCNN). Budapest, Hungary, pp. 1–8 (2019). https://doi.org/10.1109/IJCNN.2019.8851910

  33. Zhang, K., Zhang, Y., Cheng, H.D.: CrackGAN: pavement crack detection using partially accurate ground truths based on generative adversarial learning. IEEE Trans. Intell. Transp. Syst. 22(2), 1306–1319 (2020). https://doi.org/10.1109/TITS.2020.2990703

    Article  MathSciNet  Google Scholar 

  34. Lau, S.L.H., Chong, E.K.P., Yang, X.: Automated pavement crack segmentation using u-net-based convolutional neural network. IEEE Access 8, 114892–114899 (2020). https://doi.org/10.1109/ACCESS.2020.3003638

    Article  Google Scholar 

  35. **ang, X., Zhang, Y., El Saddik, A.: Pavement crack detection network based on pyramid structure and attention mechanism. IET Image Process 14(8), 1580–1586 (2020). https://doi.org/10.1049/iet-ipr.2019.0973

    Article  Google Scholar 

  36. He, K., Zhang, X., Ren, S.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. Las Vegas, NV, USA, pp. 770–778 (2016). https://doi.org/10.3390/app12188972

  37. Tan, M., Pang, R., Le, Q. V.: Efficientdet: scalable and efficient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, pp. 10781–10790 (2020). https://doi.org/10.1109/CVPR42600.2020.01079

  38. Qu, Z., Chen, W., Wang, S.Y.: A crack detection algorithm for concrete pavement based on attention mechanism and multi-features fusion. IEEE Trans. Intell. Transp. Syst. 23(8), 11710–11719 (2021). https://doi.org/10.1109/TITS.2021.3106647

    Article  Google Scholar 

  39. Chen, L.C., Papandreou, G., Kokkinos, I.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell 40(4), 834–848 (2017). https://doi.org/10.1109/TPAMI.2017.2699184

    Article  Google Scholar 

  40. Kobayashi, T: Spiral-net with F1-based optimization for image-based crack detection. In: Asian Conference on Computer Vision. Springer, Cham, pp. 88–104 (2018). https://doi.org/10.4236/jmp.2014.513121

  41. Mosinska, A., Marquez-Neila, P., Koziński, M.: Beyond the pixel-wise loss for topology-aware delineation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, pp. 3136–3145 (2018). https://doi.org/10.1109/CVPR.2018.00331

  42. Li, K., Wang, B., Tian, Y.: Fast and accurate road crack detection based on adaptive cost-sensitive loss function. IEEE Trans. Cybern. 53(2), 1051–1062 (2021). https://doi.org/10.1109/TCYB.2021.3103885

    Article  Google Scholar 

  43. Lin, T. Y., Goyal, P., Girshick, R.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy, pp. 2980–2988 (2017). https://doi.org/10.1109/ICCV.2017.324

  44. Cui, Y., Yan, L., Cao, Z., et al.: TF-Blender: Temporal feature blender for video object detection. CoRR, https://arxiv.org/abs/2108.05821 (2021). https://doi.org/10.48550/ar**v.2108.05821

  45. Wang, W., Liang, J., Liu, D: Learning equivariant segmentation with instance-unique querying (2022). https://doi.org/10.48550/ar**v.2210.00911

  46. Wang, W., Han, C., Zhou, T., et al. Visual recognition with deep nearest centroids (2023). https://doi.org/10.48550/ar**v.2209.07383

  47. Liu, D., Liang, J., Geng, T., et al.: Tripartite feature enhanced pyramid network for dense prediction. IEEE Trans. Image Proces. 32, 2678–2692 (2023). https://doi.org/10.1109/TIP.2023.3272826

    Article  Google Scholar 

  48. **e, Z., Zhang, Z., Cao, Y.: On data scaling in masked image modeling. ar**v preprint ar**v: https://arxiv.org/abs/2206.04664 (2022). https://https://doi.org/10.2307/j.ctt13x06f6.10

  49. Kulathilake, K.A., Abdullah, N.A., Sabri, A.Q.M.: A review on deep learning approaches for low-dose computed tomography restoration. Complex Intell. Syst. (2021). https://doi.org/10.1007/s40747-021-00405-x

    Article  Google Scholar 

  50. Cheng, K., Tahir, R., Eric, L.K.: An analysis of generative adversarial networks and variants for image synthesis on MNIST dataset. Multimed. Tools. Appl 79, 13725–13752 (2020). https://doi.org/10.1007/s11042-019-08600-2

    Article  Google Scholar 

  51. Pan, J., Dong, J., Liu, Y.: Physics-based generative adversarial models for image restoration and beyond. IEEE Trans. Pattern Anal. Mach. Intell 43(7), 2449–2462 (2020). https://doi.org/10.1109/TPAMI.2020.2969348

    Article  Google Scholar 

  52. Ke, L., Danelljan, M., Li, X.: Mask transfiner for high-quality instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, LA, USA, pp. 4412–4421 (2022). https://doi.org/10.1109/CVPR52688.2022.00437

  53. Huang, Z., Huang, L., Gong, Y.: Mask scoring R-CNN. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Long Beach, pp. 6409–6418 (2019). https://doi.org/10.1109/ICCST50977.2020.00016

  54. Kuo, W., Angelova, A., Malik, J.: Shapemask: learning to segment novel objects by refining shape priors. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul, Korea (South), pp. 9207–9216 (2019). https://doi.org/10.1109/ICCV.2019.00930

  55. Yu, J., Lin, Z., Yang, J.: Generative image inpainting with contextual attention. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City, pp. 5505–5514 (2018). https://doi.org/10.1109/CVPR.2018.00577

  56. Dong, X., Yu, Z., Cao, W., et al.: A survey on ensemble learning. Front. Comput. Sci. 14, 241–258 (2020). https://doi.org/10.1007/s11704-019-8208-z

    Article  Google Scholar 

  57. Liu, K., Dai, Z., Zhang, R., et al.: Prediction of the sulfate resistance for recycled aggregate concrete based on ensemble learning algorithms. Constr. Build. Mater. 317, 125917 (2022). https://doi.org/10.1016/j.conbuildmat.2021.125917

    Article  Google Scholar 

  58. Wang, S., Wu, X., Zhang, Y.: A neural network ensemble method for effective crack segmentation using fully convolutional networks and multi-scale structured forests. Mach. Vis. Appl. 31(7), 1–18 (2020). https://doi.org/10.1007/s00138-020-01114-0

    Article  Google Scholar 

  59. Cha, Y.J., Choi, W., Büyüköztürk, O.: Deep learning-based crack damage detection using convolutional neural networks. Comput. Aided Civ. Infrastruct. Eng. 32(5), 361–378 (2017). https://doi.org/10.1111/mice.12263

    Article  Google Scholar 

  60. Peng, X., Zhong, X., Zhao, C.: A UAV-based machine vision method for bridge crack recognition and width quantification through hybrid feature learning. Constr. Build. Mater. 299, 123896 (2021). https://doi.org/10.1016/j.conbuildmat.2021.123896

    Article  Google Scholar 

  61. Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning. PMLR, Long Beach, California, pp. 6105–6114 (2019). https://doi.org/10.1016/0893-6080(88)90045-7

  62. Salehi, S. S. M., Erdogmus, D., Gholipour, A.: Tversky loss function for image segmentation using 3D fully convolutional deep networks. In: International Workshop on Machine Learning in Medical Imaging. Springer, Cham, pp. 379–387 (2017). https://doi.org/10.1007/978-3-319-67389-9_44

  63. Huang, G., Liu, Z., Van Der Maaten, L.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, pp. 4700–4708. https://doi.org/10.1109/CVPR.2017.243

  64. Xu, H., Su, X., Wang, Y., et al.: Automatic bridge crack detection using a convolutional neural network. Appl. Sci. 9(14), 2867 (2019). https://doi.org/10.3390/app9142867

    Article  Google Scholar 

  65. Zhang, L., Yang, F., Zhang, Y. D., et al.: Road crack detection using deep convolutional neural network. In: 2016 IEEE International Conference on Image Processing (ICIP), pp: 3708–3712 (2016). https://doi.org/10.1109/ICIP.2016.7533052

  66. Lin, T. Y., Dollár, P., Girshick, R.: Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, pp. 2117–2125 (2017). https://doi.org/10.1109/ACCESS.2021.3100369

  67. Miao, J., Wei, Y., Yang, Y.: Memory aggregation networks for efficient interactive video object segmentation. CoRR, https://arxiv.org/abs/2003.13246 (2020). https://doi.org/10.48550/ar**v.2003.13246

  68. Zhao, H., Shi, J., Qi, X.: Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, pp. 2881–2890 (2017). https://doi.org/10.1109/CVPR.2017.660

  69. Chen, L. C., Zhu, Y., Papandreou, G.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV). Munich, Germany, pp. 801–818 (2018). https://doi.org/10.1007/978-3-030-01234-2_49

  70. **e, E., Wang, W., Yu, Z.: SegFormer: simple and efficient design for semantic segmentation with transformers. NIPS 34, 12077–12090 (2021). https://doi.org/10.1109/CVPR52688.2022.00134

    Article  Google Scholar 

Download references

Acknowledgements

The research work was supported by the National Natural Science Foundation of China (No. 52268050). The authors also would like to thank the IPC-SHM 2020 organizing committee of Harbin Institute of Technology for providing bridge image data.

Author information

Authors and Affiliations

  1. School of Infrastructure Engineering, Nanchang University, Nanchang, 330031, Jiangxi, China

    Chun Zhang, Jian Yu & Ruoqing Wan

Authors
  1. Chun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jian Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ruoqing Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chun Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, C., Yu, J. & Wan, R. An enhanced crack segmentation method using implicit classification and inference rules in steel bridge. Vis Comput 40, 4001–4021 (2024). https://doi.org/10.1007/s00371-024-03409-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-024-03409-z

Keywords

Search

Navigation