SpringerLink
Log in

Multi-dimensional feature extraction-based deep encoder–decoder network for automatic surface defect detection

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

The control of surface defects is of critical importance in manufacturing quality control systems. Today, automatic defects detection using imaging and deep learning algorithms has produced more successful results than manual inspections. Thanks to these automatic applications, manufacturing systems will increase the production quality, and thus financial losses will be prevented. However, since the appearance and dimensions of the defects on the surface are very variable, automatic surface defect detection is a complex problem. In this study, multi-dimensional feature extraction-based deep encoder–decoder network (MFE-DEDNet) network developed to solve such problems. An effective encoder–decoder model with lower parameters compared to the state-of-the-art methods is developed using the depthwise separable convolutions (DSC) layers in the proposed model. In addition, the 3D spectral and spatial features extract (3DFE) module of the proposed model is developed to extract deep spectral and spatial features, as well as deep semantic features. During the combination of these features, the multi-input attention gate (MIAG) module is used so that important details are not lost. As a result, the proposed MFE-DEDNet model based on these structures enabled the extraction of powerful and effective features for defect detection in surface datasets containing few images. In experimental studies, MVTec and MT datasets were used to evaluate the performance of the MFE-DEDNet. The experimental results achieved 80.01% and 56% mean intersection-over-union (mIoU) scores for the MT and MVTec datasets, respectively. In these results, it was observed that the proposed model produced higher success compared to other state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

Network architecture, data, and code generated or used during the study can be found here: “https://github.com/hys42/MFE-DEDNet”.

References

  1. Dong H, Song K, He Y et al (2020) PGA-Net: pyramid feature fusion and global context attention network for automated surface defect detection. IEEE Trans Industr Inf 16:7448–7458. https://doi.org/10.1109/TII.2019.2958826

    Article  Google Scholar 

  2. Song G, Song K, Yan Y (2020) EDRNet: encoder-decoder residual network for salient object detection of strip steel surface defects. IEEE Trans Instrum Meas 69:9709–9719. https://doi.org/10.1109/TIM.2020.3002277

    Article  Google Scholar 

  3. **an-guang F, **ao-dong W, Yu-xin C, **n W (2019) Image processing for three defects of topography images by SPM. Chemom Intell Lab Syst 185:12–17. https://doi.org/10.1016/j.chemolab.2018.12.013

    Article  Google Scholar 

  4. Zhang D, Song K, Xu J et al (2021) MCnet: multiple context information segmentation network of no-service rail surface defects. IEEE Trans Instrum Meas 70:1–9. https://doi.org/10.1109/TIM.2020.3040890

    Article  Google Scholar 

  5. Uzen H, Turkoglu M, Hanbay D (2021) Texture defect classification with multiple pooling and filter ensemble based on deep neural network. Expert Syst Appl 175:114838. https://doi.org/10.1016/j.eswa.2021.114838

    Article  Google Scholar 

  6. Yang Z, Zhu W, Ma F, et al. (2020) global context network for steel surface defect detection In: proceedings of 2020 3rd ınternational conference on unmanned systems, ICUS 2020. Institute of electrical and electronics engineers Inc., New York, pp. 985–990

  7. Hanbay K, Talu MF, Özgüven ÖF (2016) Fabric defect detection systems and methods—a systematic literature review. Optik (Stuttg) 127:11960–11973. https://doi.org/10.1016/j.ijleo.2016.09.110

    Article  Google Scholar 

  8. Djukic D, Spuzic S (2007) Statistical discriminator of surface defects on hot rolled steel. Proceedings of Image and Vision Computing, University of Waikato, Hamilton, pp. 158–163

  9. Mak KL, Peng P, Yiu KFC (2009) Fabric defect detection using morphological filters. Image Vis Comput 27:1585–1592

    Article  Google Scholar 

  10. Tsai DM, Huang TY (2003) Automated surface inspection for statistical textures. Image Vis Comput 21:307–323. https://doi.org/10.1016/S0262-8856(03)00007-6

    Article  Google Scholar 

  11. Medina R, Gayubo F, González-Rodrigo LM et al (2011) Automated visual classification of frequent defects in flat steel coils. Int J Adv Manuf Technol 57:1087–1097. https://doi.org/10.1007/s00170-011-3352-0

    Article  Google Scholar 

  12. Chu M, Liu X, Gong R, Liu L (2018) Multi-class classification method using twin support vector machines with multi-information for steel surface defects. Chemom Intell Lab Syst 176:108–118. https://doi.org/10.1016/j.chemolab.2018.03.014

    Article  Google Scholar 

  13. Tao X, Zhang D, Ma W et al (2018) Automatic metallic surface defect detection and recognition with convolutional neural networks. Appl Sci 8(1575):1575. https://doi.org/10.3390/APP8091575

    Article  Google Scholar 

  14. Lin H, Li B, Wang X et al (2019) Automated defect inspection of LED chip using deep convolutional neural network. J Intell Manuf 30:2525–2534. https://doi.org/10.1007/S10845-018-1415-X/TABLES/6

    Article  Google Scholar 

  15. Zheng X, Zheng S, Kong Y, Chen J (2021) Recent advances in surface defect inspection of industrial products using deep learning techniques. Int J Adv Manuf Technol 113:35–58. https://doi.org/10.1007/s00170-021-06592-8

    Article  Google Scholar 

  16. Luo Q, Fang X, Liu L et al (2020) Automated visual defect detection for flat steel surface: a survey. IEEE Trans Instrum Meas 69:626–644. https://doi.org/10.1109/TIM.2019.2963555

    Article  Google Scholar 

  17. Dib MA, Oliveira NJ, Marques AE et al (2019) Single and ensemble classifiers for defect prediction in sheet metal forming under variability. Neural Comput Appl 16(32):12335–12349. https://doi.org/10.1007/S00521-019-04651-6

    Article  Google Scholar 

  18. Öztürk Ş, Akdemir B (2017) Fuzzy logic-based segmentation of manufacturing defects on reflective surfaces. Neural Comput Appl 29(8):107–116. https://doi.org/10.1007/S00521-017-2862-6

    Article  Google Scholar 

  19. Turkoglu M, Hanbay D, Sengur A (2019) Multi-model LSTM-based convolutional neural networks for detection of apple diseases and pests. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-019-01591-w

    Article  Google Scholar 

  20. Türkoğlu M, Hanbay D (2019) Plant disease and pest detection using deep learning-based features. Turk J Electr Eng Comput Sci 23:1636–1651

    Article  Google Scholar 

  21. Turkoglu M, Hanbay D (2019) Plant recognition system based on deep features and color-LBP method. In: 27th Signal Processing and Communications Applications Conference, SIU 2019. Institute of Electrical and Electronics Engineers Inc, New York.

  22. Firat H, Hanbay D (2021) 4CF-Net: Hiperspektral uzaktan algılama görüntülerinin spektral uzamsal sınıflandırılması için yeni 3B evrişimli sinir ağı. Gazi Üniv Mühendis Mimar Fak Derg 37:439–454. https://doi.org/10.17341/GAZIMMFD.901291

    Article  Google Scholar 

  23. Tan M, Le QV (2019) EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. 36th International Conference on Machine Learning, ICML 2019 2019-June:10691–10700

  24. Dong G, Liao G, Liu H, Kuang G (2018) A review of the autoencoder and its variants: a comparative perspective from target recognition in synthetic-aperture radar images. IEEE Geosci Remote Sens Mag 6:44–68

    Article  Google Scholar 

  25. Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical ımage segmentation. In: Nvab N, Hornegger J, Wells WM, Frangi AF (eds) Lecture notes in computer science (including subseries lecture notes in artificial ıntelligence and lecture notes in bioinformatics). Springer, Verlag, pp 234–241

    Google Scholar 

  26. Lin T-Y, Dollár P, Girshick R, et al (2016) Feature pyramid networks for object detection. Accessed: May 05, 2021. [Online]. Available: http://arxiv.org/abs/1612.03144

  27. Qayyum A, Lalande A, Meriaudeau F (2020) Automatic segmentation of tumors and affected organs in the abdomen using a 3D hybrid model for computed tomography imaging. Comput Biol Med 127:104097. https://doi.org/10.1016/J.COMPBIOMED.2020.104097

    Article  Google Scholar 

  28. Bergmann P, Fauser M, Sattlegger D, Steger C (2019) MVTEC ad-A comprehensive real-world dataset for unsupervised anomaly detection In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition. IEEE computer society, New York, pp. 9584–9592

  29. Huang Y, Qiu C, Yuan K (2020) Surface defect saliency of magnetic tile. Vis Comput 36:85–96. https://doi.org/10.1007/s00371-018-1588-5

    Article  Google Scholar 

  30. Huang Y, Qiu C, Guo Y, et al (2018) Surface defect saliency of magnetic tile. In: 2018 IEEE 14th International conference on automation science and engineering (CASE). IEEE, pp 612–617

  31. Neogi N, Mohanta DK, Dutta PK (2017) defect detection of steel surfaces with global adaptive percentile thresholding of gradient ımage. J Inst Eng India Seri B 98:557–565. https://doi.org/10.1007/s40031-017-0296-2

    Article  Google Scholar 

  32. Wood EJ (1990) Applying fourier and associated transforms to pattern characterization in textiles. Text Res J 60:212–220. https://doi.org/10.1177/004051759006000404

    Article  Google Scholar 

  33. Chetverikov D, Hanbury A (2002) Finding defects in texture using regularity and local orientation. Pattern Recogn 35:2165–2180. https://doi.org/10.1016/S0031-3203(01)00188-1

    Article  MATH  Google Scholar 

  34. Zhao YJ, Yan YH, Song KC (2017) Vision-based automatic detection of steel surface defects in the cold rolling process: considering the influence of industrial liquids and surface textures. Int J Adv Manuf Technol 90:1665–1678. https://doi.org/10.1007/s00170-016-9489-0

    Article  Google Scholar 

  35. Yazdchi M, Yazdi M, Mahyari AG (2009) Steel surface defect detection using texture segmentation based on multifractal dimension In: Proceedings—2009 International Conference on Digital Image Processing, ICDIP 2009. IEEE, pp 346–350

  36. Wang J, Li Q, Gan J et al (2020) Surface defect detection via entity sparsity pursuit with intrinsic priors. IEEE Trans Industr Inf 16:141–150. https://doi.org/10.1109/TII.2019.2917522

    Article  Google Scholar 

  37. Qiu L, Wu X, Yu Z (2019) A high-efficiency fully convolutional networks for pixel-wise surface defect detection. IEEE Access 7:15884–15893. https://doi.org/10.1109/ACCESS.2019.2894420

    Article  Google Scholar 

  38. Farnsworth M, Tiwari D, Zhang Z et al (2022) Augmented classification for electrical coil winding defects. Int J Adv Manuf Technol 1–17:6949–6965. https://doi.org/10.1007/S00170-022-08671-W/TABLES/1

    Article  Google Scholar 

  39. Fırat H, Asker ME, Hanbay D (2022) Classification of hyperspectral remote sensing images using different dimension reduction methods with 3D/2D CNN. Remote Sens Appl Soc Environ 25:100694. https://doi.org/10.1016/J.RSASE.2022.100694

    Article  Google Scholar 

  40. Cha YJ, Choi W, Suh G et al (2018) Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Comput-Aided Civ Infrastruct Eng 33:731–747. https://doi.org/10.1111/mice.12334

    Article  Google Scholar 

  41. Yuan H, Chen H, Liu S, et al (2019) A deep convolutional neural network for detection of rail surface defect. 2019 IEEE Vehicle Power and Propulsion Conference, VPPC 2019—proceedings. https://doi.org/10.1109/VPPC46532.2019.8952236

  42. Li Y, Huang H, **e Q et al (2018) Research on a surface defect detection algorithm based on mobileNet-SSD. Appl Sci 8:1678. https://doi.org/10.3390/app8091678

    Article  Google Scholar 

  43. Rudolph M, Wandt B, Rosenhahn B (2020) Same Same But DifferNet: Semi-Supervised Defect Detection with Normalizing Flows. ar**v, Accessed: Apr. 22, 2021. [Online]. Available: http://arxiv.org/abs/2008.12577

  44. Cao J, Yang G, Yang X (2021) A pixel-level segmentation convolutional neural network based on deep feature fusion for surface defect detection. IEEE Trans Instrum Meas 70:1–12. https://doi.org/10.1109/TIM.2020.3033726

    Article  Google Scholar 

  45. Zavrtanik V, Kristan M, Skočaj D (2021) Reconstruction by inpainting for visual anomaly detection. Pattern Recogn 112:107706. https://doi.org/10.1016/j.patcog.2020.107706

    Article  Google Scholar 

  46. Zhou K, **ao Y, Yang J, et al (2020) Encoding Structure-Texture Relation with P-Net for Anomaly Detection in Retinal Images. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 12365 LNCS:360–377

  47. Tan DS, Chen Y-C, Chen TP-C, Chen W-C (2020) TrustMAE: A Noise-Resilient Defect Classification Framework using Memory-Augmented Auto-Encoders with Trust Regions. Accessed: Apr. 23, 2021. [Online]. Available: https://arxiv.org/abs/2012.14629v1

  48. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. Inf Softw Technol 51:769–784

    Google Scholar 

  49. He K, Zhang X, Ren S, Sun J (2016) Deep Residual Learning for Image Recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 770–778

  50. Weakly Supervised Learning for Industrial Optical Inspection | Heidelberg Collaboratory for Image Processing (HCI). https://hci.iwr.uni-heidelberg.de/content/weakly-supervised-learning-industrial-optical-inspection. Accessed 5 Mar 2021

  51. Howard AG, Zhu M, Chen B, et al (2017) MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. Accessed: May 06, 2020. [Online]. Available: http://arxiv.org/abs/1704.04861

  52. Saralioglu E, Gungor O (2020) Semantic segmentation of land cover from high resolution multispectral satellite images by spectral-spatial convolutional neural network. Geocarto Int. https://doi.org/10.1080/10106049.2020.1734871

    Article  Google Scholar 

  53. Singh SP, Wang L, Gupta S et al (2020) 3d deep learning on medical images: a review. Sensors (Switzerland) 20:1–24

    Article  Google Scholar 

  54. Imani M, Ghassemian H (2020) An overview on spectral and spatial information fusion for hyperspectral image classification: current trends and challenges. Inf Fusion 59:59–83. https://doi.org/10.1016/J.INFFUS.2020.01.007

    Article  Google Scholar 

  55. Ma C, Huang JB, Yang X, Yang MH (2019) Robust visual tracking via hierarchical convolutional features. IEEE Transact Pattern Anal Mach Intell 41:2709–2723

    Article  Google Scholar 

  56. Hu J, Shen L, Albanie S et al (2017) Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell 42:2011–2023

    Article  Google Scholar 

  57. Bhatt PM, Malhan RK, Rajendran P et al (2021) Image-based surface defect detection using deep learning: a review. J Comput Inf Sci Eng 21(4):040801

    Article  Google Scholar 

  58. Song L, Lin W, Yang Y-GG et al (2019) Weak micro-scratch detection based on deep convolutional neural network. IEEE Access 7:27547–27554. https://doi.org/10.1109/ACCESS.2019.2894863

    Article  Google Scholar 

  59. **g J, Wang Z, Rätsch M, Zhang H (2020) Mobile-Unet: an efficient convolutional neural network for fabric defect detection. Text Res J 004051752092860:30–42. https://doi.org/10.1177/0040517520928604

    Article  Google Scholar 

  60. Luo Q, Gao B, Woo WL, Yang Y (2019) Temporal and spatial deep learning network for infrared thermal defect detection. NDT and E Int 108:102164. https://doi.org/10.1016/j.ndteint.2019.102164

    Article  Google Scholar 

  61. Chen H, Pang Y, Hu Q, Liu K (2020) Solar cell surface defect inspection based on multispectral convolutional neural network. J Intell Manuf 31:453–468. https://doi.org/10.1007/s10845-018-1458-z

    Article  Google Scholar 

  62. Liu W, Li R, Zheng M, et al (2019) towards visually explaining variational autoencoders. Proceedings of the IEEE computer society conference on computer vision and pattern recognition 8639–8648

  63. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ (eds) Advances in neural ınformation processing systems 25 (NIPS 2012). Curran Associates Inc, New York, pp 1097–1105

    Google Scholar 

  64. Liu J, Song K, Feng M et al (2021) Semi-supervised anomaly detection with dual prototypes autoencoder for industrial surface inspection. Opt Lasers Eng 136:106324. https://doi.org/10.1016/j.optlaseng.2020.106324

    Article  Google Scholar 

  65. Qiu Y, Tang L, Li B et al (2020) Uneven illumination surface defects inspection based on saliency detection and intrinsic image decomposition. IEEE Access 8:190663–190676. https://doi.org/10.1109/ACCESS.2020.3032108

    Article  Google Scholar 

Download references

Funding

No funding was received for conducting this study.

Author information

Authors and Affiliations

  1. Computer Engineering Department, Engineering Faculty, Bingol University, Bingol, Turkey

    Huseyin Uzen

  2. Software Engineering Department, Engineering Faculty, Samsun University, Samsun, Turkey

    Muammer Turkoglu

  3. Computer Engineering Department, Engineering Faculty, Inonu University, Malatya, Turkey

    Davut Hanbay

Authors
  1. Huseyin Uzen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Muammer Turkoglu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Davut Hanbay
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

H.U. contributed to data curation, formal analysis, resources, visualization, and writing–original draft. M.T. contributed to methodology, software, visualization, and writing–review and editing. D.H. contributed to conceptualization, methodology, validation, and writing–review and editing.

Corresponding author

Correspondence to Huseyin Uzen.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Uzen, H., Turkoglu, M. & Hanbay, D. Multi-dimensional feature extraction-based deep encoder–decoder network for automatic surface defect detection. Neural Comput & Applic 35, 3263–3282 (2023). https://doi.org/10.1007/s00521-022-07885-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07885-z

Keywords

Search

Navigation