SpringerLink
Log in

A deep convolutional network combining layerwise images and defect parameter vectors for laser powder bed fusion process anomalies classification

  • Published:
Journal of Intelligent Manufacturing Aims and scope Submit manuscript

Abstract

Defect detection is an essential way to ensure the quality of parts made by laser powder bed fusion (LPBF) and industrial cameras are one of the commonly used tools for defect monitoring. Different lighting environments affect the visibility of defects in the images, and the illumination condition becomes one of the most important factors affecting the defect detection effect of industrial cameras, but the modification of the equipment lighting environment will increase the complexity and cost of monitoring. In this study, only an off-axis CMOS camera monitoring system is used and the lighting facilities are not changed to improve the effect of defect detection under uneven lighting conditions. A dual-input convolutional neural network fusing defect parameter vectors and layerwise images is proposed for real-time online monitoring of defects in the LPBF process using a paraxial CMOS camera monitoring system. The model integrates the image and the parameter information related to defect generation, and can distinguish some defects that are not easily discerned by images alone. To a certain extent, it avoids the problem that the same defects are visually indistinguishable in images caused by uneven light distribution and reflections on metal surfaces. The results indicate that the method has better performance than the method with a single image input, with recognition accuracies above 80.00% for all defect categories. In addition, the method is more suitable for real-time online monitoring scenarios due to its low parameter number, short training time and fast prediction speed compared to classical deep learning algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (France)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Data availability

The data set used in this study is not publicly available at this time and may be obtained from the corresponding author upon reasonable request and permission of the authors.

References

  • Aminzadeh, M., & Kurfess, T. R. (2019). Online quality inspection using Bayesian classification in powder-bed additive manufacturing from high-resolution visual camera images. Journal of Intelligent Manufacturing, 30(6), 2505–2523. https://doi.org/10.1007/s10845-018-1412-0

    Article  Google Scholar 

  • Batista, G., Bazzan, A., & Monard, M.-C. (2003). Balancing Training Data for Automated Annotation of Keywords: a Case Study. In the Proc. Of Workshop on Bioinformatics (pp. 10–18).

  • Caggiano, A., Zhang, J., Alfieri, V., Caiazzo, F., Gao, R., & Teti, R. (2019). Machine learning-based image processing for on-line defect recognition in additive manufacturing. CIRP Annals. https://doi.org/10.1016/j.cirp.2019.03.021

    Article  Google Scholar 

  • Chen, R., Imani, F., Reutzel, E. W., & Yang, H. (2018). From design complexity to build quality in additive manufacturing—A sensor-based perspective. IEEE Sensors Journal, 3(1), 1–4. https://doi.org/10.1109/LSENS.2018.2880747

    Article  Google Scholar 

  • Chen, R., Rao, P., Lu, Y., Reutzel, E. W., & Yang, H. (2021). Recurrence network analysis of design-quality interactions in additive manufacturing. Additive Manufacturing, 39, 101861. https://doi.org/10.1016/j.addma.2021.101861

    Article  Google Scholar 

  • Coeck, S., Bisht, M., Plas, J., & Verbist, F. (2019). Prediction of lack of fusion porosity in selective laser melting based on melt pool monitoring data. Additive Manufacturing, 25, 347–356. https://doi.org/10.1016/j.addma.2018.11.015

    Article  Google Scholar 

  • Conner, B. P., Manogharan, G. P., Martof, A. N., Rodomsky, L. M., Rodomsky, C. M., Jordan, D. C., & Limperos, J. W. (2014). Making sense of 3-D printing: Creating a map of additive manufacturing products and services. Additive Manufacturing, 1–4, 64–76. https://doi.org/10.1016/j.addma.2014.08.005

    Article  Google Scholar 

  • Craeghs, T., Clijsters, S., Yasa, E., & Kruth, J.-P. (2011). Online quality control of selective laser melting. In 22nd Annual International Solid Freeform Fabrication Symposium—An Additive Manufacturing Conference, SFF 2011.

  • Cui, Y., Jia, M., Lin, T.-Y., Song, Y., & Belongie, S. (2019). Class-balanced loss based on effective number of samples. ar**v e-prints, ar**v:1901.05555.

  • Davis, G., Nagarajah, R., Palanisamy, S., Rashid, R. A. R., Rajagopal, P., & Balasubramaniam, K. (2019). Laser ultrasonic inspection of additive manufactured components. The International Journal of Advanced Manufacturing Technology, 102(5), 2571–2579. https://doi.org/10.1007/s00170-018-3046-y

    Article  Google Scholar 

  • Terris, T. de, Andreau, O., Peyre, P., Adamski, F., Koutiri, I., Gorny, C., & Dupuy, C. (2019). Optimization and comparison of porosity rate measurement methods of Selective Laser Melted metallic parts. Additive Manufacturing, 28, 802–813. https://doi.org/10.1016/j.addma.2019.05.035

    Article  Google Scholar 

  • Ding, X., Guo, Y., Ding, G., & Han, J. (2019). ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 1911–1920). https://doi.org/10.1109/ICCV.2019.00200

  • Dowling, L., Kennedy, J., O’Shaughnessy, S., & Trimble, D. (2019). A review of critical repeatability and reproducibility issues in powder bed fusion. Materials & Design, 186, 108346. https://doi.org/10.1016/j.matdes.2019.108346

    Article  Google Scholar 

  • Everton, S. K., Hirsch, M., Stravroulakis, P., Leach, R. K., & Clare, A. T. (2016). Review of in-situ process monitoring and in-situ metrology for metal additive manufacturing. Materials & Design, 95, 431–445. https://doi.org/10.1016/j.matdes.2016.01.099

    Article  Google Scholar 

  • Fischer, F. G., Zimmermann, M. G., Praetzsch, N., & Knaak, C. (2022). Monitoring of the powder bed quality in metal additive manufacturing using deep transfer learning. Materials & Design, 222, 111029. https://doi.org/10.1016/j.matdes.2022.111029

    Article  Google Scholar 

  • Foster, B. K., Reutzel, E. W., Nassar, A. R., Hall, B. T., Brown, S. W., & Dickman, C. J. (2015). Optical, layerwise monitoring of powder bed fusion. Solid Freeform Fabrication Symposium Proceedings, Austin, TX, 295–307. https://doi.org/10.1017/CBO9781107415324.004

  • Gobert, C., Reutzel, E. W., Petrich, J., Nassar, A. R., & Phoha, S. (2018). Application of supervised machine learning for defect detection during metallic powder bed fusion additive manufacturing using high resolution imaging. Additive Manufacturing, 21, 517–528. https://doi.org/10.1016/j.addma.2018.04.005

    Article  Google Scholar 

  • Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., et al. (2014). Generative Adversarial Networks. ar**v e-prints, ar**v:1406.2661.

  • Grasso, M., & Colosimo, B. (2017). Process defects and In-situ monitoring methods in metal powder bed fusion: A review. Measurement Science and Technology, 28, 1–25. https://doi.org/10.1088/1361-6501/aa5c4f

    Article  Google Scholar 

  • Grasso, M., Laguzza, V., Semeraro, Q., & Colosimo, B. (2016). In-process monitoring of selective laser melting: Spatial detection of defects via image data analysis. Journal of Manufacturing Science and Engineering, 139, 051001–051011. https://doi.org/10.1115/1.4034715

    Article  Google Scholar 

  • Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., et al. (2018). Recent advances in convolutional neural networks. Pattern Recognition: The Journal of the Pattern Recognition Society, 77, 354–377.

    Article  Google Scholar 

  • Guerra, M. G., Errico, V., Fusco, A., Lavecchia, F., Campanelli, S. L., & Galantucci, L. M. (2022). High resolution-optical tomography for in-process layerwise monitoring of a laser-powder bed fusion technology. Additive Manufacturing, 55, 102850. https://doi.org/10.1016/j.addma.2022.102850

    Article  Google Scholar 

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 770–778. https://doi.org/10.1109/CVPR.2016.90

    Article  Google Scholar 

  • Imani, F., Chen, R., Diewald, E. P., Reutzel, E. W., & Yang, H. (2019). Deep learning of Variant Geometry in Layerwise Imaging Profiles for Additive Manufacturing Quality Control. Journal of Manufacturing Science and Engineering, 141(11), 1. https://doi.org/10.1115/1.4044420

    Article  Google Scholar 

  • Imani, F., Gaikwad, A., Montazeri, M., Rao, P., Yang, H., & Reutzel, E. (2018). Process map** and in-process monitoring of porosity in laser powder bed fusion using layerwise optical imaging. Journal of Manufacturing Science and Engineering. American Society of Mechanical Engineers (ASME). https://doi.org/10.1115/1.4040615

    Article  Google Scholar 

  • Jo, D. T., & Japkowicz, N. (2004). Class imbalances versus small disjuncts. SIGKDD Explorations, 6, 40–49. https://doi.org/10.1145/1007730.1007737

    Article  Google Scholar 

  • Kleszczynski, S., Zur Jacobsmühlen, J., Sehrt, J., & Witt, G. (2012). Error detection in laser beam melting systems by high resolution imaging. In 23rd Annual International Solid Freeform Fabrication Symposium—An Additive Manufacturing Conference, SFF 2012 (pp. 975–987).

  • Krizhevsky, A., Sutskever, I., & Hinton, G. (2012). ImageNet Classification with Deep Convolutional Neural Networks. Advances in neural information processing systems, 25(2).

  • Li, J., Zhou, Q., Huang, X., Li, M., & Cao, L. (2023). In situ quality inspection with layer-wise visual images based on deep transfer learning during selective laser melting. Journal of Intelligent Manufacturing, 34(2), 853–867. https://doi.org/10.1007/s10845-021-01829-5

    Article  Google Scholar 

  • Lin, M., Chen, Q., & Yan, S. (2014). Network In Network. ar**v e-prints. https://doi.org/10.48550/ar**v.1312.4400

  • Mozaffar, M., Paul, A., Al-Bahrani, R., Wolff, S., Choudhary, A., Agrawal, A., et al. (2018). Data-driven prediction of the high-dimensional thermal history in directed energy deposition processes via recurrent neural networks. Manufacturing Letters, 18, 35–39. https://doi.org/10.1016/j.mfglet.2018.10.002

    Article  Google Scholar 

  • Pagani, L., Grasso, M., Scott, P. J., & Colosimo, B. M. (2020). Automated layerwise detection of geometrical distortions in laser powder bed fusion. Additive Manufacturing, 36, 101435. https://doi.org/10.1016/j.addma.2020.101435

    Article  Google Scholar 

  • Petrich, J., Snow, Z., Corbin, D., & Reutzel, E. W. (2021). Multi-modal sensor fusion with machine learning for data-driven process monitoring for additive manufacturing. Additive Manufacturing, 48, 102364. https://doi.org/10.1016/j.addma.2021.102364

    Article  Google Scholar 

  • Qin, J., Hu, F., Liu, Y., Witherell, P., Wang, C. C. L., Rosen, D. W., et al. (2022). Research and application of machine learning for additive manufacturing. Additive Manufacturing, 52, 102691. https://doi.org/10.1016/j.addma.2022.102691

    Article  Google Scholar 

  • Razvi, S. S., Feng, S., Narayanan, A., Lee, Y.-T., & Witherell, P. (2019). A Review of Machine Learning Applications in Additive Manufacturing. In Proceedings of the ASME 2019 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference (Vol. 1, pp. 18–21). Anaheim, California, USA. https://doi.org/10.1115/DETC2019-98415

  • Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (Vol. abs/1505.04597, pp. 234–241). http://arxiv.org/abs/1505.04597

  • Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018). MobileNetV2: Inverted Residuals and Linear Bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4510–4520). https://doi.org/10.1109/CVPR.2018.00474

  • Schweier, M., Heins, J. F., Haubold, M. W., & Zaeh, M. F. (2013). Spatter formation in laser welding with beam oscillation. Physics Procedia, 41, 20–30. https://doi.org/10.1016/j.phpro.2013.03.047

    Article  Google Scholar 

  • Scime, L., & Beuth, J. (2018a). A multi-scale convolutional neural network for autonomous anomaly detection and classification in a laser powder bed fusion additive manufacturing process. Additive Manufacturing, 24, 273–286. https://doi.org/10.1016/j.addma.2018.09.034

    Article  Google Scholar 

  • Scime, L., & Beuth, J. (2018b). Anomaly detection and classification in a laser powder bed additive manufacturing process using a trained computer vision algorithm. Additive Manufacturing, 19, 114–126. https://doi.org/10.1016/j.addma.2017.11.009

    Article  Google Scholar 

  • Scime, L., Siddel, D., Baird, S., & Paquit, V. (2020). Layer-wise anomaly detection and classification for powder bed additive manufacturing processes: A machine-agnostic algorithm for real-time pixel-wise semantic segmentation. Additive Manufacturing, 36, 101453. https://doi.org/10.1016/j.addma.2020.101453

    Article  Google Scholar 

  • Shao, J., Chen, L., & Wu, Y. (2021). SRWGANTV: Image Super-Resolution Through Wasserstein Generative Adversarial Networks with Total Variational Regularization. In 2021 IEEE 13th International Conference on Computer Research and Development (ICCRD) (pp. 21–26). https://doi.org/10.1109/ICCRD51685.2021.9386518

  • Shelhamer, E., Long, J., & Darrell, T. (2016). Fully Convolutional Networks for Semantic Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39, 1–1. https://doi.org/10.1109/TPAMI.2016.2572683

    Article  Google Scholar 

  • Shi, B., & Chen, Z. (2021). A layer-wise multi-defect detection system for powder bed monitoring: Lighting strategy for imaging, adaptive segmentation and classification. Materials and Design. https://doi.org/10.1016/j.matdes.2021.110035

    Article  Google Scholar 

  • Simonyan, K., & Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. ar**v 1409.1556. https://doi.org/10.48550/arxiv.1409.1556

  • Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S. E., Anguelov, D., et al. (2014). Going Deeper with Convolutions. CoRR, abs/1409.4842. http://arxiv.org/abs/1409.4842

  • Tan, M., & Le, Q. V. (2021). EfficientNetV2: Smaller Models and Faster Training. CoRR, abs/2104.00298. https://arxiv.org/abs/2104.00298

  • Tian, C., Xu, Y., Zuo, W., Lin, C.-W., & Zhang, D. (2021). Asymmetric CNN for image super-resolution. CoRR, abs/2103.13634. https://arxiv.org/abs/2103.13634

  • Wang, D., Wu, S., Fu, F., Mai, S., Yang, Y., Liu, Y., & Song, C. (2017). Mechanisms and characteristics of spatter generation in SLM processing and its effect on the properties. Materials & Design, 117, 121–130. https://doi.org/10.1016/j.matdes.2016.12.060

    Article  Google Scholar 

  • Zhang, Y., & Yan, W. (2023). Applications of machine learning in metal powder-bed fusion in-process monitoring and control: Status and challenges. Journal of Intelligent Manufacturing, 34(6), 2557–2580. https://doi.org/10.1007/s10845-022-01972-7

    Article  Google Scholar 

  • Zhou, B., Cui, Q., Wei, X.-S., & Chen, Z.-M. (2020). BBN: Bilateral-Branch Network With Cumulative Learning for Long-Tailed Visual Recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 9716–9725). https://doi.org/10.1109/CVPR42600.2020.00974

  • Zhu, B., & Dong, X. (2017). Studies on distortions of metal parts in selective laser melting (Master Degree). Zhejiang University of Technology.

  • Zhu, X., Zhou, H., Wang, T., Hong, F., Ma, Y., Li, W., et al. (2021). Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 9934–9943). https://doi.org/10.1109/CVPR46437.2021.00981

  • Zur Jacobsmühlen, J., Kleszczynski, S., Schneider, D., & Witt, G. (2013). High resolution imaging for inspection of Laser Beam Melting systems. In Conference Record - IEEE Instrumentation and Measurement Technology Conference (pp. 707–712). https://doi.org/10.1109/I2MTC.2013.6555507

  • Zur Jacobsmühlen, J., Kleszczynski, S., Witt, G., & Merhof, D. (2015). Detection of Elevated Regions in Surface Images from Laser Beam Melting Processes. In 41st Annual Conference of the IEEE Industrial Electronics Society (pp. 1270–1275). https://doi.org/10.1109/IECON.2015.7392275

Download references

Funding

The authors would like to acknowledge the supports from the grants of Zhong Yang Gao **ao (No. 2022ZYGXZR060), Natural Science Foundation of Guandong Province (No. 2022A1515011563), Basic and Applied Basic Research Programs of Guanzhou City (No. 202102020680).

Author information

Authors and Affiliations

  1. Shien-Ming Wu School of Intelligent Engineering, South China University of Technology, **ngye Avenue, Guangzhou, 511442, Guangdong, China

    Zimeng Jiang, Aoming Zhang, Zhangdong Chen, Chenguang Ma, Zhenghui Yuan, Yifan Deng & Yingjie Zhang

Authors
  1. Zimeng Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aoming Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhangdong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chenguang Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhenghui Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yifan Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yingjie Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yingjie Zhang.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, Z., Zhang, A., Chen, Z. et al. A deep convolutional network combining layerwise images and defect parameter vectors for laser powder bed fusion process anomalies classification. J Intell Manuf 35, 2929–2959 (2024). https://doi.org/10.1007/s10845-023-02183-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10845-023-02183-4

Keywords

Search

Navigation