SpringerLink
Log in

Subpixel keypoint localization and angle prediction for lithography marks based on deep learning

  • Published:
Journal of Intelligent Manufacturing Aims and scope Submit manuscript

Abstract

The alignment precision in lithography determines the pattern transfer quality, whereas the initial angular deviation between the wafer and mask leads to the empty-window problem. Therefore, identifying the angular deviation from a single-view alignment mark image and reducing it below the threshold of empty-window is urgent. We propose a deep learning-based method to achieve high-accuracy angle estimation under low-resolution and blurry contours. The proposed network is a multitask network with a CNN as backbone and a parameter-free SVD-based head. It can simultaneously predict subpixel coordinates of alignment marks’ keypoints and registration angle. The network takes image patches rather than the entire original image as input to let itself only focus on the unobstructed keypoint regions. A multicomponent loss function based on angular relationships is also introduced. The training result verifies that the network can learn keypoint localization indirectly through angle prediction optimization. Application experiments demonstrate that the proposed method achieves an RMSE (root mean square error) of 0.093 and R\(^2\) (R-squared) of 0.976 in angle prediction error, effectively addressing the empty-window problem. Furthermore, compared to other angle estimation methods based on keypoint registration, line detection, template matching, and manual select points, our method demonstrates higher accuracy and better stability. The code and models have been publicly available at https://github.com/YuLungLee/HRNet8-SVD.

  • Our model is a multitask network with a CNN as backbone and a parameters-free SVD-based head. This network can simultaneously predict subpixel coordinates of alignment marks’ keypoints and registration angle.

  • The network takes image patches and their corresponding indices in the original image as input, rather than the entire original image. This allows the network to focus only on the unobstructed keypoint regions.

  • To train the network without manual annotation of subpixel keypoints, a multicomponent loss function based on angular relationships is introduced.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Algorithm 1
Fig. 12
Fig. 13
Algorithm 2
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Data Availability

The code and models have been publicly available at https://github.com/YuLungLee/HRNet8-SVD.

References

  • Alharbi, R. H., Alshaye, M. B., Alkanhal, M. M., Alharbi, N. M., Alzahrani, M. A., & Alrehaili, O. A. (2020). Deep learning based algorithm for automatic scoliosis angle measurement. In 2020 3rd International Conference on Computer Applications & Information Security (ICCAIS) (pp. 1–5). IEEE.

  • Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021). Review of deep learning: Concepts, cnn architectures, challenges, applications, future directions. Journal of big Data, 8, 1–74.

    Article  Google Scholar 

  • Bay, H., Ess, A., Tuytelaars, T., & Van Gool, L. (2008). Speeded-up robust features (surf). Computer Vision and Image Understanding, 110(3), 346–359.

    Article  Google Scholar 

  • Besl, P. J., & McKay, N. D. (1992). Method for registration of 3-d shapes. In Sensor Fusion IV: Control Paradigms and Data Structures (Vol. 1611, pp. 586–606). SPIE.

  • Bottou, L. (1998). Online learning and stochastic approximations. On-linelearning in neural networks, 17(9), 142.

    Google Scholar 

  • Brunner, T. A., & Fonseca, C. A. (2001) Optimum tone for various feature types: positive versus negative. In Advances in Resist Technology and Processing XVIII (Vol. 4345, pp. 30–36). SPIE.

  • Chen, M. -F., Ho, Y. -S., & Wang, S. -M. (2010). A fast positioning method with pattern tracking for automatic wafer alignment. In 2010 3rd International Congress on Image and Signal Processing (Vol. 4, pp. 1594–1598). IEEE.

  • Cheng, B., **ao, B., Wang, J., Shi, H., Huang, T. S., & Zhang, L. (2020). Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5386–5395). IEEE.

  • Cheng, H., Cai, C., Wang, Y., Liu, Z., & Yang, M. (2020). A high precision rotating line detection method for the rotation angle measurement based on machine vision. Journal of Physics: Conference Series, 1627, 012016. IOP Publishing.

    Google Scholar 

  • Chiu, M.-C., Chiang, Y.-H., & Chiu, J.-E. (2023). Develo** an explainable hybrid deep learning model in digital transformation: an empirical study. Journal of Intelligent Manufacturing, 1–18.

  • Davydova, N., Finders, J., McNamara, J., Setten, E., Lare, C., Franke, J. -H., Frommhold, A., Capelli, R., Kersteen, G., Verch, A., et al. (2020). Fundamental understanding and experimental verification of bright versus dark field imaging. In Extreme Ultraviolet Lithography 2020 (Vol. 11517, pp. 40–57). SPIE.

  • Diao, Z.-M., Peng, G.-H., & Fu, L.-M. (2010). Fast high-precision reliable image matching algorithm based on shape. Journal of Computer Applications, 30(2), 441.

    Article  Google Scholar 

  • Duda, R. O., & Hart, P. E. (1972). Use of the hough transformation to detect lines and curves in pictures. Communications of the ACM, 15(1), 11–15.

    Article  Google Scholar 

  • Ghiasi, G., Lin, T.-Y., & Le, Q. V. (2018). Dropblock: A regularization method for convolutional networks. Advances in Neural Information Processing Systems, 31, 586–606.

    Google Scholar 

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 770–778). IEEE

  • Hinterstoisser, S., Cagniart, C., Ilic, S., Sturm, P., Navab, N., Fua, P., & Lepetit, V. (2011). Gradient response maps for real-time detection of textureless objects. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(5), 876–888.

    Article  Google Scholar 

  • Horng, M.-H., Kuok, C.-P., Fu, M.-J., Lin, C.-J., Sun, Y.-N., et al. (2019). Cobb angle measurement of spine from x-ray images using convolutional neural network. Computational and Mathematical Methods in Medicine, 2019, 18.

    Article  Google Scholar 

  • Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4700–4708). IEEE.

  • Huang, A. C., Meng, S. H., & Huang, T. J. (2023). A survey on machine and deep learning in semiconductor industry: methods, opportunities, and challenges. Cluster Computing, 26(6), 3437–3472.

    Article  Google Scholar 

  • Hussain, F., Ali, Y., Irfan, M., Ashraf, M., & Ahmed, S. (2021). A data-driven model for phase angle behaviour of asphalt concrete mixtures based on convolutional neural network. Construction and Building Materials, 269, 121235.

    Article  Google Scholar 

  • Huzjan, F., Juric, F., Vujanovic, M., & Loncaric, S. (2023). Deep learning-based cone angle estimation using spray sequence images. In Proceedings of the 2023 8th International Conference on Machine Learning Technologies (pp. 208–213). Association for Computing

  • Islam, M. K., Yeasmin, M. N., Kaushal, C., Al Amin, M., Islam, M. R., & Showrov, M. I. H. (2021). Comparative analysis of steering angle prediction for automated object using deep neural network. In 2021 9th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions)(ICRITO) (pp. 1–7). IEEE/

  • Jo, C., Hwang, D., Ko, S., Yang, M. H., Lee, M. C., Han, H.-S., & Ro, D. H. (2023). Deep learning-based landmark recognition and angle measurement of full-leg plain radiographs can be adopted to assess lower extremity alignment. Knee Surgery, Sports Traumatology, Arthroscopy, 31(4), 1388–1397.

    Article  Google Scholar 

  • Kokabu, T., Kanai, S., Kawakami, N., Uno, K., Kotani, T., Suzuki, T., Tachi, H., Abe, Y., Iwasaki, N., & Sudo, H. (2021). An algorithm for using deep learning convolutional neural networks with three dimensional depth sensor imaging in scoliosis detection. The Spine Journal, 21(6), 980–987.

    Article  Google Scholar 

  • Koutaki, G., Yata, K., Uchimura, K., Kan, M., Asai, D., & Takeba, M. (2013). Fast and high accuracy pattern matching using multi-stage refining eigen template. In The 19th Korea-Japan Joint Workshop on Frontiers of Computer Vision (pp. 58–63). IEEE

  • LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444.

    Article  Google Scholar 

  • Lin, C.-J., Hsu, H.-H., Cheng, C.-H., & Li, Y.-C. (2016). Design of an image-servo mask alignment system using dual ccds with an xxy stage. Applied Sciences, 6(2), 42.

    Article  Google Scholar 

  • Liu, Z., Lei, L., Liu, X., Wang, Q., Yan, R., Zhou, J., et al. (2016). Formula for optimal matching parameters of real-time alignment by affine transformation (in chinese). Opto-Electronic Engineering, 43(4), 61.

    Google Scholar 

  • Liu, H., Nie, H., Zhang, Z., & Li, Y.-F. (2021). Anisotropic angle distribution learning for head pose estimation and attention understanding in human-computer interaction. Neurocomputing, 433, 310–322.

    Article  Google Scholar 

  • Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60, 91–110.

    Article  Google Scholar 

  • Luo, P., Peng, Y., Li, Y., & Liu, Z. (2023). Design and implementation of alignment device for proximity lithography system. In 2023 35th Chinese Control and Decision Conference (CCDC) (pp. 4419–4424). IEEE

  • Meng, C., Lang, S., Hao, F., Li, P., & Shi, J. (2023). A coarse-to-fine angle automatic correction method for glassivation passivation parts wafer. In Fifteenth International Conference on Machine Vision (ICMV 2022) (Vol. 12701, pp. 213–220). SPIE

  • MicroTec, S. (2019). Mask/Bond Aligners Gen4 Safety/Installation Manual. SUSS MicroTec.

  • Minelli, M., Cina, A., Galbusera, F., Castagna, A., Savevski, V., & Sconfienza, L. M. (2022). Measuring the critical shoulder angle on radiographs: an accurate and repeatable deep learning model. Skeletal Radiology, 51(9), 1873–1878.

    Article  Google Scholar 

  • Moreau, W. M. (2012). Semiconductor lithography: Principles, practices, and materials. Berlin: Springer.

    Google Scholar 

  • Munger, J., & Morato, C. W. (2021). How many features is an image worth? multi-channel cnn for steering angle prediction in autonomous vehicles. In CS & IT Conference Proceedings (Vol. 11). CS & IT Conference Proceedings

  • Nti, I. K., Adekoya, A. F., Weyori, B. A., & Nyarko-Boateng, O. (2022). Applications of artificial intelligence in engineering and manufacturing: A systematic review. Journal of Intelligent Manufacturing, 33(6), 1581–1601.

    Article  Google Scholar 

  • Oelen, D., Kaiser, P., Baumann, T., Schmid, R., Bühler, C., Munkhuu, B., & Essig, S. (2022). Accuracy of trained physicians is inferior to deep learning-based algorithm for determining angles in ultrasound of the newborn hip. Ultraschall in der Medizin-European Journal of Ultrasound, 43(05), 49–55.

    Article  Google Scholar 

  • Pang, S., Chen, Z., & Yin, F. (2021). Convolutional neural network-based sub-pixel line-edged angle detection with applications in measurement. IEEE Sensors Journal, 21(7), 9314–9322.

    Article  Google Scholar 

  • Phan, P. H., Nguyen, A. Q., Quach, L. -D., & Tran, H. N. (2023). Robust autonomous driving control using auto-encoder and end-to-end deep learning under rainy conditions. In Proceedings of the 2023 8th International Conference on Intelligent Information Technology (pp. 271–278). Association for Computing

  • Rahmaniar, W., Suzuki, K., & Lin, T.-L. (2024). Auto-ca: Automated cobb angle measurement based on vertebrae detection for assessment of spinal curvature deformity. IEEE Transactions on Biomedical Engineering, 71(2), 640–649.

    Article  Google Scholar 

  • Ramos, J. A., & Verriest, E. I. (1997). Total least squares fitting of two point sets in md. In Proceedings of the 36th IEEE Conference on Decision and Control (Vol. 5, pp. 5048–5053). IEEE

  • Rublee, E., Rabaud, V., Konolige, K., & Bradski, G. (2011). Orb: An efficient alternative to sift or surf. In 2011 International Conference on Computer Vision (pp. 2564–2571). IEEE.

  • Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations (ICLR 2015). Computational and Biological Learning Society

  • Sun, K., **ao, B., Liu, D., & Wang, J. (2019). Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5693–5703). IEEE

  • Sun, Y., **ng, Y., Zhao, Z., Meng, X., Xu, G., & Hai, Y. (2021). Comparison of manual versus automated measurement of cobb angle in idiopathic scoliosis based on a deep learning keypoint detection technology. European Spine Journal, 1–10.

  • Sun, Y., & Zheng, W. (2023). Hrnet-and pspnet-based multiband semantic segmentation of remote sensing images. Neural Computing and Applications, 35(12), 8667–8675.

    Google Scholar 

  • Thompson, L. F. (1983). An introduction to lithography. ACS Publications.

    Book  Google Scholar 

  • Von Gioi, R. G., Jakubowicz, J., Morel, J.-M., & Randall, G. (2008). Lsd: A fast line segment detector with a false detection control. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(4), 722–732.

    Article  Google Scholar 

  • Wang, Z., & Zhang, Z. (2021). Alignment algorithm of wafer dicing saw based on image sensor and machine learning (in chinese). Instrument Technique and Sensor, (7), 48.

  • Wang, J., Hall, T. A., Musbahi, O., Jones, G. G., & Arkel, R. J. (2023). Predicting hip-knee-ankle and femorotibial angles from knee radiographs with deep learning. The Knee, 42, 281–288.

    Article  Google Scholar 

  • Wang, N., Jiang, W., & Zhang, Y. (2021). Moiré-based sub-nano misalignment sensing via deep learning for lithography. Optics and Lasers in Engineering, 143, 106620.

    Article  Google Scholar 

  • Wang, N., Jiang, W., & Zhang, Y. (2021). Deep learning-based moiré-fringe alignment with circular gratings for lithography. Optics Letters, 46(5), 1113–1116.

  • Wang, C., Ni, M., Tian, S., Ouyang, H., Liu, X., Fan, L., Dong, P., Jiang, L., Lang, N., & Yuan, H. (2023). Deep learning model for measuring the sagittal cobb angle on cervical spine computed tomography. BMC Medical Imaging, 23(1), 196.

    Article  Google Scholar 

  • Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., Liu, W., & **ao, B. (2020). Deep high-resolution representation learning for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10), 3349–3364.

    Article  Google Scholar 

  • Wang, R., Yang, K., & Zhu, Y. (2023). A high-precision mark positioning algorithm based on sub-pixel shape template matching in wafer bonding alignment. Precision Engineering, 80, 104–114.

    Article  Google Scholar 

  • Wu, X., & Zou, G. (2013). High performance template matching algorithm based on edge geometric features. Chinese Journal of scientific instrument, 34(7), 1462–1469.

  • **ang, L., Gai, J., Bao, Y., Yu, J., Schnable, P. S., & Tang, L. (2023). Field-based robotic leaf angle detection and characterization of maize plants using stereo vision and deep convolutional neural networks. Journal of Field Robotics, 40, 1034–1053.

  • Ying, Y., Shichuan, W., Wei, Z., et al. (2023). Detection of the bolt loosening angle through semantic key point extraction detection by using an hourglass network. Structural Control and Health Monitoring, 2023, 8860412.

    Article  Google Scholar 

  • Zhang, X. (2021). An introduction to lithography machine. In 2021 6th International Conference on Modern Management and Education Technology (MMET 2021) (pp. 49–53). Atlantis Press.

  • Zhang, S., Zhao, L., & He, Y. (2021). Lithography alignment method based on image rotation matching. Journal of Physics: Conference Series, 1939, 012039. IOP Publishing.

    Google Scholar 

  • Zhao, Z., Song, A., Zheng, S., **ong, Q., & Guo, J. (2023). Dsc-hrnet: a lightweight teaching pose estimation model with depthwise separable convolution and deep high-resolution representation learning in computer-aided education. International Journal of Information Technology, 1–13.

  • Zhao, C., Cheung, C., & Liu, M. (2018). Integrated polar microstructure and template-matching method for optical position measurement. Optics Express, 26(4), 4330–4345.

    Article  Google Scholar 

  • Zhao, X., Zhang, Y., & Wang, N. (2019). Bolt loosening angle detection technology using deep learning. Structural Control and Health Monitoring, 26(1), 2292.

    Article  Google Scholar 

Download references

Funding

This work was funded by the National Natural Science Foundation of China under Grant (Grant Nos. 52171193, 61972092).

Author information

Authors and Affiliations

  1. School of Cyber Science and Engineering & Hanwei Institute of Internet of Things, Zhengzhou University, Zhengzhou, 450001, Henan, China

    Yulong Li, Yangjie Cao, Hui Tian & Ronghan Wei

  2. School of Mechanics and Safety Engineering, Zhengzhou University, Zhengzhou, 450001, Henan, China

    Shijiao Li, Guochen Qi & Ronghan Wei

  3. Engineering Technology Research Center of Henan Province for MEMS Manufacturing and Applications, Zhengzhou University, Zhengzhou, 450001, Henan, China

    Guochen Qi & Ronghan Wei

  4. Industrial Technology Research Institute, Zhengzhou University, Zhengzhou, 450001, Henan, China

    Ronghan Wei

Authors
  1. Yulong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yangjie Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shijiao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hui Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Guochen Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ronghan Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yangjie Cao or Ronghan Wei.

Ethics declarations

Conflict of interest

The authors have no relevant financial or nonfinancial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Y., Cao, Y., Li, S. et al. Subpixel keypoint localization and angle prediction for lithography marks based on deep learning. J Intell Manuf (2024). https://doi.org/10.1007/s10845-024-02400-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10845-024-02400-8

Keywords

Search

Navigation