SpringerLink
Log in

Tracking People in Video Using Neural Network Features and Facial Identification Taking into Account the Mask Mode

  • APPLICATION PROBLEMS
  • Published:
Pattern Recognition and Image Analysis Aims and scope Submit manuscript

Abstract

Detection and tracking of people in video in distributed video surveillance systems is a difficult task, which has become even more complicated in the conditions of the mask mode, when some people may be wearing masks. To solve this problem, the paper proposes algorithms for detecting masked people and further tracking them using facial recognition systems based on neural networks. To train a neural network to detect masked faces, an approach is proposed that involves applying masks to faces from existing data sets, which makes it possible to expand the training sample and increase the accuracy of recognition of masked faces. The features of masked faces are used to establish the correspondence of people in the frames. This makes it possible to increase the efficiency of detection and tracking upon hiding of people behind objects of the background, high similarity of external features of several people, and analysis of the trajectories of their movement. Examples of detection and tracking of people are shown and appropriate recommendations are given.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.

REFERENCES

  1. A. Alzu’bi, F. Albalas, T. Al-Hadhrami, L. Bani Younis, and A. Bashayreh, “Masked face recognition using deep learning: A review,” Electronics 10, 2666 (2021). https://doi.org/10.3390/electronics10212666

    Article  Google Scholar 

  2. R. Bohush, G. Ma, Ya. Weichen, and S. Ablameyko, “Object detection in video surveillance based on multiscale frame representation and block processing by a convolutional neural network,” Pattern Recognit. Image Anal. 32, 1–10 (2022). https://doi.org/10.1134/S1054661822010035

    Article  Google Scholar 

  3. P. Charles, S. Jasmine Sultana, P. Hemalatha, and E. Keerti, “Multiple person detection and tracking using convolutional neural network,” Int. J. Adv. Res. Innovation 8, 156–159 (2020).

    Google Scholar 

  4. H. Chen, R. Bohush, I. Kurnosov, G. Ma, Y. Weichen, and S. Ablameyko, “Detection of appearance and behavior anomalies in stationary camera videos using convolutional neural networks,” Pattern Recognit. Image Anal. 32, 254–265 (2022). https://doi.org/10.1134/S1054661822020067

    Article  Google Scholar 

  5. Ch. Chen, I. Kurnosov, G. Ma, Y. Weichen, and S. Ablameyko, “Masked face recognition using generative adversarial networks by restoring the face closed part,” Opt. Mem. Neural Networks 32 (1) 1–13 (2023).

  6. Sh. Chen, Ya. Liu, X. Gao, and Zh. Han, “MobileFaceNets: Efficient CNNs for accurate real-time face verification on mobile devices,” in Biometric Recognition. CCBR 2018, Ed. by J. Zhou, Yu. Wang, Zh. Sun, Zh. Jia, J. Feng, Sh. Shan, K. Ubul, and Zh. Guo, Lecture Notes in Computer Science, Vol. 10996 (Springer, Cham, 2018), pp. 428–438. https://doi.org/10.1007/978-3-319-97909-0_46

    Book  Google Scholar 

  7. F. Firdaus and R. Munir, “Masked face recognition using deep learning based on unmasked area,” in Second Int. Conf. on Power, Control and Computing Technologies (ICPC2T), Raipur, India, 2022 (IEEE, 2022), pp. 1–6. https://doi.org/10.1109/ICPC2T53885.2022.9776651

  8. G. Ciaparrone, F. L. Sánchez, S. Tabik, L. Troiano, R. Tagliaferri, and F. Herrera, “Deep learning in video multi-object tracking: A survey,” Neurocomputing 381, 61–88 (2020). https://doi.org/10.1016/j.neucom.2019.11.023

    Article  Google Scholar 

  9. R. Golwalkar and N. Mehendale, “Masked-face recognition using deep metric learning and FaceMaskNet-21,” Appl. Intell. 52, 13268–13279 (2022). https://doi.org/10.1007/s10489-021-03150-3

    Article  Google Scholar 

  10. Face Verification on Labeled Faces in the Wild. https://paperswithcode.com/sota/face-verification-on-labeled-faces-in-the. Cited October 16, 2022

  11. Face Verification on YouTube Faces DB. https://paperswithcode.com/sota/face-verification-on-youtube-faces-db. Cited October 16, 2022

  12. U. Iqbal, A. Milan, and J. Gall, “PoseTrack: Joint multi-person pose estimation and tracking,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Honolulu, Hawaii, 2017 (IEEE, 2017), pp. 4654–4663. https://doi.org/10.1109/CVPR.2017.495

  13. K. Koide, E. Menegatti, M. Carraro, M. Munaro, and J. Miura, “People tracking and re-identification by face recognition for RGB-D camera networks,” in Proc. 2017 European Conf. on Mobile Robots (ECMR), Paris, 2017 (IEEE, 2017), pp. 1–7. https://doi.org/10.1109/ECMR.2017.8098689

  14. Multiple Object Tracking Benchmark. https://motchallenge.net/. Cited October 16, 2022

  15. P. Nagrath, R. Jain, A. Madan, R. Arora, P. Kataria, and J. Hemanth, “SSDMNV2: A real time DNN-based face mask detection system using single shot multibox detector and MobileNetV2,” Sustainable Cities Soc. 66, 102692 (2021). https://doi.org/10.1016/j.scs.2020.102692

    Article  Google Scholar 

  16. F. Schroff, D. Kalenichenko, and J. Philbin, “FaceNet: A unified embedding for face recognition and clustering,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Boston, 2015 (IEEE, 2015), pp. 815–823. https://doi.org/10.1109/CVPR.2015.7298682

  17. S. Sen and K. Sawant, “Face mask detection for covid_19 pandemic using pytorch in deep learning,” IOP Conf. Series: Mater. Sci. Eng. 1070, 012061 (2021). https://doi.org/10.1088/1757-899X/1070/1/012061

    Article  Google Scholar 

  18. S. Sethia, M. Kathuria, and T. Kaushik, “Face mask detection using deep learning: An approach to reduce risk of Coronavirus spread,” J. Biomed. Inf. 120, 103848 (2021). https://doi.org/10.1016/j.jbi.2021.103848

    Article  Google Scholar 

  19. S. Singh, U. Ahuja, M. Kumar, K. Kumar, and M. Sachdeva, “Face mask detection using YOLOv3 and faster R-CNN models: COVID-19 environment,” Multimedia Tools Appl. 80, 19753–19768 (2021). https://doi.org/10.1007/s11042-021-10711-8

    Article  Google Scholar 

  20. C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4, Inception-Resnet and the impact of residual connections on learning,” Proc. AAAI Conf. Artif. Intell. 31, 4278–4284 (2017). https://doi.org/10.1609/aaai.v31i1.11231

  21. S. Ye, R. Bohush, C. Chen, I. Zakharova, and S. Ablameyko, “Person tracking and re-identification in video for indoor multi-camera surveillance systems,” Pattern Recognit. Image Anal. 30, 827–837 (2020). https://doi.org/10.1134/S1054661820040136

    Article  Google Scholar 

Download references

Funding

The work was partially supported by the Public Welfare Technology Applied Research Program of Zhejiang Province (project no. LGF19F020016) and the National High-End Foreign Experts Program (project nos. G2021016028L, G2021016002L, and G2021016001L).

Author information

Authors and Affiliations

  1. Zhejiang Shuren University, 310015, Hangzhou, China

    Shi** Ye

  2. International Science and Technology Cooperation Base of Zhejiang Province: Remote Sensing Image Processing and Application, 310015, Hangzhou, China

    Shi** Ye

  3. Belarusian State University, 220030, Minsk, Republic of Belarus

    I. L. Kurnosov & S. V. Ablameyko

  4. Polotsk State University, 211440, Novopolotsk, Republic of Belarus

    R. P. Bohush

  5. EarthView Image Inc., 313200, Huzhou, China

    Guangdi Ma

  6. United Institute for Informatics Problems, National Academy of Sciences of Belarus, 220012, Minsk, Republic of Belarus

    Yang Weichen & S. V. Ablameyko

Authors
  1. Shi** Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. I. L. Kurnosov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. R. P. Bohush
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guangdi Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yang Weichen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. S. V. Ablameyko
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Shi** Ye, I. L. Kurnosov, R. P. Bohush, Guangdi Ma, Yang Weichen or S. V. Ablameyko.

Ethics declarations

The authors declare that they have no conflicts of interest.

Additional information

Shi** Ye. Born in 1967. Professor and Vice President of Zhejiang Shuren University. Graduated from Zhejiang University in 1988. Received Master’s degree in computer science and technology from Zhejiang University in 2003. Scientific interests: application of computer graphics and images, GIS, machine learning. Author of more than 70 papers. Four research projects he has taken part in have been awarded the second prize of Zhejiang Provincial Scientific and Technological Achievement. Two teaching research programs he has presided over have been awarded first prize and second prize of Zhejiang Provincial Teaching Achievement.

Ivan Kurnosov. Born in 2001. Graduated from Belarusian State University (mathematics) in 2022. He is a software engineer, lead of Artificial Intelligence and Data Science Community in Exadel Inc. Bronze medalist of March Machine Learning Mania 2021—NCAWW competition on Kaggle platform. His scientific interests include image classification and recognition, natural language processing, and segmentation.

Rykhard Bohush. Graduated from Polotsk State University in 1997. In 2002, he received his Candidate of Sciences degree, and in 2022, he received his Doctor of Sciences degree. Head of Computer Systems and Networks Department of Polotsk State University. His scientific interests include image and video processing, object representation and recognition, intelligent systems, and machine learning.

Guangdi Ma. Born in 1985. Graduated from Chinese Academy of Surveying and Map** in 2011. Chief Engineer of EarthView Image Inc. His scientific interests are image analysis, photogrammetry, point cloud, and oblique photography aided real 3D reconstruction.

Yang Weichen. Born in 1979. Graduated from Jilin University, China, in 2001. General manager of EarthView Image Inc. His scientific interests are image analysis, photogrammetry, and geographical information systems. Pioneered the business service mode of remote sensing target recognition to assist refined social governance in China.

Sergey Ablameyko. Born in 1956, DipMath in 1978, Candidate of Sciences in 1984, Doctor of Sciences in 1990, Professor in 1992. Professor at Belarusian State University. His scientific interests are image analysis, pattern recognition, digital geometry, knowledge-based systems, geographical information systems, and medical imaging. He is on the Editorial Board of Pattern Recognition and Image Analysis, Nonlinear Phenomena in Complex Systems, and many other international and national journals. He is Fellow of IAPR, Fellow of AAIA, Academician of National Academy of Sciences of Belarus, Academician of the European Academy, and many other academies. Honorary Professor of Moscow State University (Russia), Dalian University of Technology (China), and many other universities. He is a Vice-President of Asia-Pacific Artificial Intelligence Association.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shi** Ye, Kurnosov, I.L., Bohush, R.P. et al. Tracking People in Video Using Neural Network Features and Facial Identification Taking into Account the Mask Mode. Pattern Recognit. Image Anal. 33, 208–216 (2023). https://doi.org/10.1134/S1054661823020177

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S1054661823020177

Keywords:

Search

Navigation