SpringerLink
Log in

Smart surveillance with simultaneous person detection and re-identification

  • 1158T: Role of Computer Vision in Smart Cities: Applications and Research Challenges
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

This article has been updated

Abstract

When the faces of individuals are not clearly identifiable in surveillance videos due to variations in poses, camera viewpoints and occlusions, the appearances of people play a vital role in their identification. Appearance based person re-identification (re-id) summarizes appearances of persons to identify them across multiple non-overlap** camera views. Existing person re-id solutions work on the cropped person images to learn the salient features of a person instead of working on the raw surveillance images, hence these solutions need an independent preliminary phase of preparing cropped person datasets for the training and evaluation purposes. In contrast, the proposed solution works on the raw surveillance images instead of prerequisite of the cropped person dataset and the proposed hierarchical association building among various local parts of the images results in rich person representations for person re-id. In the proposed solution of Smart Surveillance with Simultaneous Person Detection and Re-identification (SSPDR), the complete surveillance video scenes are processed to perform simultaneous person detection and re-identification for all of the persons captured by a surveillance network. We use region proposals based localization scheme for person detection with an increased confidence strategy about the estimation of bounding boxes locations and the person re-identification module learns the hierarchical associations among local salient body parts of a person. Firstly, the proposed re-id module establishes associations among local horizontal strips of two persons, and afterwards it builds associations among local salient sub-patches of already associated pairs of horizontal strips. We address two major re-id challenges i.e. background noise and scale differences using the proposed re-id solution. In context of simultaneous person detection and re-identification, the proposed method is evaluated on publicly available re-id benchmark Person Re-identification in Wild (PRW) as well as on a local surveillance dataset, and attains state-of-the performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (France)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Change history

  • 25 August 2022

    Muhammad Shahzad’s email was change from muhammad.shehzad@tum.de to muhammad.shahzad@tum.de

References

  1. Ansar W, Fraz M M, Shahzad M, Gohar I, Javed S, Jung S K (2018) Two stream deep CNN-RNN attentive pooling architecture for video-based person re-identification. In: Progress in pattern recognition, image analysis, computer vision, and applications - 23rd Iberoamerican Congress, CIARP 2018, Madrid, Spain, November 19-22, 2018, Proceedings, pp 654–661

  2. Batool S, Ali M Z, Shahzad M, Fraz M M (2018) End to end person re-identification for automated visual surveillance. In: IEEE international conference on image processing, applications and systems, IPAS 2018, Sophia Antipolis, France, December 12-14, 2018, pp 220–225

  3. Cheng D, Gong Y, Shi W, Zhang S (2018) Person re-identification by the asymmetric triplet and identification loss function. Multimed Tools Appl 77(3):3533–3550

    Article  Google Scholar 

  4. Cheng D, Gong Y, Zhou S, Wang J, Zheng N (2016) Person re-identification by multi-channel parts-based CNN with improved triplet loss function. In: 2016 IEEE Conference on computer vision and pattern recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp 1335–1344

  5. Cho H, Rybski P E, Bar-Hillel A, Zhang W (2012) Real-time pedestrian detection with deformable part models. In: Intelligent vehicles symposium. IEEE, pp 1035–1042

  6. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: CVPR (1). IEEE Computer Society, pp 886–893

  7. Dollár P, Appel R, Belongie S J, Perona P (2014) Fast feature pyramids for object detection. IEEE Trans Pattern Anal Mach Intell 36(8):1532–1545

    Article  Google Scholar 

  8. Dollár P, Wojek C, Schiele B, Perona P (2012) Pedestrian detection: an evaluation of the state of the art. IEEE Trans Pattern Anal Mach Intell 34(4):743–761

    Article  Google Scholar 

  9. Felzenszwalb P F, Girshick R B, McAllester D A, Ramanan D (2010) Object detection with discriminatively trained part-based models. IEEE Trans Pattern Anal Mach Intell 32(9):1627–1645

    Article  Google Scholar 

  10. He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. ar**v:1512.03385

  11. He Y, Zhu C, Wang J, Savvides M, Zhang X (2019) Bounding box regression with uncertainty for accurate object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2888–2897

  12. He Z, Jung C, Fu Q, Zhang Z (2019) Deep feature embedding learning for person re-identification based on lifted structured loss. Multimed Tools Appl 78(5):5863–5880

    Article  Google Scholar 

  13. Kwak J Y, Ko B, Nam J-Y (2015) Multi-person tracking based on body parts and online random ferns learning of thermal images. In: 2015 IEEE Winter conference on applications of computer vision, WACV 2015, Waikoloa, HI, USA, January 5-9, 2015, pp 41–46

  14. Leng Q, Ye M, Tian Q (2019) A survey of open-world person re-identification. IEEE Trans Circuits Syst Video Technol, 1–1. https://doi.org/10.1109/TCSVT.2019.2898940, https://ieeexplore.ieee.org/document/8640834/

  15. Li D, Chen X, Zhang Z, Huang K (2017) Learning deep context-aware features over body and latent parts for person re-identification. ar**v:http://arxiv.org/

  16. Li T, Sun L, Han C, Guo J (2018) Person re-identification using salient region matching game. Multimed Tools Appl 77(16):21393–21415

    Article  Google Scholar 

  17. Li W, Zhao R, **ao T, Wang X (2014) Deepreid: deep filter pairing neural network for person re-identification. In: CVPR. IEEE Computer Society, pp 152–159

  18. Liu H, Feng J, Jie Z, Karlekar J, Zhao B, Qi M, Jiang J, Yan S (2017) Neural person search machines. In: IEEE International conference on computer vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp 493–501

  19. Luo H, Jiang W, Zhang X, Fan X, Qian J, Zhang C (2019) Alignedreid++: dynamically matching local information for person re-identification. Pattern Recogn 94:53–61

    Article  Google Scholar 

  20. Mubariz N, Mumtaz S, Hamayun M M, Fraz M M (2018) Optimization of person re-identification through visual descriptors. In: Proceedings of (VISIGRAPP 2018) - volume 4: VISAPP, Funchal, Madeira, Portugal, January 27-29, 2018, pp 348–355

  21. Mumtaz S, Mubariz N, Saleem S, Fraz M M (2017) Weighted hybrid features for person re-identification. In: Seventh international conference on image processing theory, tools and Applications, IPTA 2017, Montreal, QC, Canada, November 28 - December 1, 2017, pp 1–6

  22. Munjal B, Amin S, Tombari F, Galasso F (2019) Query-guided end-to-end person search. ar**v:1905.01203

  23. Nam W, Dollár P, Han J H (2014) Local decorrelation for improved pedestrian detection. In: Advances in neural information processing systems 27: annual conference on neural information processing systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp 424–432

  24. Ou X, Ma Q, Wang Y (2019) Improving person re-identification by multi-task learning. Multimed Tools Appl 78(19):28257–28283

    Article  Google Scholar 

  25. Perwaiz N, Fraz M M, Shahzad M (2019) Hierarchical refined local associations for robust person re-identification. In: 2019 International conference on robotics and automation in industry (ICRAI), pp 1–6

  26. Perwaiz N, Fraz MM, Shahzad M (2020) Smart visual surveillance: proactive person re-identification instead of impulsive person search. In: 2020 IEEE 23rd international multitopic conference (INMIC). IEEE, pp 1–6

  27. Perwaiz N, Fraz MM, Shahzad M (2021) Stochastic attentions and context learning for person re-identification. In: 2021 PeerJ Computer Science. PeerJ

  28. Perwaiz N, Fraz M M, Shahzad M (2018) Person re-identification using hybrid representation reinforced by metric learning. IEEE Access 6:77334–77349

    Article  Google Scholar 

  29. Qi M, Han J, Jiang J, Liu H (2019) Deep feature representation and multiple metric ensembles for person re-identification in security surveillance system. Multimed Tools Appl 78(19):27029–27043

    Article  Google Scholar 

  30. Ren S, He K, Girshick R B, Sun J (2017) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149

    Article  Google Scholar 

  31. Tang S, Ye M, Xu P, Li X (2019) Adaptive pedestrian detection by predicting classifier. Neural Comput Applic 31(4):1189–1200

    Article  Google Scholar 

  32. Tian Y, Li Q, Wang D, Wan B (2019) Robust joint learning network: improved deep representation learning for person re-identification. Multimed Tools Appl 78(17):24187–24203

    Article  Google Scholar 

  33. Verma A, Hebbalaguppe R, Vig L, Kumar S, Hassan E (2015) Pedestrian detection via mixture of CNN experts and thresholded aggregated channel features. In: ICCV workshops. IEEE Computer Society, pp 555–563

  34. Wu J, Jiang J, Qi M, Liu H (2019) Independent metric learning with aligned multi-part features for video-based person re-identification. Multimed Tools Appl 78(20):29323–29341

    Article  Google Scholar 

  35. **ao J, **e Y, Tillo T, Huang K, Wei Y, Feng J (2017) IAN: the individual aggregation network for person search. ar**v:1705.0555

  36. **ao T, Li S, Wang B, Lin L, Wang X (2017) Joint detection and identification feature learning for person search. In: CVPR. IEEE Computer Society, pp 3376–3385

  37. Xu Y, Ma B, Huang R, Lin L (2014) Person search in a scene by jointly modeling people commonness and person uniqueness. In: ACM Multimedia. ACM, pp 937–940

  38. Yang D, Zhang J, Xu S, Ge S, Kumar G H, Zhang X (2018) Real-time pedestrian detection via hierarchical convolutional feature. Multimed Tools Appl 77(19):25841–25860

    Article  Google Scholar 

  39. Zhai S, Liu S, Wang X, Tang J (2019) Fmt: fusing multi-task convolutional neural network for person search. Multimed Tools Appl

  40. Zhang N, Paluri M, Taigman Y, Fergus R, Bourdev L D (2015) Beyond frontal faces: improving person recognition using multiple cues. In: CVPR. IEEE Computer Society, pp 4804–4813

  41. Zhang S, Benenson R, Schiele B (2015) Filtered channel features for pedestrian detection. In: CVPR. IEEE Computer Society, pp 1751–1760

  42. Zhang S, Benenson R, Schiele B (2017) Citypersons: a diverse dataset for pedestrian detection. In: CVPR. IEEE Computer Society, pp 4457–4465

  43. Zhang S, Yang J, Schiele B (2018) Occluded pedestrian detection through guided attention in cnns. In: CVPR. IEEE Computer Society, pp 6995–7003

  44. Zhao K, Deng J, Cheng D (2018) Real-time moving pedestrian detection using contour features. Multimedia Tools Appl 77(23):30891–30910

    Article  Google Scholar 

  45. Zheng L, Yang Y, Hauptmann A G (2016) Person re-identification: past, present and future. ar**v:1610.02984

  46. Zheng L, Zhang H, Sun S, Chandraker M, Yang Y, Tian Q (2017) Person re-identification in the wild. In: 2017 IEEE Conference on computer vision and pattern recognition (CVPR). https://doi.org/10.1109/CVPR.2017.357, http://ieeexplore.ieee.org/document/8099840/. IEEE, Honolulu, pp 3346–3355

  47. Zhu F, Kong X, Wu Q, Fu H, Li M (2018) A loss combination based deep model for person re-identification. Multimed Tools Appl 77(3):3049–3069

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. National University of Sciences and Technology (NUST), Islamabad, Pakistan

    N. Perwaiz, M. M. Fraz & M. Shahzad

  2. Technical University of Munich, 80333, Munich, Germany

    M. Shahzad

  3. National Center of Artificial Intelligence (NCAI), Islamabad, Pakistan

    M. Shahzad

Authors
  1. N. Perwaiz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. M. Fraz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. M. Shahzad
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to N. Perwaiz.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Perwaiz, N., Fraz, M.M. & Shahzad, M. Smart surveillance with simultaneous person detection and re-identification. Multimed Tools Appl 83, 15461–15482 (2024). https://doi.org/10.1007/s11042-022-13458-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13458-y

Keywords

Search

Navigation