SpringerLink
Log in

Grid sampling based hypergraph matching technique for multiple objects tracking in video frames

  • 1229: Multimedia Data Analysis for Smart City Environment Safety
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Multiple Object Tracking (MOT), which is a vibrant research area in computer vision, has many multi-disciplinary applications. The evolution of Machine Learning (ML) techniques has brought out a wide variety of data analytics schemes especially in video processing for IOT implementations. Recently graph powered ML techniques with its added topological capabilities have gained much importance due to their applicability in computer vision applications. Yet complexity issues remain unresolved as several graph based techniques reduce the computational complexity at the cost of accuracy. Hypergraphs (HG) with its topological and geometric features add values to the existing features which increase the accuracy with reduced complexity and also pave a way to track multiple objects simultaneously. This paper presents one such novel HG matching algorithm (objects described by modified KAZE key-points) and a feature extraction technique to process video frames with multiple objects. For ascertaining independency of HG based KAZE features Chi – square test has been conducted with its value (1.0845), observed to be much less than the tabular value (7.962) to accept the null hypothesis at 5% level of significance. Among the 8 pre-processing schemes KAZE has been found to be an apt model to suit grid sampling-based HG representation and it has been proved by computational experiments that it produces new matches between frames. The evaluation of the proposed technique is done in terms of matching accuracy, score, and processing time. Grid sampling with HG, exhibited an average better tracking performance of 81.51% (of all consecutive pairs of frames with multiple objects) than the recently reported one of 73.88% accuracy (with single object image matching) with same reduced tensor size. Moreover, the matching accuracy between every pair of consecutive video frames is observed to be consistent and lying between 77.78 and 81.51%. Results obtained from this investigation clearly indicate the superiority of the proposed algorithm over recently reported ones in the literature in terms of consistency, accuracy and matching scores and its applicability in quick abnormal event detection in surveillance videos.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Algorithm 1
Algorithm 2
Fig. 18
Algorithm 3
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27

Similar content being viewed by others

Data Availability

Data will be made available on reasonable request.

References

  1. Guha P, Jain M, Pande N, Oberoi T (2011) Multiple face tracking with appearance modes and reasoning. In: Proceedings of the International Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV). The Steering Committee of The World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp), p 1. https://www.semanticscholar.org/paper/Multiple-Face-Tracking-with-Appearance-Modes-and-Guha-Jain/fc209dfbe9e7318d1c191f57c5e8a0b927cd96b4

  2. Fiscante N, Addabbo P et al (2021) A Track-Before-Detect Strategy Based on Sparse Data Processing for Air Surveillance Radar Applications, MDPI/Remote Sens. https://doi.org/10.3390/rs13040662

  3. Bloisi DD, Previtali F et al (2016) Enhancing Automatic Maritime Surveillance Systems With Visual Information. IEEE Trans Intell Transp Syst. https://doi.org/10.1109/TITS.2016.2591321

    Article  Google Scholar 

  4. Smal I, Meijering E, Draegestein K, Galjart N, Grigoriev I, Akhmanova A, van Royen ME, Houtsmuller AB, Niessen W (2008) Multiple object tracking in molecular bioimaging by rao-blackwellized marginal particle filtering. Med Image Anal 12(6):764–777. https://doi.org/10.1016/j.media.2008.03.004

    Article  Google Scholar 

  5. Smal I et al (2008) Multiple object tracking in molecular bioimaging by Rao-Blackwellized marginal particle filtering. Med Image Anal. https://doi.org/10.1007/978-3-540-73273-0_10

  6. Shen D, Wu G, Suk H-I (2021) Deep Learning in Medical Image Analysis, Annu Biomed Eng. https://doi.org/10.1146/annurev-bioeng-071516-044442

  7. Doran MM, Hoffman JE (2010) The role of visual attention in multiple object tracking evidence from ERPs, Atten Percept Psychophys. https://doi.org/10.3758/APP.72.1.33

  8. Park Y, Dang LM et al. (2021) Multiple object tracking in deep learning approaches: a survey, MDPI/Electron. https://doi.org/10.3390/electronics10192406

  9. Ji Z, Zhang Y, Pang, Y Li X (2019) Hypergraph dominant set based multi-video summarization, Signal Process 41–50. https://doi.org/10.1016/j.sigpro.2018.01.028

  10. **ng Junliang et al (2021) Multiple object tracking: A literature review. Artif Intell 293:103448. https://doi.org/10.1016/j.artint.2020.103448

    Article  MathSciNet  Google Scholar 

  11. Lombardi E, Wolf C, Celiktutan O, Sankur B (2015) Activity recognition from videos with parallel hypergraph matching on GPUs, Comput Vis Pattern Recognit. https://doi.org/10.48550/ar**v.1505.0058

  12. Dinesh Singha C, Mohana Krishna (2019) Graph formulation of video activities for Abnormal activity recognition. Pattern Recognition 65(265):272. https://doi.org/10.1016/j.patcog.2017.01.001

    Article  Google Scholar 

  13. Chen X-J, Zhan, Ke J, Chen X-B (2016) Complex video event detection via pairwise fusion of trajectory and multi-label hypergraphs, Multimed Tools Appl 15079–15100. https://doi.org/10.1007/s11042-015-2514-8

  14. Yan Y, Qin JY, Chen J, Liu L, Zhu F, Tai Y, Shao L (n.d.) Learning Multi-Granular Hypergraphs for Video-Based Person Re-Identification. https://doi.org/10.48550/ar**v.2104.14913

  15. Chen, HX, Zhang J-G, Ke J et. al (2016) Hypergraph Partitioning for Video Event Detection, IEEE Symp Serv-Orient Syst Eng. https://doi.org/10.1109/SOSE.2016.33

  16. Caetano T S, McAuley JJ et. al (2009) Learning Graph Matching, IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.48550/ar**v.0806.2890

  17. Cho M, Alahari K, Ponce J (2013) Learning Graphs to Match. IEEE Int Conf Comput Vis. https://doi.org/10.1109/ICCV.2013.11

  18. Yadav P, Curry E (2019) VEKG: Video Event Knowledge Graph to Represent Video Streams for Complex Event Pattern Matching in First International Conference on Graph Computing. https://doi.org/10.1109/GC46384.2019.00011

  19. Hasan, M, Choi, J, Neumann, J, Roy-Chowdhury, AK, Davis, LS (2016) Learning temporal regularity in video sequences. IEEE Access, 733–742. https://doi.org/10.48550/ar**v.1604.04574

  20. Li W, Liu X, Yuan Y (2022) SIGMA: Semantic-complete Graph Matching for Domain Adaptive Object Detection, Comput Vis Pattern Recognit, Cornell University. https://doi.org/10.48550/ar**v.2203.06398

  21. Munjal B, Aftab AR, Amin S, Brandlmaier MD, Tombari F, Galasso F (2020) Joint detection and tracking in videos with identification features, Image Vis Comput. https://doi.org/10.48550/ar**v.2005.10905

  22. Fan Y, Wen G, Li D, Qiu S, Levine MD, **ao F (2020) Video anomaly detection and localization via gaussian mixture fully convolutional variational autoencoder, Comput Vis Image Underst. https://doi.org/10.48550/ar**v.1805.11223

  23. Chen D, Wang P, Yue L, Zhang Y, Jia T (2020) Anomaly detection in surveillance video based on bidirectional prediction, Image Vis Comput. https://doi.org/10.1016/j.imavis.2020.103915

  24. Srinivasan P, Doraipandian M (2020) Framework for rare event detection using Artificial Neural, Network based context free grammar. J Intell Fuzzy Syst. https://doi.org/10.3233/JIFS-189164

    Article  Google Scholar 

  25. Sukumaran V, Samuelsson J, Forslow J (2016) U.S. Patent No. 9,246,924. U.S. Patent and Trademark Office, Washington, DC

  26. Poulding S, Alexander R, Clark JA, Hadleyb MJ (2015) The optimization of stochastic grammars to enable cost-effective probabilistic structural testing. J Syst Softw 103:296–310. https://doi.org/10.1016/j.jss.2014.11.042

    Article  Google Scholar 

  27. Kwon J, Lee KM (2015) A unified framework for event summarization and rare event detection from multiple views, IEEE Access, 1737–1750. https://doi.org/10.1109/TPAMI.2014.2385695

  28. Kwon J, Lee KM (2012) A unified framework for event summarization and rare event detection, IEEE Access 266–1273. https://doi.org/10.1109/CVPR.2012.6247810

  29. Wang H, Kläser A, Schmid C, Liu C-L (2011) Action recognition by dense trajectories in CVPR 2011, IEEE Access, 3169-3176. https://doi.org/10.1109/CVPR.2011.5995407

  30. Chau DP, Brémond F, Thonnat M, Corvée E (2011) Robust mobile object tracking based on multiple feature similarity and trajectory filtering. VISAPP 2011 - Proceedings of the Sixth International Conference on Computer Vision Theory and Applications, Vilamoura, Algarve, Portugal, 569–574. https://doi.org/10.48550/ar**v.1106.2695

  31. Zaidenberg S, Boulay B, Garate C, Chau DP, Corveeand E, Bremond F (2011) Group interaction and group tracking for video surveillance in underground railway stations, International Workshop on Behaviour Analysis and Video Understanding (ICVS). Sophia Antipolis, France. https://doi.org/10.1109/ICME.2008.4607367

    Article  Google Scholar 

  32. Ryoo MS, Aggarwal JK (2006) Recognition of composite human activities through context-free grammar based representation, IEEE Access,1–8. https://doi.org/10.1109/CVPR.2006.242

  33. Park S, Aggarwal JK (2004) Semantic-level understanding of human actions and interactions using event hierarchy, IEEE Access, 66–78. https://doi.org/10.1109/CVPR.2004.434

  34. Moon S, Lee J, Nam D, Kim H, Kim W (2017) A comparative study on multi-object tracking methods for sports events, 19th Int Conf Adv Commun Technol. https://doi.org/10.23919/ICACT.2017.7890221

  35. Bansal M, Kumar M, Sachdeva M, Mittal A (2021) Transfer learning for image classification using VGG19: Caltech-101 image data set. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-021-03488-z

    Article  Google Scholar 

  36. Walia S, Kumar K, Kumar M (2023) Unveiling digital image forgeries using Markov based quaternions in frequency domain and fusion of machine learning algorithms. Multimed Tools Appl. https://doi.org/10.1007/s11042-022-13610-8

    Article  Google Scholar 

  37. Kumar A, Kumar M, Kaur A (2021) Face detection in still images under occlusion and non-uniform illumination. Multimed Tools Appl. https://doi.org/10.1007/s11042-020-10457-9

    Article  Google Scholar 

  38. Shaheed K, Mao A, Qureshi I, Kumar M et al (2022) DS-CNN: A pre-trained Xception model based on depth-wise separable convolutional neural network for finger vein recognition. Expert Syst Appl. https://doi.org/10.1016/j.eswa.2021.116288

    Article  Google Scholar 

  39. Ahuja U, Singh S, Kumar M, Kumar K, Sachdeva M (2021) COVID-19: Social distancing monitoring using faster-RCNN and YOLOv3 algorithms. Multimed Tools Appl. https://doi.org/10.1007/s11042-022-13718-x

    Article  Google Scholar 

  40. Bretto A, Cherifi H, Ubéda S (2001) An efficient algorithm for Helly property recognition in a linear hypergraph. Electron Notes Theoretic Comput Sci 46:181–191. https://doi.org/10.1016/S1571-0661(04)80985-X

    Article  Google Scholar 

  41. Kannan K, Kanna BR, Aravindan C (2010) Root mean square filter for noisy images based on hyper graph model. Image Visual Computing 28(9):1329–1338. https://doi.org/10.1016/j.imavis.2010.01.013

    Article  Google Scholar 

  42. Dharmarajan R (2016) Studies in Hypergraphs with a few applications in Image Processing (Doctoral dissertation). SASTRA Deemed to be University, Thanjavur

    Google Scholar 

  43. Rajesh Khanna B (2012) Development of hypergraph-based techniques for selected image engineering applications. (Doctoral Dissertation), SASTRA Deemed to be University, Thanjavur

    Google Scholar 

  44. Wang R, Yan J, Yang X (2015) Neural graph matching network: learning lawler’s quadratic assignment problem with extension to hypergraph and multiple-graph matching. J Latex Class Files. https://doi.org/10.1109/TPAMI.2021.3078053

    Article  Google Scholar 

  45. Du D, Qi H, Wen L, Tian Q, Huang Q, Lyu S (2016) Geometric hypergraph learning for visual tracking. IEEE Trans Cybern. https://doi.org/10.48550/ar**v.1603.05930

  46. Gao Y, Zhang Z, Lin H, Zhao X, Shaoyi Du, Zou C (2022) Hypergraph learning: methods and practices. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2020.3039374

    Article  Google Scholar 

  47. Zhang H, Du B, Wang Y, Ren P (2015) A hypergraph matching framework for refining multi-source feature correspondences. Int Workshop Graph-Based Represent Pattern Recognit. https://doi.org/10.1007/978-3-319-18224-7_11

  48. Sichao Fu, Liu W, Zhou Y, LiqiangNie (2019) HpLapGCN: hypergraph p- laplacian graph convolutional networks. Neurocomputing. https://doi.org/10.1016/j.neucom.2019.06.068

    Article  Google Scholar 

  49. Nguyen Q, Tudisco F, Gautier A, Hein M (2017) An efficient multilinear optimization framework for hypergraph matching. IEEE Trans Pattern Anal Mach Intell 39(6):1054–1075. https://doi.org/10.1109/TPAMI.2016.2574706

    Article  Google Scholar 

  50. Divya LK, Rajappa M, Krithivasan K, Roy DS (2019) Helly hypergraph based matching framework using deterministic sampling techniques for spatially improved point feature-based image matching. Multimed Tools Appl 78(11):14657–14681. https://doi.org/10.1007/s11042-018-6852-1

    Article  Google Scholar 

  51. Wen L, Lei Z, SiweiLyu SZ, Li F, IEEE, and Ming-Hsuan Yang (2016) Exploiting hierarchical dense structures on hypergraphs for multi-object tracking. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2015.2509979

    Article  Google Scholar 

  52. Hou J, Yuan H (2021) Efficient and accurate hypergraph matching. IEEE Intern Conf Multimed Expo (ICME). https://doi.org/10.1109/ICME51207.2021.9428156

    Article  Google Scholar 

  53. Ass R, Shashua A (2008) Probabilistic graph and hypergraph matching. In 26th IEEE Conference on Computer Vision and Pattern Recognition, CVPR. https://doi.org/10.1109/CVPR.2008.4587500

  54. Lee J, MCho, KM Lee (2011) Hypergraph matching via reweighted random walks. CVPR. https://doi.org/10.1109/CVPR.2011.5995387

    Article  Google Scholar 

  55. Cho M, Lee KM (2012) Progressive graph matching: Making a move of graphs via probabilistic voting, IEEE Conf Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR.2012.6247701

  56. Du D, Qi H, Wen L, Tian Q, Huang Q, Lyu S (2016) Geometric hypergraph learning for visual tracking. IEEE Trans Cybern. https://doi.org/10.48550/ar**v.1603.05930

  57. Somu N, Kirthivasan K, Shankar SS (2017) A computational model for ranking cloud service providers using hypergraph-based techniques. Fut Gen Comput Syst 68:14–30. https://doi.org/10.1016/j.future.2016.08.014

    Article  Google Scholar 

  58. Acar Esra, Hopfgartner Frank, Albayrak Sahin (2017) A comprehensive study on mid-level representation and ensemble learning for emotional analysis of video material. Multimed Tools Appl 76:11809–11837. https://doi.org/10.1007/s11042-016-3618-5

    Article  Google Scholar 

  59. Janjua Zaffar Haider, Vecchio Massimo, Antonini Mattia, Antonelli Fabio (2019) An intelligent rare-event detection system using unsupervised learning on the IoT edge. Eng Appl Artif Intell 84:41–50. https://doi.org/10.1016/j.engappai.2019.05.011

    Article  Google Scholar 

  60. Shokri M, Harati A, Taba K (2020) Salient object detection in video using deep non-local neural networks. J Vis Commun Image Represent, 102769. https://doi.org/10.1016/j.jvcir.2020.102769

  61. Shaheed K, Mao A, Qureshi I, Kumar M, Hussain S, Zhang X (2022) Recent advancements in finger vein recognition technology: methodology, challenges and opportunities. ELSEVIER Inf Fus. https://doi.org/10.1016/j.inffus.2021.10.004

    Article  Google Scholar 

  62. Bansal Monika, Kumar Munish, Kumar Manish (2021) 2D object recognition: A comparative analysis of SIFT, SURF and ORB feature descriptor. Multimed Tools Appl 80(12):18839–18857. https://doi.org/10.1007/s11042-021-10646-0

    Article  Google Scholar 

  63. Walia S, Kumar K, Kumar M, Gao X-Z (2021) Fusion of handcrafted and deep features for forgery detection in digital images. IEEE Access. https://doi.org/10.1109/ACCESS.2021.3096240

    Article  Google Scholar 

  64. Bansal M, Kumar M, Sachdeva M, Mitta A (2021) Transfer learning for image classification using VGG19: Caltech-101 image data set. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-021-03488-z

    Article  Google Scholar 

  65. Monika MK, Kumar M (2021) XGBoost:2D-Object Recognition Using Shape Descriptors and Extreme Gradient Boosting Classifier. Comput Methods Data Eng Adv Intell Syst Comput. https://doi.org/10.1007/978-981-15-6876-3_16

    Article  Google Scholar 

  66. Te G, WHu, Z Guo (2020) Exploring hypergraph representation on face anti-spoofing beyond 2D attacks. IEEE Int Conf Multimed Expo (ICME). https://doi.org/10.1109/ICME46284.2020.9102720

    Article  Google Scholar 

Download references

Acknowledgements

The authors are grateful to express their sincere thanks to SASTRA Deemed to be University, Thanjavur for extending ‘computing center’ support to carry out this research work. One of the authors of the paper wishes to acknowledge Department of Science and Technology, Government of India for their financial support to carry out this project (Grant No. DST/CRG/2023/006090) and BIRAC-BIG Grant (BIRAC/CCAMP0949/BIG-14/19), CCamp (Centre for Cellular and Molecular Platforms) for their financial support.

Author information

Authors and Affiliations

  1. SASTRA Deemed University, Thanjavur, Tamil Nadu, India

    Palanivel Srinivasan, Manivannan Doraipandiyan, Vamsi Panchada & Kannan Krithivasan

  2. VIT University Vellore, Vellore, Tamil Nadu, India

    K. Divya Lakshmi

Authors
  1. Palanivel Srinivasan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Manivannan Doraipandiyan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. K. Divya Lakshmi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Vamsi Panchada
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kannan Krithivasan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Manivannan Doraipandiyan.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of Interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Srinivasan, P., Doraipandiyan, M., Lakshmi, K.D. et al. Grid sampling based hypergraph matching technique for multiple objects tracking in video frames. Multimed Tools Appl 83, 62349–62378 (2024). https://doi.org/10.1007/s11042-023-17486-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-17486-0

Keywords

Search

Navigation