SpringerLink
Log in

Dynamic graph neural network-based computational paradigm for video summarization

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In recent times, there has been a significant amount of interest in video summary technology. The reason behind video summarizing is to condense the information from the original video into a concise yet comprehensive summary, resulting in a brief and easy-to-understand version. In this study, looking at video summarization as an examination of graph issues and proposing a dynamic graph neural network to quantify the significance of individual videos and their overall contribution to the collection are covered. In prior graph network-based techniques, a single graph neural network (GNN) had often been used. The advantages of various graph filters or graph neural networks have not been fully exploited. The problem of oversmoothing still exists with traditional GNNs. To address issues like the one mentioned above, a spectral filter and an autoregressive moving average filter for the dynamic graph neural network have been proposed. In addition, a regularization of diversity to promote the model and provide a diverse summary has been recommended. Numerous tests have been run, and innovative and conventional graph models have been compared with the most advanced video summarizing techniques. The F-score for the two datasets got a big boost from 1.9 to 3.1% when the proposed method was used.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

The data used to support the findings of this study are available from the corresponding author upon request.

References

  1. Fei M, Jiang W, Mao W (2017) Memorable and rich video summarization. J Vis Commun Image Represent 42:207–217

    Article  Google Scholar 

  2. Fei M, Jiang W, Mao W (2018) Creating memorable video summaries that satisfy the user’s intention for taking the videos. Neurocomputing 275:1911–1920

    Article  Google Scholar 

  3. Ji Z, Ma Y, Pang Y, Li X (2019) Query-aware sparse coding for web multi-video summarization. Inf Sci 478:152–166

    Article  Google Scholar 

  4. Potapov D, Douze M, Harchaoui Z, Schmid C (2014) Category-specific video summarization. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds) Computer vision – ECCV 2014. ECCV 2014. Lecture Notes in Computer Science, vol 8694. Springer, Cham. https://doi.org/10.1007/978-3-319-10599-4_35

  5. Cahuina EJYC, Chavez GC (2013) A new method for static video summarization using local descriptors and video temporal segmentation, 2013 XXVI Conference on Graphics, Patterns and Images, Arequipa, Peru, 226–233. https://doi.org/10.1109/SIBGRAPI.2013.39

  6. Sha-Sha Z, Hui Y, Yan S, Ru Z (2023) Unsupervised video summarization using deep non-local video summarization networks. Neurocomputing 519:26–35. https://doi.org/10.1016/j.neucom.2022.11.028

    Article  Google Scholar 

  7. Sunil SH, Shaik RS, Vikash K, Sunil Kumar B, Adithya VA, Veena IP (2022) Robust video summarization algorithm using supervised machine learning. Global Trans Proc 3(1):131–135. https://doi.org/10.1016/j.gltp.2022.04.009

    Article  Google Scholar 

  8. Jung Y, Cho D, Kim D, Woo S, Kweon I (2109) S Discriminative Feature Learning for Unsupervised Video Summarization. Proceedings of the AAAI Conference on Artificial Intelligence 33(01):8537–8544. https://doi.org/10.1609/aaai.v33i01.33018537

  9. Zhou K, **ang T, Cavallaro A (2018) Video summarisation by classification with deep reinforcement learning. pp 1–13. https://doi.org/10.48550/ar**v.1807.03089

  10. Zhang K, Chao WL, Sha F, Grauman K (2016) Video summarization with long short-term memory. In: European Conference on Computer Vision; Springer: Berlin/Heidelberg, Germany 766–782

  11. Li P, Ye Q, Zhang L, Yuan L, Xu X, Shao L (2021) Exploring global diverse attention via pairwise temporal relation for video summarization. Pattern Recognit 111:107677

    Article  Google Scholar 

  12. Guangyu G, Ziming L, Guangjun Z, **yang L, Qin AK (2023) DANet: semi-supervised differentiated auxiliaries guided network for video action recognition. Neural Netw 158:121–131. https://doi.org/10.1016/j.neunet.2022.11.009

    Article  Google Scholar 

  13. Xufeng H, Yang H, Tao S, Zongpu Z, Zhengui X, Ruhui M, Neil R (2019) Unsupervised Video Summarization with Attentive Conditional Generative Adversarial Networks, Proceedings of the 27th ACM International Conference on Multimedia 2296–2304. https://doi.org/10.1145/3343031.3351056

  14. Zhou K, Qiao Y, **ang T (2018) Deep reinforcement learning for unsupervised video summarization with diversity-representativeness reward. In: Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, 32:1–15

  15. Yoon UN, Hong MD, Jo GS (2021) Interp-SUM: unsupervised video summarization with Piecewise Linear Interpolation. Sensors 21:1–13

    Article  Google Scholar 

  16. Fan L, Wang W, Zhu SC, Tang X, Huang S (2019) Understanding human gaze communication by spatio-temporal graph reasoning. In: The IEEE International Conference on Computer Vision, 5723–5732

  17. Junjie J, Zaixing H, Shuyou Z, **nyue Z, Jianrong T (2021) Learning to transfer focus of graph neural network for scene graph parsing. Pattern Recogn 112:107707. https://doi.org/10.1016/j.patcog.2020.107707

    Article  Google Scholar 

  18. Wei-Chia H, Chiao-Ting C, Chi L, Fan-Hsuan K, Szu-Hao H (2023) Attentive gated graph sequence neural network-based time-series information fusion for financial trading. Inform Fusion 91:261–276. https://doi.org/10.1016/j.inffus.2022.10.006

    Article  Google Scholar 

  19. Mohammadreza G, Mohammadreza K, Mohammad TH, Amin RB (2023) Spatial graph convolutional neural network via structured subdomain adaptation and domain adversarial learning for bearing fault diagnosis. Neurocomputing 517:44–61. https://doi.org/10.1016/j.neucom.2022.10.057

    Article  Google Scholar 

  20. Hui F, Guilin C, Haixiang X, Shuzhi SG (2022) IS-STGCNN: an Improved Social spatial-temporal graph convolutional neural network for ship trajectory prediction. Ocean Eng 266:1–15. https://doi.org/10.1016/j.oceaneng.2022.112960

    Article  Google Scholar 

  21. Minyao Q, **aoqi L, Siyao D, Yufang L, Yanlan K, **qing W, Hu M (2022) A unified GCNN model for predicting CYP450 inhibitors by using graph convolutional neural networks with attention mechanism. Comput Biol Med 150:1–16. https://doi.org/10.1016/j.compbiomed.2022.106177

    Article  Google Scholar 

  22. Hamilton WL, Ying R, Leskovec J (2017) Inductive representation learning on large graphs. Adv Neural Inf Process Syst (NIPS 2017), Long Beach, CA, USA pp 1–19. https://doi.org/10.48550/ar**v.1706.02216

  23. Defferrard M, Bresson X, Vandergheynst P (2016) Convolutional neural networks on graphs with fast localized spectral filtering, 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain. pp 1–9. https://doi.org/10.48550/ar**v.1606.09375

  24. Min Z, **anjun L, Zaiyu X, Jiliang M, Shihao X (2023) Diagnosis of brake friction faults in high-speed trains based on 1DCNN and GraphSAGE under data imbalance. Measurement 207:1–16. https://doi.org/10.1016/j.measurement.2022.112378

    Article  Google Scholar 

  25. Liu J, Ong GP, Chen X (2022) GraphSAGE-Based traffic speed forecasting for Segment Network with Sparse Data. IEEE Trans Intell Transp Syst 23(3):1755–1766. https://doi.org/10.1109/TITS.2020.3026025

    Article  Google Scholar 

  26. Karnyoto AS, Sun C, Liu B, Wang X (2022) Augmentation and heterogeneous graph neural network for AAAI2021-COVID-19 fake news detection. Int. J. Mach. Learn. & Cyber. 13:2033–2043. https://doi.org/10.1007/s13042-021-01503-5

  27. Desheng W, Quanbin W, David LO (2023) Industry classification based on supply chain network information using graph neural networks. Appl Soft Comput 132:1–14. https://doi.org/10.1016/j.asoc.2022.109849

    Article  Google Scholar 

  28. Chellaswamy C, Muthammal R, Geetha TS (2018) A new methodology for optimal rail track condition measurement using acceleration signals. Meas Sci Technol 29:075901. https://doi.org/10.1088/1361-6501/aabe48

    Article  Google Scholar 

  29. Mei S, Guan G, Wang Z, Wan S, He M, Feng DD (2015) Video summarization via minimum sparse reconstruction. Pattern Recogn Lett 48(2):522–533

    Article  Google Scholar 

  30. Yang C, Yuan J, Luo J (2012) Towards scalable summarization of consumer videos via sparse dictionary selection. IEEE Trans Multimedia 14(1):66–75

    Article  Google Scholar 

  31. Ma M, Mei S, Wan S, Wang Z, Feng D (2019) Video summarization via nonlinear sparse dictionary selection. IEEE Access 7:11763–11774

    Article  Google Scholar 

  32. Guan G, Wang Z, Lu S, Deng JD, Feng DD (2013) Keypoint based keyframe selection. IEEE Trans Circuits Syst Video Technol 23(4):729–734

    Article  Google Scholar 

  33. Demir M, Isil Bozma H (2015) Video summarization via segments summary graphs, IEEE International Conference on Computer Vision Workshops 19–25

  34. Hannane R, Elboushaki A, Afdel K (2016) Efficient video summarization based on motion sift-distribution histogram, IEEE 13th International Conference on Computer Graphics, Imaging and Visualization (CGiV) 312–317

  35. Hannane R, Elboushaki A, Afdel K (2018) Mskvs: adaptive mean shift-based keyframe extraction for video summarization and a new objective verification approach. J Vis Commun Image Represent 55:179–200

    Article  Google Scholar 

  36. Mundur P, Rao Y, Yesha Y (2006) Keyframe-based video summarization using delaunay clustering. Int J Digit Libr 6(2):219–232

    Article  Google Scholar 

  37. Kannappan S, Liu Y, Tiddeman B (2019) Dfp-alc: automatic video summarization using distinct frame patch index and appearance based linear clustering. Pattern Recogn Lett 120:8–16

    Article  Google Scholar 

  38. Cirne MVM, Pedrini H (2013) A video summarization method based on spectral clustering. Iberoamerican Congress on Pattern Recognition. Springer, pp 479–486

    Google Scholar 

  39. Yunjae J, Donghyeon C, Dahun K, Sanghyun W, Kweon IS (2019) Discriminative feature learning for unsupervised video summarization. The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19), 8537–8544

  40. Zhao M, Yang J, Zhang J, Wang S (2022) Aggregated graph convolutional networks for aspect-based sentiment classification. Information Sciences 600:73–93. https://doi.org/10.1016/j.ins.2022.03.082

Download references

Funding

No funding received by the authors.

Author information

Authors and Affiliations

  1. Department of Artificial Intelligence & Data Science, Anand Institute of Higher Technology, Chennai, Tamil Nadu, 603103, India

    R. Deepa

  2. Department of Computer Science and Engineering, College of Engineering, Guindy, Anna University, Chennai, Tamil Nadu, 600025, India

    T. Sree Sharmila

  3. Department of Electronics and Communication Engineering, St. Joseph’s College of Engineering, OMR, Chennai, 600119, India

    R. Niruban

Authors
  1. R. Deepa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. T. Sree Sharmila
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. R. Niruban
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to R. Deepa.

Ethics declarations

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Deepa, R., Sree Sharmila, T. & Niruban, R. Dynamic graph neural network-based computational paradigm for video summarization. Multimed Tools Appl 83, 51227–51250 (2024). https://doi.org/10.1007/s11042-023-17412-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-17412-4

Keywords

Search

Navigation