SpringerLink
Log in

Semantic Guided Attention for Weakly Supervised Group Activity Recognition

  • Conference paper
  • First Online:
Image and Graphics Technologies and Applications (IGTA 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1910))

Included in the following conference series:

  • 295 Accesses

Abstract

The objective of group activity recognition is to identify behaviors performed by multiple individuals within a given scene. However, current weakly supervised approaches often rely on object detectors or use self-attention mechanisms. The former approach is susceptible to background clutter and entails high computational costs, while the latter method learns weights from the input video and assigns them to key targets which is not reliable enough to find the key person. To address these limitations, we present a novel weakly supervised framework. Our proposed framework eliminates the need for ground-truth bounding boxes or object detectors. Meanwhile, it incorporates the semantics of individual action labels to replace self-attention to guide the learning process, enabling the extraction of more sophisticated semantic features relevant to activity. This approach also explores the interactions to promote group activity classification. Experimental results demonstrate that our method achieves state-of-the-art performances on both volleyball and collective datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Wang, L., Li, W., Li, W., Van Gool, L.: Appearance-and-relation networks for video classification. In: CVPR, pp. 1430–1439 (2018)

    Google Scholar 

  2. Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: NIPS, pp. 568–576 (2014)

    Google Scholar 

  3. Gan, C., Wang, N., Yang, Y., Yeung, D.-Y., Hauptmann, A.G.: DevNet: a deep event network for multimedia event detection and evidence recounting. In: CVPR, pp. 2568–2577 (2015)

    Google Scholar 

  4. Carreira, J., Zisserman, A.: Quo vadis, action recognition? A new model and the kinetics dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

    Google Scholar 

  5. Fan, L., Huang, W., Gan, C., Ermon, S., Gong, B., Huang, J.: End-to-end learning of motion representation for video understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

    Google Scholar 

  6. Girdhar, R., Carreira, J., Doersch, C., Zisserman, A.: Video action transformer network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 244–253 (2019)

    Google Scholar 

  7. Kwon, H., Kim, M., Kwak, S., Cho, M.: MotionSqueeze: neural motion feature learning for video understanding. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12361, pp. 345–362. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58517-4_21

    Chapter  Google Scholar 

  8. Lin, J., Gan, C., Han, S.: TSM: temporal shift module for efficient video understanding. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 7083–7093 (2019)

    Google Scholar 

  9. Piergiovanni, A.J., Ryoo, M.S.: Representation flow for action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9945–9953 (2019)

    Google Scholar 

  10. Wang, L., **ong, Y., Zhe Wang, Yu., Qiao, D.L., Tang, X., Gool, L.: Temporal segment networks: towards good practices for deep action recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 20–36. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_2

    Chapter  Google Scholar 

  11. Ibrahim, M.S., Muralidharan, S., Deng, Z., Vahdat, A., Mori, G.: A hierarchical deep temporal model for group activity recognition. In: CVPR (2016)

    Google Scholar 

  12. Tang, Y., Wang, Z., Li, P., Lu, J., Yang, M., Zhou, J.: Mining semantics-preserving attention for group activity recognition. In: ACM MM (2018)

    Google Scholar 

  13. Tang, J., Shu, X., Yan, R., Zhang, L.: Coherence constrained graph LSTM for group activity recognition. TPAMI 44(2), 636–647 (2019)

    Article  Google Scholar 

  14. Shu, T., Todorovic, S., Zhu, S.C.: CERN: confidence-energy recurrent network for group activity recognition. In: CVPR (2017)

    Google Scholar 

  15. Wang, M., Ni, B., Yang, X.: Recurrent modeling of interaction context for collective activity recognition. In: CVPR (2017)

    Google Scholar 

  16. Azar, S.M., Atigh, M.G., Nickabadi, A., Alahi, A.: Convolutional relational machine for group activity recognition. In: CVPR (2019)

    Google Scholar 

  17. Bagautdinov, T., Alahi, A., Fleuret, F., Fua, P., Savarese, S.: Social scene understanding: end-to-end multi-person action localization and collective activity recognition. In: CVPR (2017)

    Google Scholar 

  18. Wu, J., Wang, L., Wang, L., Guo, J., Wu, G.: Learning actor relation graphs for group activity recognition. In: CVPR (2019)

    Google Scholar 

  19. Yan, R., **e, L., Tang, J., Shu, X., Tian, Qi.: Social adaptive module for weakly-supervised group activity recognition. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VIII, pp. 208–224. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58598-3_13

    Chapter  Google Scholar 

  20. Zhang, P., Tang, Y., Hu, J.-F., Zheng, W.-S.: Fast collective activity recognition under weak supervision. IEEE Trans. Image Process. 29, 29–43 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  21. Kim, D., Lee, J., Cho, M., Kwak, S.: Detector-free weakly supervised group activity recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20083–20093 (2022)

    Google Scholar 

  22. Zhang, D., Daniel, G.P., Bengio, S., et al.: Modeling individual and group actions in meetings: a two-layer HMM framework. In: Proceedings of the CVPR Workshops, p. 117 (2004)

    Google Scholar 

  23. Dai, P., Di, H., Dong, L., et al.: Group interaction analysis in dynamic context. IEEE Trans. Syst. Man Cybern. 38(1), 275–282 (2008)

    Article  Google Scholar 

  24. Lan, T., Wang, Y., Yang, W., et al.: Beyond actions: discriminative models for contextual group activities. In: Proceedings of the NIPS, pp. 1216−1224 (2010)

    Google Scholar 

  25. Kaneko, T., Shimosaka, M., Odashima, S., et al.: Consistent collective activity recognition with fully connected CRFs. In: Proceedings of the ICPR, pp. 2792−2795 (2012)

    Google Scholar 

  26. Amer, M.R., Todorovic, S., Fern, A., et al.: Monte Carlo tree search for scheduling activity recognition. In: Proceedings of the ICCV, pp. 1353−1360 (2013)

    Google Scholar 

  27. Amer, M.R., Lei, P., Todorovic, S.: HiRF: hierarchical random field for collective activity recognition in videos. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 572–585. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10599-4_37

    Chapter  Google Scholar 

  28. Qi, M., Qin, J., Li, A., Wang, Y., Luo, J., Gool, L.: StagNet: an attentive semantic RNN for group activity recognition. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11214, pp. 104–120. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01249-6_7

    Chapter  Google Scholar 

  29. Ibrahim, M.S., Mori, G.: Hierarchical relational networks for group activity recognition and retrieval. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8–14, 2018, Proceedings, Part III, pp. 742–758. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_44

    Chapter  Google Scholar 

  30. Hu, G., Cui, B., He, Y., et al.: Progressive relation learning for group activity recognition. In: Proceedings of the CVPR, pp. 977−986 (2020)

    Google Scholar 

  31. Xu, D., Fu, H., Wu, L., et al.: Group activity recognition by using effective multiple modality relation representation with temporalspatial attention. IEEE Access 8, 65689–65698 (2020)

    Article  Google Scholar 

  32. Li, X., Chuah, M.C.: SBGAR: semantics based group activity recognition. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2895–2904 (2017). https://doi.org/10.1109/ICCV.2017.313

  33. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)

    Google Scholar 

  34. Sun, D., Yang, X., Liu, M.-Y., Kautz, J.: PWC-Net: CNNs for optical flow using pyramid, war**, and cost volume. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8934–8943 (2018)

    Google Scholar 

  35. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: NeurIPS, pp. 3111–3119 (2013)

    Google Scholar 

  36. Kim,J.-H., On, K.-W., Lim, W., Kim, J., Ha, J.-W., Zhang, B.-T.: Hadamard product for low-rank bilinear pooling. ar**v preprint ar**v:1610.04325 (2016)

  37. Li, Y., Tarlow, D., Brockschmidt, M., Zemel, R.: Gated graph sequence neural networks. ar**v preprint ar**v:1511.05493 (2015)

  38. Chen, T., Lin, L., Chen, R., Wu, Y., Luo, X.: Knowledge-embedded representation learning for fine-grained image recognition. In: Proceedings of International Joint Conference on Artificial Intelligence (IJCAI), pp. 627–634 (2018)

    Google Scholar 

  39. Wang, Z., Chen, T., Ren, J., Yu, W., Cheng, H., Lin, L.: Deep reasoning with knowledge graph for social relationship understanding. In: Proceedings of International Joint Conference on Artificial Intelligence (IJCAI), pp. 2021–2018 (2018)

    Google Scholar 

  40. Chen, T., Yu, W., Chen, R., Lin, L.: Knowledge-embedded routing network for scene graph generation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  41. Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the International Conference on Machine Learning (ICML) (2010)

    Google Scholar 

  42. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the International Conference on Learning Representations (ICLR) (2015)

    Google Scholar 

  43. Yan, R., **e, L., Tang, J., Shu, X., Tian, Q.: Higcin: hierarchical graph-based cross inference network for group activity recognition. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 45(6), 6955–6968 (2020)

    Article  Google Scholar 

  44. Gavrilyuk, K., Sanford, R., Javan, M., Snoek, C.G.M.: Actor-transformers for group activity recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 839–848 (2020)

    Google Scholar 

  45. Pramono, R.R.A., Chen, Y.T., Fang, W.H.: Empowering relational network by self-attention augmented conditional random fields for group activity recognition. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I, pp. 71–90. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_5

    Chapter  Google Scholar 

  46. Yuan, H., Ni, D., Wang, M.: Spatio-temporal dynamic inference network for group activity recognition. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 7476–7485 (2021)

    Google Scholar 

  47. Yuan, H., Ni, D.: Learning visual context for group activity recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), vol. 35, pp. 3261–3269 (2021)

    Google Scholar 

  48. Li, S., et al.: GroupFormer: group activity recognition with clustered spatial-temporal transformer. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 13668–13677, 2021

    Google Scholar 

  49. Lu, L., Lu, Y., Yu, R., Di, H., Zhang, L., Wang, S.: GAIM: graph attention interaction model for collective activity recognition. IEEE Trans. Multimedia 22(2), 524–539 (2019)

    Article  Google Scholar 

  50. Ehsanpour, M., Abedin, A., Saleh, F., Shi, J., Reid, I., Rezatofighi, H.: Joint learning of social groups, individuals action and sub-group activities in videos. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12354, pp. 177–195. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58545-7_11

    Chapter  Google Scholar 

  51. Wu, L., Lang, X., **ang, Y., Chen, C., Li, Z., Wang, Z.: Active spatial positions based hierarchical relation inference for group activity recognition. IEEE Trans. Circ. Syst. Video Technol. 33(6), 2839–2851 (2023). https://doi.org/10.1109/TCSVT.2022.3228731

    Article  Google Scholar 

Download references

Acknowledgments

This work has been supported by the National Natural Science Foundation of China under Grants No. 62106011, 62336010, 61976010, and 62106010. We gratefully acknowledge their financial support, which has enabled us to conduct this research.

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Bei**g University of Technology, Bei**g, 100124, China

    Tingting Liu, Ye **ang, Lifang Wu & Ge Shi

Authors
  1. Tingting Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ye **ang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lifang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ge Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lifang Wu .

Editor information

Editors and Affiliations

  1. Bei**g Institute of Technology, Bei**g, China

    Wang Yongtian

  2. Bei**g University of Technology, Bei**g, China

    Wu Lifang

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Liu, T., **ang, Y., Wu, L., Shi, G. (2023). Semantic Guided Attention for Weakly Supervised Group Activity Recognition. In: Yongtian, W., Lifang, W. (eds) Image and Graphics Technologies and Applications. IGTA 2023. Communications in Computer and Information Science, vol 1910. Springer, Singapore. https://doi.org/10.1007/978-981-99-7549-5_17

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-7549-5_17

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-7548-8

  • Online ISBN: 978-981-99-7549-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation