Abstract
Temporal action localization in untrimmed long videos is an important yet challenging problem. The temporal ambiguity and the intra-class variations of temporal structure of actions make existing methods far from being satisfactory. In this paper, we propose a novel framework which firstly models each action clip based on its temporal evolution, and then adopts a deep multiple instance learning (MIL) network for jointly classifying action clips and refining their temporal boundaries. The proposed network utilizes a MIL scheme to make clip-level decisions based on temporal-instance-level decisions. Besides, a temporal smoothness constraint is introduced into the multi-task loss. We evaluate our framework on THUMOS Challenge 2014 benchmark and the experimental results show that it achieves considerable improvements as compared to the state-of-the-art methods. The performance gain is especially remarkable under precise localization with high tIoU thresholds, e.g. mAP@tIoU=0.5 is improved from 31.0% to 35.0%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Cinbis, R.G., Verbeek, J., Schmid, C.: Weakly supervised object localization with multi-fold multiple instance learning. IEEE transactions on pattern analysis and machine intelligence 39(1), 189–203 (2017)
Dai, X., Singh, B., Zhang, G., Davis, L.S., Chen, Y.Q.: Temporal context network for activity localization in videos. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5727–5736. IEEE (2017)
Dietterich, T.G., Lathrop, R.H., Lozano-Pérez, T.: Solving the multiple instance problem with axis-parallel rectangles. Artif. Intell. 9(1–2), 31–71 (1997)
Gao, J., Yang, Z., Nevatia, R.: Cascaded boundary regression for temporal action detection. ar**v preprint ar**v:1705.01180 (2017)
Gao, J., Yang, Z., Sun, C., Chen, K., Nevatia, R.: Turn tap: Temporal unit regression network for temporal action proposals. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 3648–3656. IEEE (2017)
Jiang, Y.G., et al.: THUMOS challenge: action recognition with a large number of classes. http://crcv.ucf.edu/THUMOS14/ (2014)
Jolliffe, I.: Principal Component Analysis. International Encyclopedia of Statistical Science, pp. 1094–1096. Springer, New York (2011). https://doi.org/10.1007/978-3-642-04898-2_455
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. ar**v preprint ar**v:1412.6980 (2014)
Kraus, O.Z., Ba, J.L., Frey, B.J.: Classifying and segmenting microscopy images with deep multiple instance learning. Bioinformatics 32(12), i52–i59 (2016)
Mallya, A., Lazebnik, S.: Learning models for actions and person-object interactions with transfer to question answering. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 414–428. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_25
Montes, A., Salvador, A., Pascual, S., Giro-i Nieto, X.: Temporal activity detection in untrimmed videos with recurrent neural networks. ar**v preprint ar**v:1608.08128 (2016)
Shou, Z., Chan, J., Zareian, A., Miyazawa, K., Chang, S.F.: CDC: convolutional-de-convolutional networks for precise temporal action localization in untrimmed videos. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1417–1426. IEEE (2017)
Shou, Z., Wang, D., Chang, S.F.: Temporal action localization in untrimmed videos via multi-stage CNNs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1049–1058 (2016)
Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Advances in Neural Information Processing Systems, pp. 568–576 (2014)
Sultani, W., Chen, C., Shah, M.: Real-world anomaly detection in surveillance videos. Center for Research in Computer Vision (CRCV), University of Central Florida (UCF) (2018)
Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3D convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4489–4497 (2015)
Wang, H., Kläser, A., Schmid, C., Liu, C.L.: Action recognition by dense trajectories. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3169–3176. IEEE (2011)
Wang, H., Schmid, C.: Action recognition with improved trajectories. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3551–3558 (2013)
Wu, J., Yu, Y., Huang, C., Yu, K.: Deep multiple instance learning for image classification and auto-annotation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3460–3469 (2015)
Xu, H., Das, A., Saenko, K.: R-c3d: region convolutional 3D network for temporal activity detection. In: IEEE International Conference on Computer Vision (ICCV), pp. 5794–5803 (2017)
Yeung, S., Russakovsky, O., **, N., Andriluka, M., Mori, G., Fei-Fei, L.: Every moment counts: dense detailed labeling of actions in complex videos. Int. J. Comput. Vis. 126(2–4), 375–389 (2018)
Yun, K., Honorio, J., Chattopadhyay, D., Berg, T.L., Samaras, D.: Two-person interaction detection using body-pose features and multiple instance learning. In: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 28–35. IEEE (2012)
Zhao, Y., **ong, Y., Wang, L., Wu, Z., Tang, X., Lin, D.: Temporal action detection with structured segment networks. In: ICCV, October 2 (2017)
Acknowledgments
This work was supported in part by the National Nature Science Foundation of China under Grants 61672285; the work of **angbo Shu is supported by the National Natural Science Foundation of China (Grant No. 61702265), Natural Science Foundation of Jiangsu Province (Grant No. BK20170856), and CCF-Tencent Open Research Fund (PI: **angbo Shu).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Yang, M., Song, Y., Shu, X., Tang, J. (2019). Temporal Action Localization Based on Temporal Evolution Model and Multiple Instance Learning. In: Kompatsiaris, I., Huet, B., Mezaris, V., Gurrin, C., Cheng, WH., Vrochidis, S. (eds) MultiMedia Modeling. MMM 2019. Lecture Notes in Computer Science(), vol 11296. Springer, Cham. https://doi.org/10.1007/978-3-030-05716-9_28
Download citation
DOI: https://doi.org/10.1007/978-3-030-05716-9_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-05715-2
Online ISBN: 978-3-030-05716-9
eBook Packages: Computer ScienceComputer Science (R0)