Abstract
A guiding robot aims to effectively bring people to and from specific places within environments that are possibly unknown to them. During this operation the robot should be able to detect and track the accompanied person, trying never to lose sight of her/him. A solution to minimize this event is to use an omnidirectional camera: its \(360^{\circ }\) Field of View (FoV) guarantees that any framed object cannot leave the FoV if not occluded or very far from the sensor. However, the acquired panoramic videos introduce new challenges in perception tasks such as people detection and tracking, including the large size of the images to be processed, the distortion effects introduced by the cylindrical projection and the periodic nature of panoramic images. In this paper, we propose a set of targeted methods that allow to effectively adapt to panoramic videos a standard people detection and tracking pipeline originally designed for perspective cameras. Our methods have been implemented and tested inside a deep learning-based people detection and tracking framework with a commercial \(360^{\circ }\) camera. Experiments performed on datasets specifically acquired for guiding robot applications and on a real service robot show the effectiveness of the proposed approach over other state-of-the-art systems. We release with this paper the acquired and annotated datasets and the open-source implementation of our method.
Alberto Bacchin and Filippo Berno: The authors equally contribute to this paper.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Burgard, W., Cremers, A.B., Fox, D., Hähnel, D., Lakemeyer, G., Schulz, D., Steiner, W., Thrun, S.: Experiences with an interactive museum tour-guide robot. Art. Intell. 114(1), 3–55 (1999)
Koide, K., Miura, J., Menegatti, E.: Monocular person tracking and identification with on-line deep feature selection for person following robots. Rob. Auton. Syst. 124 (2020)
Cao, Z., Hidalgo, G., Simon, T., Wei, S., Sheikh, Y.: Openpose: realtime multi-person 2d pose estimation using part affinity fields. IEEE Trans. Pattern Anal. Mach. Intell. 43(01), 172–186 (2021)
Aguilar, W.G., Luna, M.A., Moya, J.F., Abad, V., Ruiz, H., Parra, H., Lopez, W.: Cascade classifiers and saliency maps based people detection. In: De Paolis, L.T., Bourdot, P., Mongelli, A. (eds.) Augmented Reality, Virtual Reality, and Computer Graphics, pp. 501–510. Springer International Publishing, Cham (2017)
Geronimo, D., Sappa, A., López, A., Ponsa, D.: Pedestrian detection using adaboost learning of features and vehicle pitch estimation. In Proceedings of the IASTED Int. Conf. on Visualization, Imaging and Image Processing, pp. 400–4005 (2006)
Ji, X., Fang, Q., Dong, J., Shuai, Q., Jiang, W., Zhou, X.: A survey on monocular 3d human pose estimation. Virtual Reality Intell. Hardware 2(6), 471–500 (2020)
Zhao, X., Li, W., Zhang, Y., Gulliver, T.A., Chang, S., Feng, Z.: A faster RCNN-based pedestrian detection system. In: 2016 IEEE 84th Vehicular Technology Conference (VTC-Fall), pp. 1–5 (2016)
Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 28 (2015)
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779–788 (2016)
Ahmad, M., Ahmed, I., Adnan, A.: Overhead view person detection using yolo. In: IEEE 10th Annual Ubiquitous Computing. Electronics Mobile Communication Conference (UEMCON), vol. 2019, pp. 0627–0633 (2019)
Sharma, S., Ansari, J.A., Krishna Murthy, J., Madhava Krishna, K.: Beyond pixels: Leveraging geometry and shape cues for online multi-object tracking. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 3508–3515 (2018)
Niu, Y., Xu, Z., Xu, E., Li, G., Huo, Y., Sun, W.: Monocular pedestrian 3d localization for social distance monitoring. Sensors 21 (2021)
Kreiss, S., Bertoni, L., Alahi, A.: OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association. IEEE Trans. Intell. Transp. Syst. 1–14 (2021, March)
Bertoni, L., Kreiss, S., Alahi, A.: Monoloco: Monocular 3d pedestrian localization and uncertainty estimation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6860–6870 (2019)
Kollmitz, M., Eitel, A., Vasquez, A., Burgard, W.: Deep 3d perception of people and their mobility aids. Rob. Auton. Syst. 114, 29–40 (2019)
Munaro, M., Menegatti, E.: Fast RGB-D people tracking for service robots. Auton. Robots 37 (2014)
Zheng, K., Wu, F., Chen, X.: Laser-based people detection and obstacle avoidance for a hospital transport robot. Sensors 21(3) (2021)
Choi, W., Savarese, S.: Multiple target tracking in world coordinate with single, minimally calibrated camera. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) Computer Vision—ECCV 2010, pp. 553–567. Heidelberg, Springer, Berlin Heidelberg, Berlin (2010)
Ardiyanto, I., Miura, J.: Partial least squares-based human upper body orientation estimation with combined detection and tracking. Image Vis. Comput. 32(11), 904–915 (2014)
Liu, Y., Yin, J., Yu, D., Zhao, S., Shen, J.: Multiple people tracking with articulation detection and stitching strategy. Neurocomputing 386, 18–29 (2020)
Ondrúška, P., Posner, I.: Deep tracking: seeing beyond seeing using recurrent neural networks. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. AAAI’16, AAAI Press, pp. 3361–3367 (2016)
Thaler, M., Bailer, W.: Real-time person detection and tracking in panoramic video. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1027–1032 (2013)
Tai, K.C., Tang, C.W.: Siamese networks based people tracking for 360-degree videos with equi-angular cubemap format. In: 2020 IEEE International Conference on Consumer Electronics—Taiwan (ICCE-Taiwan), pp. 1–2 (2020)
Li, J., Zang, Z., **e, H., Wang, G.: Implementation of person tracking system in panorama based on personalized distribution. In: 2020 39th Chinese Control Conference (CCC), pp. 7123–7128 (2020)
Shere, M., Kim, H., Hilton, A.: 3d multi person tracking with dual \(360^{\circ }\) cameras. In: 2020 IEEE International Conference on Image Processing (ICIP), pp. 2765–2769 (2020)
Yang, F., Li, F., Wu, Y., Sakti, S., Nakamura, S.: Using panoramic videos for multi-person localization and tracking in a 3d panoramic coordinate. In: ICASSP 2020—2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1863–1867 (2020)
Wan, E., Van Der Merwe, R.: The unscented kalman filter for nonlinear estimation. In: Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No.00EX373), pp. 153–158 (2000)
Nghiem, A.T., Bremond, F., Thonnat, M., Valentin, V.: Etiseo, performance evaluation for video surveillance systems. In: 2007 IEEE Conference on Advanced Video and Signal Based Surveillance, pp. 476–481 (2007)
Vandyke, M., Schwartz, J., Hall, C.: Unscented kalman filtering for spacecraft attitude state and parameter estimation. Adv. Astronaut. Sci. 119 (2004)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)
Acknowledgements
This research was partially supported by MIUR (Italian Minister for Education) under the initiative “Departments of Excellence” (Law 232/2016).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Bacchin, A., Berno, F., Menegatti, E., Pretto, A. (2023). People Tracking in Panoramic Video for Guiding Robots. In: Petrovic, I., Menegatti, E., Marković, I. (eds) Intelligent Autonomous Systems 17. IAS 2022. Lecture Notes in Networks and Systems, vol 577. Springer, Cham. https://doi.org/10.1007/978-3-031-22216-0_28
Download citation
DOI: https://doi.org/10.1007/978-3-031-22216-0_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-22215-3
Online ISBN: 978-3-031-22216-0
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)