SpringerLink
Log in

Online learning for 3D LiDAR-based human detection: experimental analysis of point cloud clustering and classification methods

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

This paper presents a system for online learning of human classifiers by mobile service robots using 3D LiDAR sensors, and its experimental evaluation in a large indoor public space. The learning framework requires a minimal set of labelled samples (e.g. one or several samples) to initialise a classifier. The classifier is then retrained iteratively during operation of the robot. New training samples are generated automatically using multi-target tracking and a pair of “experts” to estimate false negatives and false positives. Both classification and tracking utilise an efficient real-time clustering algorithm for segmentation of 3D point cloud data. We also introduce a new feature to improve human classification in sparse, long-range point clouds. We provide an extensive evaluation of our the framework using a 3D LiDAR dataset of people moving in a large indoor public space, which is made available to the research community. The experiments demonstrate the influence of the system components and improved classification of humans compared to the state-of-the-art.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. http://www.cvpapers.com/datasets.html.

  2. https://github.com/LCAS/online_learning.

  3. http://lcas.lincoln.ac.uk/wp/research/data-sets-software/l-cas-3d-point-cloud-people-dataset/.

  4. http://www.flobot.eu.

  5. Due to privacy issues, currently panoramic images are not included in the dataset.

  6. http://pointclouds.org/documentation/tutorials/pcd_file_format.php.

  7. Although the “group” samples are not used in this paper, these annotations are included in our dataset to be used by other researchers for group tracking or other applications.

  8. The data is available at: https://github.com/LCAS/cloud_annotation_tool.

  9. Dataset collection is also in progress at University of Lincoln, Czech Technical University in Prague, and University of Technology of Belfort-Montbéliard.

References

  • Arras, K. O., Mozos, O. M., & Burgard, W. (2007). Using boosted features for the detection of people in 2d range data. In Proceedings of the 2007 IEEE international conference on robotics and automation (ICRA) (pp. 3402–3407).

  • Bellotto, N., & Hu, H. (2009). Multisensor-based human detection and tracking for mobile service robots. IEEE Transactions on Systems, Man, and Cybernetics Part B, 39(1), 167–181.

    Article  Google Scholar 

  • Bellotto, N., & Hu, H. (2010). Computationally efficient solutions for tracking people with a mobile robot: An experimental evaluation of bayesian filters. Autonomous Robots, 28, 425–438.

    Article  Google Scholar 

  • Beyer, L., Hermans, A., Linder, T., Arras, K. O., & Leibe, B. (2018). Deep person detection in two-dimensional range data. IEEE Robotics and Automation Letters, 3(3), 2726–2733.

    Article  Google Scholar 

  • Bianco, S., Ciocca, G., Napoletano, P., & Schettini, R. (2015). An interactive tool for manual, semi-automatic and automatic video annotation. Computer Vision and Image Understanding, 131, 88–99.

    Article  Google Scholar 

  • Bogoslavskyi, I., & Stachniss, C. (2016) Fast range image-based segmentation of sparse 3d laser scans for online operation. In Proceedings of the 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 163–169).

  • Chang, C. C., & Lin, C. J. (2011). LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2, 1–27.

    Article  Google Scholar 

  • Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297.

    MATH  Google Scholar 

  • Dequaire, J., Ondruska, P., Rao, D., Wang, D., & Posner, I. (2017). Deep tracking in the wild: End-to-end tracking using recurrent neural networks. International Journal of Robotics Research (IJRR), 37, 1–21.

    Google Scholar 

  • Deuge, M.D., Quadros, A., Hung, C., & Douillard, B. (2013). Unsupervised feature learning for classification of outdoor 3d scans. In Proceedings of the Australasian conference on robotics and automation (ACRA).

  • Dewan, A., Caselitz, T., Tipaldi, G.D., & Burgard, W. (2016). Motion-based detection and tracking in 3D LiDAR scans. In Proceedings of the 2016 IEEE international conference on robotics and automation (ICRA) (pp. 4508–4513).

  • Dondrup, C., Bellotto, N., Jovan, F., & Hanheide, M. (2015). Real-time multisensor people tracking for human-robot spatial interaction. In Proceedings of the 2015 IEEE international conference on robotics and automation (ICRA), workshop on machine learning for social robotics.

  • Douillard, B., Underwood, J. P., Kuntz, N., Vlaskine, V., Quadros, A. J., Morton, P., & Frenkel, A. (2011). On the segmentation of 3d LIDAR point clouds. In Proceedings of the 2011 IEEE international conference on robotics and automation (ICRA) (pp. 2798–2805).

  • Everingham, M., Gool, L. J. V., Williams, C. K. I., Winn, J. M., & Zisserman, A. (2010). The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 88, 303–338.

    Article  Google Scholar 

  • Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving? the KITTI vision benchmark suite. In Proceedings of the 2012 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 3354–3361).

  • González, A., Villalonga, G., Xu, J., Vázquez, D., Amores, J., & López, A.M. (2015). Multiview random forest of local experts combining RGB and LIDAR data for pedestrian detection. In Proceedings of the 2015 IEEE intelligent vehicles symposium (IV) (pp. 356–361).

  • Häselich, M., Jöbgen, B., Wojke, N., Hedrich, J., & Paulus, D. (2014). Confidence-based pedestrian tracking in unstructured environments using 3d laser distance measurements. In Proceedings of the 2014 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 4118–4123).

  • Jafari, O. H., Mitzel, D., & Leibe, B. (2014). Real-time RGB-D based people detection and tracking for mobile robots and head-worn cameras. In Proceedings of the 2014 IEEE international conference on robotics and automation (ICRA) (pp. 5636–5643).

  • Julier, S. J., & Uhlmann, J. K. (2004). Unscented filtering and nonlinear estimation. Proceedings of the IEEE, 92(3), 401–422.

    Article  Google Scholar 

  • Kalal, Z., Mikolajczyk, K., & Matas, J. (2012). Tracking-learning-detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34, 1409–1422.

    Article  Google Scholar 

  • Kavasidis, I., Palazzo, S., Salvo, R. D., Giordano, D., & Spampinato, C. (2012). A semi-automatic tool for detection and tracking ground truth generation in videos. In Proceedings of the 1st international workshop on visual interfaces for ground truth collection in computer vision applications.

  • Keerthi, S. S., & Lin, C. J. (2003). Asymptotic behaviors of support vector machines with gaussian kernel. Neural Computation, 15(7), 1667–1689.

    Article  Google Scholar 

  • Kidono, K., Miyasaka, T., Watanabe, A., Naito, T., & Miura, J. (2011). Pedestrian recognition using high-definition LIDAR. In Proceedings of the 2011 IEEE intelligent vehicles symposium (IV) (pp. 405–410).

  • Krajnik, T., Fentanes, J. P., Santos, J. M., & Duckett, T. (2017). Fremen: Frequency map enhancement for long-term mobile robot autonomy in changing environments. IEEE Transactions on Robotics, 33, 964–977.

    Article  Google Scholar 

  • Leigh, A., Pineau, J., Olmedo, N. A., & Zhang, H. (2015). Person tracking and following with 2d laser scanners. In Proceedings of the 2015 IEEE international conference on robotics and automation (ICRA) (pp. 726–733).

  • Li, K., Wang, X., Xu, Y., & Wang, J. (2016). Density enhancement-based long-range pedestrian detection using 3-d range data. IEEE Transactions on Intelligent Transportation Systems, 17, 1368–1380.

    Article  Google Scholar 

  • Li, X. R., & Jilkov, V. P. (2003). Survey of maneuvering target tracking. part i: Dynamic models. IEEE Transactions on Aerospace and Electronic Systems, 39(4), 1333–1364.

    Article  Google Scholar 

  • Linder, T., Breuers, S., Leibe, B., & Arras, K. O. (2016). On multi-modal people tracking from mobile platforms in very crowded and dynamic environments. In Proceedings of the 2016 IEEE international conference on robotics and automation (ICRA) (pp. 5512–5519).

  • Luber, M., & Arras, K. O. (2013). Multi-hypothesis social grou** and tracking for mobile robots. In Proceedings of Robotics: Science and Systems.

  • Mohler, B. J., Thompson, W. B., Creem-Regehr, S. H., Pick, H. L., & Warren, W. H. (2007). Visual flow influences gait transition speed and preferred walking speed. Experimental Brain Research, 181(2), 221–228.

    Article  Google Scholar 

  • Moosmann, F., Pink, O., & Stiller, C. (2009). Segmentation of 3d lidar data in non-flat urban environments using a local convexity criterion. In Proceedings of the 2009 IEEE intelligent vehicles symposium (IV) (pp. 1931–0587).

  • Munaro, M., & Menegatti, E. (2014). Fast RGB-D people tracking for service robots. Autonomous Robots, 37, 227–242.

    Article  Google Scholar 

  • Navarro-Serment, L. E., Mertz, C., & Hebert, M. (2009). Pedestrian detection and tracking using three-dimensional ladar data. In Proceedings of the 7th conference on field and service robotics (FSR) (pp. 103–112).

  • Premebida, C., Carreira, J., Batista, J., & Nunes, U. (2014). Pedestrian detection combining RGB and dense LIDAR data. In Proceedings of the 2014 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 4112–4117).

  • Quigley, M., Conley, K., Gerkey, B.P., Faust, J., Foote, T., Leibs, J., Wheeler, R., & Ng, A.Y. (2009). ROS: An open-source robot operating system. In DICRA workshop on open source software.

  • Read, J., Bifet, A., Pfahringer, B., & Holmes, G. (2012). Batch-incremental versus instance-incremental learning in dynamic and evolving data. In Proceedings of the eleventh international symposium on intelligent data analysis (IDA 2012) (pp. 313–323).

  • Rusu, R.B. (2009). Semantic 3D object maps for everyday manipulation in human living environments. Ph.D. thesis, Computer Science department, Technische Universitaet Muenchen, Germany.

  • Rusu, R.B., & Cousins, S. (2011). 3D is here: Point Cloud Library (PCL). In Proceedings of the 2011 IEEE international conference on robotics and automation (ICRA).

  • Shackleton, J., Voorst, B. V., & Hesch, J. A. (2010). Tracking people with a 360-degree lidar. In Proceedings of the Seventh IEEE international conference on advanced video and signal based surveillance (AVSS) (pp. 420–426).

  • Spinello, L., & Arras, K.O. (2011). People detection in RGB-D data. In Proceedings of the 2011 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 3838–3843).

  • Spinello, L., Luber, M., & Arras, K.O. (2011). Tracking people in 3d using a bottom-up top-down detector. In Proceedings of the 2011 IEEE international conference on robotics and automation (ICRA) (pp. 1304–1310).

  • Sun, L., Yan, Z., Mellado, S.M., Hanheide, M., & Duckett, T. (2018). 3DOF pedestrian trajectory prediction learned from long-term autonomous mobile robot deployment data. In In proceedings of the 2018 IEEE international conference on robotics and automation (ICRA). Brisbane, Australia.

  • Sun, L., Yan, Z., Zaganidis, A., Zhao, C., & Duckett, T. (2018). Recurrent-octomap: Learning state-based map refinement for long-term semantic map** with 3-d-lidar data. IEEE Robotics and Automation Letters, 3(4), 3749–3756.

    Article  Google Scholar 

  • Teichman, A., Levinson, J., & Thrun, S. (2011). Towards 3d object recognition via classification of arbitrary object tracks. In Proceedings of the 2011 IEEE international conference on robotics and automation (ICRA) (pp. 4034–4041).

  • Teichman, A., & Thrun, S. (2012). Tracking-based semi-supervised learning. International Journal of Robotics Research (IJRR), 31(7), 804–818.

    Article  Google Scholar 

  • Vintr, T., Yan, Z., Duckett, T., & Krajnik, T. (2019). Spatio-temporal representation for long-term anticipation of human presence in service robotics. In Proceedings of the 2018 IEEE international conference on robotics and automation (ICRA). Montreal, Canada.

  • Wang, D. Z., & Posner, I. (2015). Voting for voting in online point cloud object detection. In Proceedings of Robotics: Science and Systems.

  • Yan, Z., Duckett, T., & Bellotto, N. (2017). Online learning for human classification in 3D LiDAR-based tracking. In In Proceedings of the 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS). Vancouver, Canada.

  • Yan, Z., Sun, L., Duckett, T., & Bellotto, N. (2018). Multisensor online transfer learning for 3d lidar-based human detection with a mobile robot. In Proceedings of the 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS). Madrid, Spain.

  • Zermas, D., Izzat, I., & Papanikolopoulos, N. (2017). Fast segmentation of 3d point clouds: A paradigm on lidar data for autonomous vehicle applications. In Proceedings of the 2018 IEEE international conference on robotics and automation (ICRA). Singapore.

  • Zhou, Y., & Tuzel, O. (2018). Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the 2018 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 4490–4499).

Download references

Author information

Authors and Affiliations

  1. Distributed Artificial Intelligence and Knowledge Laboratory (CIAD), University of Technology of Belfort-Montbéliard (UTBM), Belfort, France

    Zhi Yan

  2. Lincoln Centre for Autonomous Systems (L-CAS), University of Lincoln, Lincoln, UK

    Tom Duckett & Nicola Bellotto

Authors
  1. Zhi Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tom Duckett
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicola Bellotto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhi Yan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under Grant agreement No 645376 (FLOBOT).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yan, Z., Duckett, T. & Bellotto, N. Online learning for 3D LiDAR-based human detection: experimental analysis of point cloud clustering and classification methods. Auton Robot 44, 147–164 (2020). https://doi.org/10.1007/s10514-019-09883-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-019-09883-y

Keywords

Mathematics Subject Classification

Search

Navigation