Abstract
The use of 3D Deep Learning (DL) models for LiDAR data segmentation has attracted much interest in recent years. However, the generation of labeled point cloud data, which is a prerequisite for training DL models, is a highly resource-intensive exercise. Simulated LiDAR data, which are already labeled, provide a cost-effective alternative, but their efficacy and usefulness must be evaluated. This paper examines the role of simulated LiDAR point clouds in training DL models. A high-fidelity 3D terrain model representing the real environment is developed, and the in-house physics-based simulator “Limulator” is used to generate labeled point clouds through various realizations. The paper outlines a few major hypotheses to assess the usefulness of simulated data in training DL models. The hypotheses are designed to assess the role of simulated data alone or in combination with real data or by strategic boosting of minor classes in simulated data. Several experiments are carried out to test these hypotheses. An experiment involves training a DL model, PointCNN in this case, using various combinations of simulated and real LiDAR data and measuring its performance to segment the test data. Results show that training using simulated data alone can produce an overall accuracy (OA) of 89% and the weighted-averaged F1 score of 88.81%. It is further observed that training using a combination of simulated and real data can achieve accuracies comparable to when only a large quantity of real data is employed. Strategic boosting of minor classes in simulated data improves the accuracies of minor classes by up to 23% compared to only real data. Training a DL model using simulated data, due to the ease in its generation and positive impact on segmentation accuracy, can be highly beneficial in the use of DL for LiDAR data. The use of simulated data for training has the potential to minimize the resource-intensive exercise of develo** labeled real data.
This is a preview of subscription content, log in via an institution to check access.
Data Availability
The datasets generated during the study (simulated data) are not publicly available. However, the data can be available from the authors upon reasonable request. Furthermore, the secondary data (Real data) supporting the findings of this study are included in this published article.
References
Abu Alhaija, H., Mustikovela, S. K., Mescheder, L., Geiger, A., & Rother, C. (2018). Augmented reality meets computer vision: Efficient data generation for urban driving scenes. International Journal of Computer Vision, 126(9), 961–972. https://doi.org/10.1007/s11263-018-1070-x
Adams, H. (2023). Adams. Hexagon. https://hexagon.com/products/product-groups/computer-aided-engineering-software/adams. Accessed 4 June 2023
Bousmalis, K., Silberman, N., Dohan, D., Erhan, D., & Krishnan, D. (2017). Unsupervised pixel-level domain adaptation with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3722–3731)
Chen, C., Seff, A., Kornhauser, A., & **ao, J. (2015). Deepdriving: Learning affordance for direct perception in autonomous driving. In Proceedings of the IEEE international conference on computer vision (pp. 2722–2730)
Dayal, S., Goel, S., Lohani, B., Mittal, N., & Mishra, R. K. (2021). Comprehensive airborne laser scanning (ALS) simulation. Journal of the Indian Society of Remote Sensing, 49(7), 1603–1622.
de Melo, C. M., Rothrock, B., Gurram, P., Ulutan, O., & Manjunath, B. S. (2020). Vision-based gesture recognition in human-robot teams using synthetic data. In 2020 IEEE/RSJ International conference on intelligent robots and systems (IROS) (pp. 10278–10284). https://doi.org/10.1109/IROS45743.2020.9340728
de Melo, C. M., Torralba, A., Guibas, L., DiCarlo, J., Chellappa, R., & Hodgins, J. (2022). Next-generation deep learning based on simulators and synthetic data. Trends in Cognitive Sciences, 26(2), 174–187. https://doi.org/10.1016/j.tics.2021.11.008
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017). CARLA: An open urban driving simulator. In Conference on robot learning (pp. 1–16). PMLR
Fang, J., Yan, F., Zhao, T., Zhang, F., Zhou, D., Yang, R., et al. (2018). Simulating LIDAR point cloud for autonomous driving using real-world scenes and traffic flows. https://openreview.net/forum?id=rJMw747l_4
Fang, J., Zhou, D., Yan, F., Zhao, T., Zhang, F., Ma, Y., et al. (2020). Augmented lidar simulator for autonomous driving. IEEE Robotics and Automation Letters, 5(2), 1931–1938.
Fawaz, H. I., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2018). Data augmentation using synthetic data for time series classification with deep residual networks. ar**v preprint retrieved from ar**v:1808.02455
Foretellix. (2023). Foretellix—Automated driving systems testing tools for simulations. Foretellix. https://www.foretellix.com/. Accessed 4 June 2023
Gaidon, A., Wang, Q., Cabon, Y., & Vig, E. (2016). Virtual worlds as proxy for multi-object tracking analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4340–4349)
Gusmão, G. F., Barbosa, C. R. H., & Raposo, A. B. (2020). Development and validation of LiDAR sensor simulators based on parallel raycasting. Sensors, 20(24), 7186.
He, R., Sun, S., Yu, X., Xue, C., Zhang, W., Torr, P., et al. (2022). Is synthetic data from generative models ready for image recognition? ar**v preprint retrieved from ar**v:2210.07574
IPG, A. (2023). CarMaker|IPG Automotive. https://ipg-automotive.com/en/products-solutions/software/carmaker. Accessed 1 June 2023
Jaipuria, N., Zhang, X., Bhasin, R., Arafa, M., Chakravarty, P., Shrivastava, S., et al. (2020). Deflating dataset bias using synthetic data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (pp. 772–773)
Jordon, J., Yoon, J., & Van Der Schaar, M. (2018). PATE-GAN: Generating synthetic data with differential privacy guarantees. In International conference on learning representations
Khosla, C., & Saini, B. S. (2020). Enhancing performance of deep learning models with different data augmentation techniques: A survey. In 2020 International Conference on Intelligent Engineering and Management (ICIEM) (pp. 79–85). IEEE
Kumar, B., Pandey, G., Lohani, B., & Misra, S. C. (2019). A multi-faceted CNN architecture for automatic classification of mobile LiDAR data and an algorithm to reproduce point cloud samples for enhanced training. ISPRS Journal of Photogrammetry and Remote Sensing, 147, 80–89.
Li, Y., Bu, R., Sun, M., Wu, W., Di, X., & Chen, B. (2018). Pointcnn: Convolution on x-transformed points. Advances in neural information processing systems, 31
Lohani, B., & Mishra, R. K. (2007). Generating LiDAR data in laboratory: LiDAR simulator. Int. Arch. Photogramm. Remote Sens, 52(01), 264–269.
Manivasagam, S., Wang, S., Wong, K., Zeng, W., Sazanovich, M., Tan, S., et al. (2020). Lidarsim: Realistic lidar simulation by leveraging the real world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11167–11176).
Morse. (2023). Modular open robots simulation engine by morse-simulator. http://morse-simulator.github.io/. Accessed 4 June 2023
Movshovitz-Attias, Y., Kanade, T., & Sheikh, Y. (2016). How useful is photo-realistic rendering for visual learning? In Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8–10 and 15–16, 2016, Proceedings, Part III 14 (pp. 202–217). Springer.
Negrut, D., Serban, R., & Elmquist, A. (2020). Physics-based sensor models for virtual simulation of connected and autonomous vehicles. https://openreview.net/forum?id=rJMw747l_4
Nikolenko, S. I. (2021). Synthetic data for deep learning (Vol. 174). Springer International Publishing. https://doi.org/10.1007/978-3-030-75178-4
NVIDIA. (2023). NVIDIA DRIVE Sim Powered by Omniverse. NVIDIA. https://www.nvidia.com/en-us/self-driving-cars/simulation/. Accessed 4 June 2023
Park, T., Efros, A. A., Zhang, R., & Zhu, J.-Y. (2020). Contrastive learning for unpaired image-to-image translation. In A. Vedaldi, H. Bischof, T. Brox, & J.-M. Frahm (Eds.), Computer vision—ECCV 2020 (pp. 319–345). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-58545-7_19
Prakash, A., Boochoon, S., Brophy, M., Acuna, D., Cameracci, E., State, G., et al. (2019). Structured domain randomization: bridging the reality gap by context-aware synthetic data. In 2019 International Conference on Robotics and Automation (ICRA) (pp. 7249–7255). https://doi.org/10.1109/ICRA.2019.8794443
Ravuri, S., & Vinyals, O. (2019). Seeing is not necessarily believing: Limitations of biggans for data augmentation. International Conference on Learning Representations. https://openreview.net/forum?id=rJMw747l_4
Reitmann, S., Neumann, L., & Jung, B. (2021). BLAINDER—a blender AI add-on for generation of semantically labeled depth-sensing data. Sensors, 21(6), 2144.
Richter, S. R., AlHaija, H. A., & Koltun, V. (2022). Enhancing photorealism enhancement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2), 1700–1715.
Rong, G., Shin, B. H., Tabatabaee, H., Lu, Q., Lemke, S., Možeiko, M., et al. (2020). Lgsvl simulator: A high fidelity simulator for autonomous driving. In 2020 IEEE 23rd International conference on intelligent transportation systems (ITSC) (pp. 1–6). IEEE
Ros, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. M. (2016). The SYNTHIA Dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In 2016 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 3234–3243). Presented at the 2016 IEEE conference on computer vision and pattern recognition (CVPR). https://doi.org/10.1109/CVPR.2016.352
Sallab, A. E., Sobh, I., Zahran, M., & Essam, N. (2019a). LiDAR Sensor modeling and data augmentation with GANs for autonomous driving. ar**v preprint retrieved from ar**v:1905.07290
Sallab, A. E., Sobh, I., Zahran, M., & Shawky, M. (2019b). Unsupervised neural sensor models for synthetic lidar data augmentation. ar**v preprint retrieved from ar**v:1911.10575
Shafaei, A., Little, J., & Schmidt, M. (2016). Play and learn: Using video games to train computer vision models. In Procedings of the British machine vision conference 2016 (pp. 26.1–26.13). Presented at the British machine vision conference 2016, York, UK: British Machine Vision Association. https://doi.org/10.5244/C.30.26
Shah, S., Dey, D., Lovett, C., & Kapoor, A. (2018). Airsim: High-fidelity visual and physical simulation for autonomous vehicles. Field and service robotics (pp. 621–635). Cham: Springer.
Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., & Webb, R. (2017). Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2107–2116).
Singer, N. M., & Asari, V. K. (2021). Dales objects: A large scale benchmark dataset for instance segmentation in aerial lidar. IEEE Access, 9, 97495–97504.
Tiago, C., Gilbert, A., Beela, A. S., Aase, S. A., Snare, S. R., Šprem, J., & McLeod, K. (2022). A data augmentation pipeline to generate synthetic labeled datasets of 3D echocardiography images using a GAN. IEEE Access, 10, 98803–98815. https://doi.org/10.1109/ACCESS.2022.3207177
Trabucco, B., Doherty, K., Gurinas, M., & Salakhutdinov, R. (2023). Effective data augmentation with diffusion models. ar**v preprint retrieved from ar**v:2302.07944
Tran, T., Pham, T., Carneiro, G., Palmer, L., & Reid, I. (2017). A bayesian data augmentation approach for learning deep models. Advances in neural information processing systems, 30
Tremblay, J., Prakash, A., Acuna, D., Brophy, M., Jampani, V., Anil, C., et al. (2018). Training deep networks with synthetic data: bridging the reality gap by domain randomization. In 2018 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW) (pp. 1082–10828). Presented at the 2018 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW). https://doi.org/10.1109/CVPRW.2018.00143
van Breugel, B., Kyono, T., Berrevoets, J., & van der Schaar, M. (2021). Decaf: Generating fair synthetic data using causally-aware generative networks. Advances in Neural Information Processing Systems, 34, 22221–22233.
Wang, F., Zhuang, Y., Gu, H., & Hu, H. (2019). Automatic generation of synthetic LiDAR point clouds for 3-D data analysis. IEEE Transactions on Instrumentation and Measurement, 68(7), 2671–2673.
Winiwarter, L., Pena, A. M. E., Weiser, H., Anders, K., Sanchez, J. M., Searle, M., & Höfle, B. (2021). Virtual laser scanning with HELIOS++: A novel take on ray tracing-based simulation of topographic 3D laser scanning. ar**v preprint retrieved from ar**v:2101.09154.
**ao, A., Huang, J., Guan, D., Zhan, F., & Lu, S. (2022). Transfer learning from synthetic to real LiDAR point cloud for semantic segmentation. In Proceedings of the AAAI conference on artificial intelligence (vol. 36, pp. 2795–2803).
Xu, Y., Noy, A., Lin, M., Qian, Q., Li, H., & **, R. (2020). Wemix: How to better utilize data augmentation. ar**v preprint retrieved from ar**v:2010.01267
Ye, Z., Xu, Y., Huang, R., Tong, X., Li, X., Liu, X., et al. (2020). Lasdu: A large-scale aerial lidar dataset for semantic labeling in dense urban areas. ISPRS International Journal of Geo-Information, 9(7), 450.
Yue, X., Wu, B., Seshia, S. A., Keutzer, K., & Sangiovanni-Vincentelli, A. L. (2018). A lidar point cloud generator: from a virtual world to autonomous driving. In Proceedings of the 2018 ACM on international conference on multimedia retrieval (pp. 458–464).
Zhang, X., Chen, R., Li, A., **ang, F., Qin, Y., Gu, J., et al. (2023). Close the optical sensing domain gap by physics-grounded active stereo sensor simulation. IEEE transactions on robotics
Zhu, J. -Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision (pp. 2223–2232).
Acknowledgements
The authors used the support provided by the cluster project under the Data Science (DS) Research of Frontier and Futuristic Technologies (FFT) and National Geospatial Programme (NGP) Division of the Department of Science and Technology (DST), Government of India, New Delhi.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Bharat Lohani has supervised the experiments and the work to compile the manuscript. 3D models were created by Parvej Khan and simulated LiDAR data were generated by Siddhartha Gupta. The first draft of the manuscript was written by Parvej Khan and Vaibhav Kumar and all authors commented on previous versions of the manuscript. Experiments were performed by Parvej Khan and all the authors performed thorough discussion and analysis of the outcomes.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Ethical Approval and Informed Consent
This article does not contain any studies involving animals performed by any of the authors.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Lohani, B., Khan, P., Kumar, V. et al. Role of Simulated Lidar Data for Training 3D Deep Learning Models: An Exhaustive Analysis. J Indian Soc Remote Sens (2024). https://doi.org/10.1007/s12524-024-01905-2
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s12524-024-01905-2