Abstract
Deep Reinforcement Learning (DRL) has gained increasing popularity in the robotics community for tasks such as gras**, manipulation, and decision making. In recent years, DRL has been utilized for autonomous navigation as well—with remarkable results. However, most research works either focus on providing an end-to-end DRL approach or conduct an isolated training and integrate the DRL agent as a separate entity later on. This in turn, comes along with a number of problems such as catastrophic forgetfulness, inefficient navigation behavior, and non-optimal synchronization between different entities of the navigation stack. To address these issues, we propose a holistic Deep Reinforcement Learning training approach in which the training procedure is involving all entities of the navigation stack. This should enhance the synchronization between- and understanding of all entities of the navigation stack and as a result, improve navigational performance in crowded environments. We developed six agents with different observation spaces to study the impact of different input on the navigation behavior of the agent. We did extensive evaluations against multiple learning-based and classic model-based navigation approaches, our proposed agent could outperform the baselines in terms of efficiency and safety attaining shorter path lengths, less roundabout paths, and less collisions especially in situations with a high number of pedestrians.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Marder-Eppstein, E., Berger, E., Foote, T., Gerkey, B., Konolige, K.: The office marathon: robust navigation in an indoor office environment. In: IEEE International Conference on Robotics and Automation, pp. 300–307. IEEE (2010)
Kästner, L., Cox, J., Buiyan, T., Lambrecht, J.: All-in-one: a drl-based control switch combining state-of-the-art navigation planners. In: International Conference on Robotics and Automation (ICRA), pp. 2861–2867 (2022)
Qian, K., Ma, X., Dai, X., Fang, F.: Socially acceptable pre-collision safety strategies for human-compliant navigation of service robots. Adv. Robot. 24(13), 1813–1840 (2010)
Kästner, L., Buiyan, T., Zhao, X., Jiao, L., Shen, Z., Lambrecht, J.: Towards deployment of deep-reinforcement-learning-based obstacle avoidance into conventional autonomous navigation systems. ar**v preprint ar**v:2104.03616 (2021)
Dugas, D., Nieto, J., Siegwart, R., Chung, J.J.: Navrep: unsupervised representations for reinforcement learning of robot navigation in dynamic human environments. ar**v preprint ar**v:2012.04406 (2020)
Chen, C., Liu, Y., Kreiss, S., Alahi, A.: Crowd-robot interaction: crowd-aware robot navigation with attention-based deep reinforcement learning. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 6015–6022. IEEE (2019)
Faust, A., Oslund, K., Ramirez, O., Francis, A., Tapia, L., Fiser, M., Davidson, J.: Prm-rl: long-range robotic navigation tasks by combining reinforcement learning and sampling-based planning. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 5113–5120. IEEE (2018)
**ao, X., Liu, B., Warnell, G., Stone, P.: Motion planning and control for mobile robot navigation using machine learning: a survey. In: Autonomous Robots, pp. 1–29 (2022)
Brito, B., Everett, M., How, J.P., Alonso-Mora, J.: Where to go next: learning a subgoal recommendation policy for navigation in dynamic environments. IEEE Robot. Autom. Lett. 6(3), 4616–4623 (2021)
Kästner, L., Zhao, X., Buiyan, T., Li, J., Shen, Z., Lambrecht, J., Marx, C.: Connecting deep-reinforcement-learning-based obstacle avoidance with conventional global planners using waypoint generators. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1213–1220. IEEE
Chiang, H.-T.L., Faust, A., Fiser, M., Francis, A.: Learning navigation behaviors end-to-end with autorl. IEEE Robot. Autom. Lett. 4(2), 2007–2014 (2019)
Yavas, U., Kumbasar, T., Ure, N.K.: Model-based reinforcement learning for advanced adaptive cruise control: a hybrid car following policy. In: IEEE Intelligent Vehicles Symposium (IV), pp. 1466–1472. IEEE (2022)
Atoui, H., Sename, O., Milanés, V., Martinez, J.J.: Intelligent control switching for autonomous vehicles based on reinforcement learning. In: IEEE Intelligent Vehicles Symposium (IV), pp. 792–797. IEEE (2022)
Everett, M., Chen, Y.F., How, J.P.: Motion planning among dynamic, decision-making agents with deep reinforcement learning. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3052–3059. IEEE (2018)
LaValle, S.M., et al.: Rapidly-exploring random trees: a new tool for path planning (1998)
Hart, P.E., Nilsson, N.J., Raphael, B.: A formal basis for the heuristic determination of minimum cost paths. IEEE Trans. Syst. Sci. Cybern. 4(2), 100–107 (1968)
Güldenring, R., Görner, M., Hendrich, N., Jacobsen, N.J., Zhang, J.: Learning local planners for human-aware navigation in indoor environments
Regier, P., Gesing, L., Bennewitz, M.: Deep reinforcement learning for navigation in cluttered environments (2020)
Bansal, S., Tolani, V., Gupta, S., Malik, J., Tomlin, C.: Combining optimal control and learning for visual navigation in novel environments. In: Conference on Robot Learning. PMLR, pp. 420–429 (2020)
Kästner, L., Zhao, X., Shen, Z., Lambrecht, J.: Obstacle-aware waypoint generation for long-range guidance of deep-reinforcement-learning-based navigation approaches. ar**v preprint ar**v:2109.11639 (2021)
Rösmann, C., Hoffmann, F., Bertram, T.: Planning of multiple robot trajectories in distinctive topologies. In: 2015 European Conference on Mobile Robots (ECMR), pp. 1–6. IEEE (2015)
Khatib, O.: Real-time obstacle avoidance for manipulators and mobile robots. In: Autonomous Robot Vehicles, pp. 396–404. Springer, Berlin (1986)
Rösmann, C., Makarow, A., Bertram, T.: Online motion planning based on nonlinear model predictive control with non-Euclidean rotation groups. ar**v preprint ar**v:2006.03534 (2020)
Rösmann, C., Hoffmann, F., Bertram, T.: Timed-elastic-bands for time-optimal point-to-point nonlinear model predictive control. In: European Control Conference (ECC), pp. 3352–3357. IEEE (2015)
Rösmann, C.: Time-optimal nonlinear model predictive control. Ph.D. dissertation, Technische Universität Dortmund (2019)
Helbing, D., Molnar, P.: Social force model for pedestrian dynamics. Phys. Rev. E 51(5), 4282 (1995)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Meusel, M., Kästner, L., Bhuiyan, T., Lambrecht, J. (2024). Enhancing Navigational Performance with Holistic Deep-Reinforcement-Learning. In: Lee, SG., An, J., Chong, N.Y., Strand, M., Kim, J.H. (eds) Intelligent Autonomous Systems 18. IAS 2023. Lecture Notes in Networks and Systems, vol 795. Springer, Cham. https://doi.org/10.1007/978-3-031-44851-5_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-44851-5_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44850-8
Online ISBN: 978-3-031-44851-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)