SpringerLink
Log in

Deep reinforcement learning based mapless navigation for industrial AMRs: advancements in generalization via potential risk state augmentation

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

This article introduces a novel Deep Reinforcement Learning (DRL)-based approach for mapless navigation in Industrial Autonomous Mobile Robots, emphasizing advancements in generalization through Potential Risk State Augmentation (PRSA) and an adaptive safety optimization reward function. Traditional LiDAR-based state representations often fail to capture environmental intricacies, leading to suboptimal performance. PRSA addresses this by improving the representation of high-dimensional LiDAR data, focusing on essential risk-related information to reduce redundancy and enhance the DRL agent’s generalization across various industrial settings. The adaptive reward function integrated with intrinsic reward mitigates the issue of sparse rewards in complex tasks, promoting faster learning and optimal policy convergence. Extensive experiments demonstrate that our method maintains a high success rate (over 90%) and low collision risk in narrow and dynamic environments compared to existing DRL-based methods. Meanwhile, compared with the classic navigation baseline, the proposed method improves the success rate by about 33% and reduces the mean navigation time by about 48% in real-world navigation tasks. The direct transfer of policies trained in simulations to real-world environments has demonstrated significant potential for enhancing both the efficacy and reliability of autonomous navigation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Algorithm 1
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Data Availability Statement

Data cannot be shared openly but are available on request from authors: Data sets generated during the current study are available from the corresponding author on reasonable request.

References

  1. Cadena C, Carlone L, Carrillo H et al (2016) Past, present, and future of simultaneous localization and map**: Toward the robust-perception age. IEEE Trans Robot 32(6):1309–1332

    Article  Google Scholar 

  2. Yang H, Xu X, Hong J (2022) Automatic parking path planning of tracked vehicle based on improved a* and dwa algorithms. IEEE Trans Transp Electrif 9(1):283–292

    Article  Google Scholar 

  3. Liu J, Ji J, Ren Y et al (2021) Path planning for vehicle active collision avoidance based on virtual flow field. Int J Automot Technol 22:1557–1567

    Article  Google Scholar 

  4. Zhu K, Zhang T (2021) Deep reinforcement learning based mobile robot navigation: a review. Tsinghua Sci Technol 26(5):674–691

    Article  Google Scholar 

  5. Tai L, Paolo G, Liu M (2017) Virtual-to-real deep reinforcement learning: continuous control of mobile robots for mapless navigation. In: 2017 IEEE/RSJ International conference on intelligent robots and systems (IROS), IEEE, pp 31–36

  6. Shi H, Shi L, Xu M et al (2019) End-to-end navigation strategy with deep reinforcement learning for mobile robots. IEEE Trans Ind Inform 16(4):2393–2402

    Article  Google Scholar 

  7. Wu K, Wang H, Esfahani MA et al (2021) Learn to navigate autonomously through deep reinforcement learning. IEEE Trans Ind Electron 69(5):5342–5352

    Article  Google Scholar 

  8. Luong M, Pham C (2021) Incremental learning for autonomous navigation of mobile robots based on deep reinforcement learning. J Intell Robot Syst 101(1):1

    Article  Google Scholar 

  9. Zhang W, Zhang Y, Liu N et al (2022) Ipaprec: A promising tool for learning high-performance mapless navigation skills with deep reinforcement learning. IEEE/ASME Trans Mechatron 27(6):5451–5461

    Article  Google Scholar 

  10. Wang C, Wang J, Shen Y et al (2019) Autonomous navigation of uavs in large-scale complex environments: a deep reinforcement learning approach. IEEE Trans Veh Technol 68(3):2124–2136

    Article  MathSciNet  Google Scholar 

  11. **e Z, Dames P (2023) Drl-vo: Learning to navigate through crowded dynamic scenes using velocity obstacles. IEEE Trans Robot

  12. De Ryck M, Versteyhe M, Debrouwere F (2020) Automated guided vehicle systems, state-of-the-art control algorithms and techniques. J Manuf Syst 54:152–173

    Article  Google Scholar 

  13. Sprunk C, Lau B, Pfaff P et al (2017) An accurate and efficient navigation system for omnidirectional robots in industrial environments. Auton Robots 41:473–493

    Article  Google Scholar 

  14. Liu X, Wang W, Li X et al (2022) Mpc-based high-speed trajectory tracking for 4wis robot. ISA Trans 123:413–424

    Article  Google Scholar 

  15. Rasekhipour Y, Khajepour A, Chen SK et al (2016) A potential field-based model predictive path-planning controller for autonomous road vehicles. IEEE Trans Intell Transp Syst 18(5):1255–1267

    Article  Google Scholar 

  16. Yang H, Wang Z, **a Y et al (2023) Empc with adaptive apf of obstacle avoidance and trajectory tracking for autonomous electric vehicles. ISA Trans 135:438–448

    Article  Google Scholar 

  17. **ao X, Liu B, Warnell G et al (2022) Motion planning and control for mobile robot navigation using machine learning: a survey. Auton Robots 46(5):569–597

    Article  Google Scholar 

  18. Zhu Y, Mottaghi R, Kolve E et al (2017) Target-driven visual navigation in indoor scenes using deep reinforcement learning. In: 2017 IEEE international conference on robotics and automation (ICRA), IEEE, pp 3357–3364

  19. Yokoyama K, Morioka K (2020) Autonomous mobile robot with simple navigation system based on deep reinforcement learning and a monocular camera. In: 2020 IEEE/SICE International Symposium on System Integration (SII), IEEE, pp 525–530

  20. Zhou Z, Zhu P, Zeng Z et al (2022) Robot navigation in a crowd by integrating deep reinforcement learning and online planning. Appl Intell 52(13):15600–15616

    Article  Google Scholar 

  21. Chen Y, Liu C, Shi BE et al (2020) Robot navigation in crowds by graph convolutional networks with attention learned from human gaze. IEEE Robot Autom Lett 5(2):2754–2761

    Article  Google Scholar 

  22. Sun X, Zhang Q, Wei Y et al (2023) Risk-aware deep reinforcement learning for robot crowd navigation. Electronics 12(23):4744

    Article  Google Scholar 

  23. Liu L, Dugas D, Cesari G, et al (2020) Robot navigation in crowded environments using deep reinforcement learning. In: 2020 IEEE/RSJ International conference on intelligent robots and systems (IROS), IEEE, pp 5671–5677

  24. Pfeiffer M, Schaeuble M, Nieto J et al (2017) From perception to decision: A data-driven approach to end-to-end motion planning for autonomous ground robots. In: 2017 IEEE international conference on robotics and automation (icra), IEEE, pp 1527–1533

  25. Francis A, Faust A, Chiang HTL et al (2020) Long-range indoor navigation with prm-rl. IEEE Trans Robot 36(4):1115–1134

    Article  Google Scholar 

  26. Pfeiffer M, Shukla S, Turchetta M et al (2018) Reinforced imitation: sample efficient deep reinforcement learning for mapless navigation by leveraging prior demonstrations. IEEE Robot Autom Lett 3(4):4423–4430

    Article  Google Scholar 

  27. Li W, Yue M, Shangguan J et al (2023) Navigation of mobile robots based on deep reinforcement learning: Reward function optimization and knowledge transfer. Int J Control Autom Syst 21(2):563–574

    Article  Google Scholar 

  28. Guo H, Ren Z, Lai J et al (2023) Optimal navigation for agvs: a soft actor-critic-based reinforcement learning approach with composite auxiliary rewards. Eng Appl Artif Intell 124:106613

    Article  Google Scholar 

  29. Martinez-Baselga D, Riazuelo L, Montano L (2023) Improving robot navigation in crowded environments using intrinsic rewards. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE, pp 9428–9434

  30. Jiang H, Esfahani MA, Wu K et al (2022) itd3-cln: Learn to navigate in dynamic scene through deep reinforcement learning. Neurocomputing 503:118–128

    Article  Google Scholar 

  31. Jang Y, Baek J, Han S (2021) Hindsight intermediate targets for mapless navigation with deep reinforcement learning. IEEE Trans Ind Electron 69(11):11816–11825

    Article  Google Scholar 

  32. Zhu W, Hayashibe M (2022) A hierarchical deep reinforcement learning framework with high efficiency and generalization for fast and safe navigation. IEEE Trans Ind Electron 70(5):4962–4971

    Article  Google Scholar 

  33. Miranda VR, Neto AA, Freitas GM, et al (2023) Generalization in deep reinforcement learning for robotic navigation by reward sha**. IEEE Trans Ind Electron

  34. Yan C, Qin J, Liu Q et al (2022) Mapless navigation with safety-enhanced imitation learning. IEEE Trans Ind Electron 70(7):7073–7081

    Article  Google Scholar 

  35. Chang L, Shan L, Zhang W et al (2023) Hierarchical multi-robot navigation and formation in unknown environments via deep reinforcement learning and distributed optimization. Robot Comput-Integr Manuf 83:102570

    Article  Google Scholar 

  36. Lim J, Ha S, Choi J (2020) Prediction of reward functions for deep reinforcement learning via gaussian process regression. IEEE/ASME Trans Mechatron 25(4):1739–1746. https://doi.org/10.1109/TMECH.2020.2993564

    Article  Google Scholar 

  37. Zhang W, Liu N, Zhang Y (2021) Learn to navigate maplessly with varied lidar configurations: a support point-based approach. IEEE Robot Autom Lett 6(2):1918–1925. https://doi.org/10.1109/LRA.2021.3061305

    Article  Google Scholar 

  38. Haarnoja T, Zhou A, Abbeel P, et al (2018) Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International conference on machine learning, PMLR, pp 1861–1870

  39. Yang J, Lu S, Han M et al (2023) Mapless navigation for uavs via reinforcement learning from demonstrations. Sci China Technol Sci 66(5):1263–1270

    Article  Google Scholar 

  40. Huang W, Zhou Y, He X, et al (2023) Goal-guided transformer-enabled reinforcement learning for efficient autonomous navigation. IEEE Trans Intell Transp Syst

  41. Gao X, Yan L, Li Z et al (2023) Improved deep deterministic policy gradient for dynamic obstacle avoidance of mobile robot. IEEE Trans Syst, Man, Cybern Syst 53(6):3675–3682

    Article  Google Scholar 

  42. Pathak D, Agrawal P, Efros AA et al (2017) Curiosity-driven exploration by self-supervised prediction. In: International conference on machine learning, PMLR, pp 2778–2787

Download references

Author information

Authors and Affiliations

  1. School of Automation, Central South University, Changsha, 410083, China

    Degang Xu, Peng Chen, **anhan Zhou, Yizhi Wang & Guanzheng Tan

Authors
  1. Degang Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. **anhan Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yizhi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Guanzheng Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yizhi Wang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, D., Chen, P., Zhou, X. et al. Deep reinforcement learning based mapless navigation for industrial AMRs: advancements in generalization via potential risk state augmentation. Appl Intell (2024). https://doi.org/10.1007/s10489-024-05679-5

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-05679-5

Keywords

Search

Navigation