SpringerLink
Log in

Probabilistic Deep Q Network for real-time path planning in censorious robotic procedures using force sensors

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

In recent years, enormous advancement has taken place in biomedical engineering, which has paved the way for robot-assisted surgery in various complex surgical procedures. In robotic surgery, the reinforcement-based Temporal Difference (TD) based approach through assistive approaches has tremendous potential. Probabilistic Roadmap (PR) can be used for recognition of the path to the region of interest without any obstacles and, Inverse Kinematics (IK) approach can be used for the accurate approximation of the pixel space to the real-time workspace. Our proposed system would be more effective in approximating the path length, depth evaluation, and less invasive contact force sensor. This article presents a robust algorithm that would assist in robotic surgery for censorious surgeries in real-time. For working on such soft tissues, software-driven procedures and algorithms must be more precise in choosing the optimal path for reaching out to the procedural region. The statistical analysis has proven that the proposed approach would be outperforming under favorable learning rate, discount factor, and the exploration factor.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Palep, J.H.: Robotic-assisted minimally invasive surgery. J. Minimal Access Surg. 5(1), 1–7 (2009). https://doi.org/10.4103/0972-9941.51313

    Article  Google Scholar 

  2. Siddaiah-Subramanya, M., Tiang, K., Nyandowe, M.: A new era of minimally invasive surgery: progress and development of major technical innovations in general surgery over the last decade. Surg. J. 3(4), e163–e166 (2017)

    Article  Google Scholar 

  3. Vitielloet, V., Lee, S.-L., Cundy, T.P., Yang, G.-Z.: Emerging robotic platforms for minimally invasive surgery. IEEE Rev. Biomed. Eng. 6, 111–126 (2012)

    Article  Google Scholar 

  4. Ozmen, M.M., Ozmen, A., Koç, Ç.K.: Artificial intelligence for next-generation medical robotics. In: Atallah, S. (ed.) Digital surgery. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-49100-0_3

    Chapter  Google Scholar 

  5. Yang, L., Qi, J., Song, D., **ao, J., Han, J., **a, Y.: Survey of robot 3D path planning algorithms. J. Control Sci. Eng. (2016). https://doi.org/10.1155/2016/7426913

    Article  MathSciNet  MATH  Google Scholar 

  6. Ashrafian, H., Clancy, O., Grover, V., Darzi, A.: The evolution of robotic surgery: surgical and anesthetic aspects. Br. J. Anaesth. 119(1), 172–184 (2017)

    Article  Google Scholar 

  7. Chand, M., Ramachandran, N., Stoyanov, D., et al.: Robotics, artificial intelligence, and distributed ledgers in surgery: data is key. Tech. Coloproctol. 22, 645–648 (2018). https://doi.org/10.1007/s10151-018-1847-5

    Article  Google Scholar 

  8. Kassahun, Y., Yu, B., Tibebu, A.T., et al.: Surgical robotics beyond enhanced dexterity instrumentation: a survey of machine learning techniques and their role intelligent and autonomous surgical actions. Int. J. CARS 11, 553–568 (2016). https://doi.org/10.1007/s11548-015-1305-z

    Article  Google Scholar 

  9. Zhang, L., Huang, T., Hu, X., Zhang, Z., Wang, W., Guan, D., Zhao, C., Kim, S.: A distributed covert channel of the packet ordering enhancement model based on data compression. CMC-Comput. Mater. Contunia 64(3), 2013–2030 (2020)

    Google Scholar 

  10. Li, X., Lv, Z., Wang, S., Wei, Z., Wu, L.: A reinforcement learning model based on temporal difference algorithm. IEEE Access 7, 121922–121930 (2019). https://doi.org/10.1109/ACCESS.2019.2938240

    Article  Google Scholar 

  11. Baek, D., Hwang, M., Kim, H., Kwon, D.: Path planning for automation of surgery robot based on probabilistic roadmap and reinforcement learning. In: 2018 15th International Conference on Ubiquitous Robots (UR), Honolulu, HI, pp. 342–347 (2018). https://doi.org/10.1109/URAI.2018.8441801

  12. Zhiye, L., **ong, C.: Path planning approach based on the probabilistic roadmap for sensor based car-like robot in unknown environments. In: 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583), The Hague, Vol. 3 pp. 2907–2912 (2004). https://doi.org/10.1109/ICSMC.2004.1400774

  13. Tahoun, M., Tayba, A., Alfayad, S., Wahdan, A., Chassagne, L.: Force localized interaction sensing system for HYDROïD humanoid robot. IEEE Sens. J. 20(8), 4112–4119 (2020). https://doi.org/10.1109/JSEN.2019.2962281

    Article  Google Scholar 

  14. Aspland, E., Gartner, D., Harper, P.: Clinical pathway modelling: a literature review. Health Syst. 10, 1–23 (2019)

    Article  Google Scholar 

  15. Francesco S. et al.: A multi robots teleoperated platform for artificial intelligence training data collection in minimally invasive surgery. In 2019 International Symposium on Medical Robotics (ISMR), Atlanta, GA, USA, 2019, pp. 1–7 (2019). https://doi.org/10.1109/ISMR.2019.8710209.

  16. Marban, A., Srinivasan, V., Samek, W., Fernandez, J., Casals, A.: A recurrent convolutional neural network approach for sensorless force estimation in robotic surgery. Biomed. Signal Process. Control 50, 134–150 (2018)

    Article  Google Scholar 

  17. Gadekallu, T.R., Alazab, M., Kaluri, R., et al.: Hand gesture classification using a novel CNN-crow search algorithm. Complex Intell. Syst. (2021). https://doi.org/10.1007/s40747-021-00324-x

    Article  Google Scholar 

  18. Khalid, S., Goldenberg, M., Grantcharov, T., Taati, B., Rudzicz, F.: Evaluation of deep learning models for identifying surgical actions and measuring performance. JAMA Netw. Open. 3(3), e201664 (2020)

    Article  Google Scholar 

  19. Vasan, D., Alazab, M., Wassan, S., Naeem, H., Safaei, B., Zheng, Q.: IMCFN: Image-based malware classification using fine-tuned convolutional neural network architecture. Comput. Netw. 171, 107138 (2020)

    Article  Google Scholar 

  20. Thai, M. T., Phan, P. T., Wong, S., Lovell, N., Do, T.: Advanced intelligent systems for surgical robotics. (2020)

  21. Wang, W. and Su, C.: Ccbrsn: a system with high embedding capacity for covert communication in bitcoin. In: IFIP International Conference on ICT Systems Security and Privacy Protection. Springer, Cham, pp. 324–337 (2020)

  22. Wang, W., Huang, H., Zhang, L., Su, C.: Secure and efficient mutual authentication protocol for smart grid under blockchain. Peer-to-Peer Netw. Appl. (2020). https://doi.org/10.1007/s12083-020-01020-2

    Article  Google Scholar 

  23. Zhang, J., Gao, X.: Object extraction via deep learning-based marker-free tracking framework of surgical instruments for laparoscope-holder robots. Int. J. CARS 15, 1335–1345 (2020). https://doi.org/10.1007/s11548-020-02214-y

    Article  Google Scholar 

  24. Naga, S.P., Rao, T., Balas, V.: Volumetric estimation of the damaged area in the human brain from 2D MR image. Int. J. Inf. Syst. Model. Design. 11, 74–92 (2020). https://doi.org/10.4018/IJISMD.2020010105

    Article  Google Scholar 

  25. Wang, Z., Majewicz Fey, A.: Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery. Int. J. CARS 13, 1959–1970 (2018). https://doi.org/10.1007/s11548-018-1860-1

    Article  Google Scholar 

  26. Almusawi, A.R., Dülger, L.C., Kapucu, S.: Artificial neural network based kinematics: case study on robotic surgery. In: Uhl, T. (ed.) Advances in mechanism and machine science. IFToMM WC 2019. Mechanisms and machine science, vol. 73. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20131-9_182

    Chapter  Google Scholar 

  27. Shrivastava, R., Kumar, P., Tripathi, S., Tiwari, V., Rajput, D.S., Gadekallu, T.R., et al.: A novel grid and place neuron’s computational modeling to learn spatial semantics of an environment. Appl. Sci. 10(15), 5147 (2020)

    Article  Google Scholar 

  28. Nguyen, N. D., Nguyen, T., Nahavandi, S., Bhatti, A., Guest, G.: Manipulating soft tissues by deep reinforcement learning for autonomous robotic surgery (2019).

  29. Meyer, E., Robinson, H., Rasheed, A., San, O.: Taming an autonomous surface vehicle for path following and collision avoidance using deep reinforcement learning. IEEE Access 8, 41466–41481 (2020). https://doi.org/10.1109/ACCESS.2020.2976586

    Article  Google Scholar 

  30. Richter, F., Orosco, R. K., Yip, M.: Open-sourced reinforcement learning environments for surgical robotics (2019). ar**v:1903.02090

  31. Tan, X., Chng, C., Su, Y., Lim, K., Chui, C.: Robot-assisted training in laparoscopy using deep reinforcement learning. IEEE Robot. Autom. Lett (2019). https://doi.org/10.1109/LRA.2019.2891311

    Article  Google Scholar 

  32. Shvets A.A., Rakhlin, A., Kalinin, A.A., Iglovikov V.I.: Automatic instrument segmentation in robot-assisted surgery using deep learning. In: 2018 17th IEEE international conference on machine learning and applications (ICMLA), Orlando, FL, USA, pp. 624–628 (2018)

  33. Gadekallu, T.R., Rajput, D.S., Reddy, M.P.K., et al.: A novel PCA–whale optimization-based deep neural network model for classification of tomato plant diseases using GPU. J. Real-Time Image Proc. (2020). https://doi.org/10.1007/s11554-020-00987-8

    Article  Google Scholar 

  34. Naga, S.P., Rao, T., Dicu, A., Mihaela & Mnerie, Corina & Olariu, Iustin: A comparative review of optimisation techniques in segmentation of brain MR images. J. Intell. Fuzzy Syst. 38, 1–12 (2020). https://doi.org/10.3233/JIFS-179688

    Article  Google Scholar 

  35. Abeywardena, S., Yuan, Q., Tzemanaki, A., Psomopoulou, E., Droukas, L., Melhuish, C., Dogramadzi, S.: Estimation of tool-tissue forces in robot-assisted minimally invasive surgery using neural networks. Front. Robot. AI 6, 56 (2019)

    Article  Google Scholar 

  36. Mingo, H.E., Rocchi, A., Tsagarakis, N.G., Caldwell, D.G.: Robot dynamics constraint for inverse kinematics. In: Lenarčič, J., Merlet, J.P. (eds.) Advances in robot kinematics 2016. Springer proceedings in advanced robotics, vol. 4. Springer, Cham (2018)

    Google Scholar 

  37. Alterovitz R., Lim A., Goldberg K., Chirikjian G.S., Okamura A.M.: Steering flexible needles under Markov motion uncertainty. Intelligent Robots and Systems, IEEE/RSJ International Conference on; IEEE, pp. 1570–1575 (2005).

  38. Duindam, V., Xu, J., Alterovitz, R., Sastry, S., Goldberg, K.: 3D motion planning algorithms for steerable needles using inverse kinematics. In: Chirikjian, G.S., Choset, H., Morales, M., Murphey, T. (eds.) Algorithmic foundation of robotics VIII. Springer tracts in advanced robotics, vol. 57. Springer, Berlin (2009). https://doi.org/10.1007/978-3-642-00312-7_33

    Chapter  Google Scholar 

  39. Jackson, R.C., Cavusoglu, M.C.: Needle path planning for autonomous robotic surgical suturing. Robotics and Automation (ICRA), In 2013 IEEE International Conference on; IEEE; pp. 1669–1675 (2013).

  40. Bernardes, M., Adorno, B.V., Poignet, P., Borges, G.: Robot-assisted automatic insertion of steerable needles with closed-loop imaging feedback and intraoperative trajectory replanning. Mechatronics 23(6), 630–645 (2013)

    Article  Google Scholar 

  41. Noh, Y., Han, S., Gawenda, P., Li, W., Sareh, S., Rhode, K.: A contact force sensor based on S-shaped beams and optoelectronic sensors for flexible manipulators for minimally invasive surgery (MIS). IEEE Sens. J. 20(7), 3487–3495 (2020). https://doi.org/10.1109/JSEN.2019.2945163

    Article  Google Scholar 

  42. Nair, D., Supriya, P.: Comparison of temporal difference learning algorithm and Dijkstra's algorithm for robotic path planning. In: Proceedings of the 2nd International Conference on Intelligent Computing and Control Systems, pp. 1619–1624 (2018).

  43. Kala, R., Shukla, A., Tiwari, R.: Robot path planning using dynamic programming with accelerating nodes. Paladyn J. Behav. Robot. 3(1), 23–34 (2012)

    Google Scholar 

  44. Janson, L., Schmerling, E., Pavone, M.: Monte carlo motion planning for robot trajectory optimization under uncertainty. In: Bicchi, A., Burgard, W. (eds.) Robotics research. Springer proceedings in advanced robotics, vol. 3. Springer, Cham (2018)

    Google Scholar 

  45. Lei, X., Zhang, Z., Dong, P.: Dynamic path planning of unknown environment based on deep reinforcement learning. J. Robot. (2018). https://doi.org/10.1155/2018/5781591

    Article  Google Scholar 

  46. Mosavi, A., Faghan, Y., Ghamisi, P., Duan, P., Ardabili, S.F., Salwana, E., Band, S.S.: Comprehensive review of deep reinforcement learning methods and applications in economics. Mathematics 8, 1640 (2020)

    Article  Google Scholar 

  47. Yu, J., Su, Y., Liao, Y.: The path planning of mobile robot by neural networks and hierarchical reinforcement learning. Front. Neurorobotics 4, 63 (2020)

    Article  Google Scholar 

  48. Chen, H., Ji, Y., Niu, L.: Reinforcement learning path planning algorithm based on obstacle area expansion strategy. Intel. Serv. Robotics 13, 289–297 (2020). https://doi.org/10.1007/s11370-020-00313-y

    Article  Google Scholar 

  49. Ahmidi, N., Hager, G.D., Ishii, L., Gallia, G.L., Ishii, M.: Robotic path planning for surgeon skill evaluation in minimally-invasive sinus surgery. In: Ayache, N., Delingette, H., Golland, P., Mori, K. (eds.) Medical image computing and computer-assisted intervention – MICCAI 2012. MICCAI 2012. Lecture notes in computer science, vol. 7510. Springer, Heidelberg (2012)

    Google Scholar 

  50. Mackeprang, J., Dasari, D.B.R., Wrachtrup, J.: A reinforcement learning approach for quantum state engineering. Quantum Mach. Intell. 2, 5 (2020)

    Article  Google Scholar 

Download references

Acknowledgements

We thank the anonymous referees for their useful suggestions in improvising the paper

Author information

Authors and Affiliations

  1. Computer Science and Engineering, GITAM Institute of Technology, GITAM Deemed to be University, Visakhapatnam, 530045, India

    Parvathaneni Naga Srinivasu

  2. Department of Electrical and Electronics Engineering, Sikkim Manipal Institute of Technology, Sikkim Manipal University, Majitar, Sikkim, 737136, India

    Akash Kumar Bhoi

  3. Department of Computer Science and Engineering, PanditDeendayal Petroleum University, Gandhinagar, India

    Rutvij H. Jhaveri

  4. School of Information Technology and Engineering, Vellore Institute of Technology, Vellore, Tamil Nadu, India

    Gadekallu Thippa Reddy

  5. Department of Computer Engineering, Hankuk University of Foreign Studies, Yongin-si, 17035, South Korea

    Muhammad Bilal

Authors
  1. Parvathaneni Naga Srinivasu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Akash Kumar Bhoi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rutvij H. Jhaveri
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gadekallu Thippa Reddy
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Muhammad Bilal
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All the authors have equally contributed to this manuscript.

Corresponding author

Correspondence to Muhammad Bilal.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Consent for publication

As per the journal guidelines and norms.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Srinivasu, P.N., Bhoi, A.K., Jhaveri, R.H. et al. Probabilistic Deep Q Network for real-time path planning in censorious robotic procedures using force sensors. J Real-Time Image Proc 18, 1773–1785 (2021). https://doi.org/10.1007/s11554-021-01122-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-021-01122-x

Keywords

Search

Navigation