SpringerLink
Log in

Visual Localisation for Knee Arthroscopy

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose 

Navigation in visually complex endoscopic environments requires an accurate and robust localisation system. This paper presents the single image deep learning based camera localisation method for orthopedic surgery.

Methods 

The approach combines image information, deep learning techniques and bone-tracking data to estimate camera poses relative to the bone-markers. We have collected one arthroscopic video sequence for four knee flexion angles, per synthetic phantom knee model and a cadaveric knee-joint.

Results 

Experimental results are shown for both a synthetic knee model and a cadaveric knee-joint with mean localisation errors of 9.66mm/0.85\(^\circ \) and 9.94mm/1.13\(^\circ \) achieved respectively. We have found no correlation between localisation errors achieved on synthetic and cadaveric images, and hence we predict that arthroscopic image artifacts play a minor role in camera pose estimation compared to constraints introduced by the presented setup. We have discovered that the images acquired for 90°and 0°knee flexion angles are respectively most and least informative for visual localisation.

Conclusion 

The performed study shows deep learning performs well in visually challenging, feature-poor, knee arthroscopy environments, which suggests such techniques can bring further improvements to localisation in Minimally Invasive Surgery.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Jaiprakash A, O’Callaghan WB, Whitehouse SL, Pandey A, Wu L, Roberts J, Crawford R (2017) Orthopaedic surgeon attitudes towards current limitations and the potential for robotic and technological innovation in arthroscopic surgery. J Orthop Surg 25(1):1–6

    Google Scholar 

  2. Banach A, Leibrandt K, Grammatikopoulou M, Yang G-Z (2019) Active contraints for tool-shaft collision avoidance in minimally invasive surgery. In: IEEE ICRA, pp 1556–1562

  3. Marmol A, Corke P, Peynot T (2018) ArthroSLAM : multi-sensor robust visual localization for minimally invasive orthopedic surgery. In: IEEE IROS

  4. Bailey T, Durrant-Whyte H (2006) Simultaneous localization and map** (SLAM): part I. IEEE Robot Autom Mag 13(3):108–117

    Article  Google Scholar 

  5. Bailey T, Durrant-whyte H (2006) Simultaneous localization and map** (SLAM): part II. IEEE Robot Autom Soc 13(September):108–117

    Article  Google Scholar 

  6. Cadena C, Carlone L, Carrillo H, Latif Y, Scaramuzza D, Reid I, Leonard JJ, Neira J, Reid I, Leonard JJ (2016) Past, present, and future of simultaneous localization and map**: toward the robust-perception age. IEEE Trans 32(6):1309–1332

    Google Scholar 

  7. Maier-Hein L, Mountney P, Bartoli A, Elhawary H, Elson DS, Groch A, Kolb A, Rodrigues M, Sorger J, Speidel S, Stoyanov D (2013) Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery. Med Image Anal 17(8):974–996

    Article  CAS  Google Scholar 

  8. Stoyanov D, Yang G-Z (2009) Soft tissue deformation tracking for robotic assisted minimally invasive surgery. In: Annual international conference of the IEEE engineering in medicine and biology society, pp. 254–257, 2009

  9. Mahmoud N, Grasa ÓG, Nicolau SA, Doignon C, Soler L, Marescaux J, Montiel JM (2017) On-patient see-through augmented reality based on visual SLAM. Int J Comput Assist Radiol Surg 12(1):1–11

    Article  Google Scholar 

  10. Strydom M, Crawford R, Roberts J, Jaiprakash A (2019) Robotic arthroscopy: the uncertainty in internal knee joint measurement. IEEE Access 7:168382–168394

    Article  Google Scholar 

  11. Marmol A, Peynot T, Eriksson A, Jaiprakash A, Roberts J, Crawford R (2017) Evaluation of keypoint detectors and descriptors in arthroscopic images for feature-based matching applications. IEEE Robot Autom Lett 2(4):2135–2142

    Article  Google Scholar 

  12. Marmol A, Banach A, Peynot T (2019) Dense-ArthroSLAM: dense intra-articular 3-D reconstruction with robust localization prior for arthroscopy. IEEE Robot Autom Lett 4(2):918–925

    Article  Google Scholar 

  13. Mur-Artal R, Tardós JD (2017) Orb-slam2: an open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Trans Robot 33(5):1255–1262

    Article  Google Scholar 

  14. Engel J, Schops T, Cremers D (2014) LSD-SLAM: large-scale direct monocular SLAM. In: European conference on computer vision, pp 834–849

  15. Banach A, Strydom M, Jaiprakash A, Carneiro G, Brown C, Crawford R, McFadyen A (2020) Saliency improvement in feature-poor surgical environments using local Laplacian of Specified Histograms. IEEE Access 8:1–1

    Article  Google Scholar 

  16. Goodfellow I, Bengio Y, Courville A (2016) Deep learning: whole book. Nature

  17. Mahmood F, Chen R, Durr NJ (2018) Unsupervised reverse domain adaptation for synthetic medical images via adversarial training. IEEE Trans Med Imaging 37(12):2572–2581

    Article  Google Scholar 

  18. Visentini-scarzanella M, Koto S, Sugiura T, Kaneko T (2017) Deep monocular 3D reconstruction for assisted navigation in bronchoscopy. Int J Comput Assist Radiol Surg 7:1089–1099

    Article  Google Scholar 

  19. Hou B, Alansary A, McDonagh S, Davidson A, Rutherford M, Hajnal JV, Rueckert, Glocker DB, Kainz B (2017) Predicting slice-to-volume transformation in presence of arbitrary subject motion. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 296–304

  20. Kausch L, Thomas S, Kunze H, Privalov M, Vetter S, Franke J, Mahnken AH, Maier-Hein L, Maier-Hein K (2020) Toward automatic c-arm positioning for standard projections in orthopedic surgery. Int J Comput Assist Radiol Surg 15(7):1095–1105

    Article  Google Scholar 

  21. Sganga J, Eng D, Graetzel C, Camarillo D (2018) OffsetNet: deep learning for localization in the lung using rendered images. Ar**v e-prints

  22. Sganga J, Eng D, Graetzel C, Camarillo DB (2019) Deep learning for localization in the lung. Ar**v e-prints

  23. Zhou T, Berkeley UC, Brown M, Snavely N, Lowe DG (1851) Unsupervised learning of depth and ego-motion from video, pp 1851–1860

  24. Wang R, Pizer SM, Frahm JM (2019) Recurrent neural network for (un-)supervised learning of monocular video visual odometry and depth. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, 2019

  25. Kendall A, Cipolla R (2015) PoseNet: a convolutional network for real-time 6-DOF camera relocalization. In: International conference on computer vision, pp 2938–2946

  26. Strydom M, Banach A, Wu L, Jaiprakash A, Crawford R, Roberts J (2020) Anatomical joint measurement with application to medical robotics. IEEE Access

  27. Strydom ML, Banach A, Roberts J, Crawford R, Jaiprakash AT (2020) Kinematic model of the human leg using dh parameters. IEEE Access 8:191737–191750

  28. Wengert C, Reeff M, Cattin PC, Székely G (2006) Fully automatic endoscope calibration for intraoperative use. In: Bildverarbeitung für die Medizin 2006

  29. Bouguet J-Y (2002) Camera calibration toolbox for matlab

  30. Abadi M, Barham P, Chen J, Chen Z, Davis A, Jeffrey D, Devin M, Ghemawat S, Irving G, Isard M, Kudlur M, Levenberg V, Monga R, Moore S, Murray D, Steiner B, Tucker P, Vasudevan V, Warden P, Wicke M, Yu Y, Zheng X (2016) TensorFlow: a system for large-scale machine learning. In: Proceedings of the 12th USENIX conference on operator system design and implementation, 2016

  31. Mayo B, Hazan T, Tal A (2021) Visual navigation with spatial attention. ar**v preprint ar**v:2104.09807

  32. Shen M, Gu Y, Liu N, Yang GZ (2019) Context-aware depth and pose estimation for bronchoscopic navigation. IEEE Robot Autom Lett 4(2):732–739

    Article  Google Scholar 

  33. Lu T-W, O’connor J (1999) Bone position estimation from skin marker co-ordinates using global optimisation with joint constraints. J Biomech 32(2):129–134

    Article  CAS  Google Scholar 

  34. Charlton IW, Tate P, Smyth P, Roren L (2004) Repeatability of an optimised lower body model. Gait Posture 20(2):213–221

    Article  CAS  Google Scholar 

  35. Carse B, Meadows B, Bowers R, Rowe P (2013) Affordable clinical gait analysis: an assessment of the marker tracking accuracy of a new low-cost optical 3D motion analysis system. Physiotherapy (United Kingdom) 99(4):347–351

    Google Scholar 

  36. Allen MW, Jacofsky DJ (2019) Evolution of robotics in arthroplasty. In: Lonner JH (ed) Robotics in knee and hip arthroplasty. Springer, Berlin, pp 13–25

    Chapter  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Dr. Yu Takuda and Dr. Andres Marmol-Velez for fruitful discussions and help during the cadaveric experiments.

Funding

This research was funded by the AISRF - Biotechnology Collaborative Research Project.

Author information

Authors and Affiliations

  1. QUT Centre for Robotics, Queensland University of Technology, Brisbane, 4000, Australia

    Artur Banach, Mario Strydom, Anjali Jaiprakash, Ross Crawford & Aaron McFadyen

  2. University of Queensland, St Lucia, Queensland, 4072, Australia

    Anders Eriksson

  3. University of Adelaide, Adelaide, 5005, Australia

    Gustavo Carneiro

Authors
  1. Artur Banach
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mario Strydom
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anjali Jaiprakash
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gustavo Carneiro
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Anders Eriksson
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ross Crawford
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Aaron McFadyen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Artur Banach.

Ethics declarations

Availability of data and material

The dataset collected for this work will be made publicly available upon acceptance.

Conflict of interest

The authors (AB, MS, AJ, GC, AE, RC, AM) do not report any conflict of interest.

Ethics approval

Cadaveric experiments were approved by the Australian National Health and Medical Research Council (NHMRC) - Committee no. EC00171.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Banach, A., Strydom, M., Jaiprakash, A. et al. Visual Localisation for Knee Arthroscopy. Int J CARS 16, 2137–2145 (2021). https://doi.org/10.1007/s11548-021-02444-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-021-02444-8

Search

Navigation