SpringerLink
Log in

Sensor-based machine learning for workflow detection and as key to detect expert level in laparoscopic suturing and knot-tying

  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Introduction

The most common way of assessing surgical performance is by expert raters to view a surgical task and rate a trainee’s performance. However, there is huge potential for automated skill assessment and workflow analysis using modern technology. The aim of the present study was to evaluate machine learning (ML) algorithms using the data of a Myo armband as a sensor device for skills level assessment and phase detection in laparoscopic training.

Materials and methods

Participants of three experience levels in laparoscopy performed a suturing and knot tying task on silicon models. Experts rated performance using Objective Structured Assessment of Surgical Skills (OSATS). Participants wore Myo armbands (Thalmic Labs™, Ontario, Canada) to record acceleration, angular velocity, orientation, and Euler orientation. ML algorithms (decision forest, neural networks, boosted decision tree) were compared for skill level assessment and phase detection.

Results

28 participants (8 beginner, 10 intermediate, 10 expert) were included, and 99 knots were available for analysis. A neural network regression model had the lowest mean absolute error in predicting OSATS score (3.7 ± 0.6 points, r2 = 0.03 ± 0.81; OSATS min.-max.: 4–37 points). An ensemble of binary-class neural networks yielded the highest accuracy in predicting skill level (beginners: 82.2% correctly identified, intermediate: 3.0%, experts: 79.5%) whereas standard statistical analysis failed to discriminate between skill levels. Phase detection on raw data showed the best results with a multi-class decision jungle (average 16% correctly identified), but improved to 43% average accuracy with two-class boosted decision trees after Dynamic time war** (DTW) application.

Conclusion

Modern machine learning algorithms aid in interpreting complex surgical motion data, even when standard analysis fails. Dynamic time war** offers the potential to process and compare surgical motion data in order to allow automated surgical workflow detection. However, further research is needed to interpret and standardize available data and improve sensor accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Germany)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Delaney CP et al (2003) Case-matched comparison of clinical and financial outcome after laparoscopic or open colorectal surgery. Ann Surg 238(1):67

    PubMed  PubMed Central  Google Scholar 

  2. Reza M et al (2006) Systematic review of laparoscopic versus open surgery for colorectal cancer. Br J Surg 93(8):921–928

    Article  CAS  Google Scholar 

  3. Nguyen KT et al (2011) Comparative benefits of laparoscopic vs open hepatic resection: a critical appraisal. Arch Surg 146(3):348–356

    Article  Google Scholar 

  4. Shabanzadeh DM, Sørensen LT (2012) Laparoscopic surgery compared with open surgery decreases surgical site infection in obese patients: a systematic review and meta-analysis. Ann Surg 256(6):934–945

    Article  Google Scholar 

  5. Vassiliou MC et al (2005) A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 190(1):107–113

    Article  Google Scholar 

  6. Miskovic D et al (2012) Learning curve and case selection in laparoscopic colorectal surgery: systematic review and international multicenter analysis of 4852 cases. Dis Colon Rectum 55(12):1300–1310

    Article  Google Scholar 

  7. Nickel F et al (2016) Sequential learning of psychomotor and visuospatial skills for laparoscopic suturing and knot tying-a randomized controlled trial “The shoebox study” DRKS00008668. Langenbecks Arch Surg 401(6):893–901

    Article  Google Scholar 

  8. Martin J et al (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84(2):273–278

    Article  CAS  Google Scholar 

  9. Loukas C (2017) Video content analysis of surgical procedures. Surg Endosc 32:553

    Article  Google Scholar 

  10. Ahmadi S-A et al (2006) Recovery of surgical workflow without explicit models. In: International Conference on medical image computing and computer-assisted intervention. Springer, New York

  11. Bardram JE et al (2011) Phase recognition during surgical procedures using embedded and body-worn sensors. IT University of Copenhagen, Copenhagen

    Book  Google Scholar 

  12. Padoy N et al (2007) A boosted segmentation method for surgical workflow analysis. In: International Conference on medical image computing and computer-assisted intervention. Springer, New York

  13. Katic D et al (2016) Bridging the gap between formal and experience-based knowledge for context-aware laparoscopy. Int J Comput Assist Radiol Surg 11(6):881–888

    Article  Google Scholar 

  14. Rosen J et al (2001) Markov modeling of minimally invasive surgery based on tool/tissue interaction and force/torque signatures for evaluating surgical skills. IEEE Trans Biomed Eng 48(5):579–591

    Article  CAS  Google Scholar 

  15. Reiley CE et al (2011) Review of methods for objective surgical skill evaluation. Surg Endosc 25(2):356–366

    Article  Google Scholar 

  16. Spangenberg N et al (2017) Method for intra-surgical phase detection by using real-time medical device data. In: IEEE 30th International Symposium on computer-based medical systems

  17. Ganni S et al (2018) A software-based tool for video motion tracking in the surgical skills assessment landscape. Surg Endosc 32(6):2994

    Article  Google Scholar 

  18. Lin P-J, Chen HY (2018) Design and implement of a rehabilitation system with surface electromyography technology. In: 2018 IEEE International Conference on applied system invention (ICASI). IEEE

  19. Ryser F et al (2017) Fully embedded myoelectric control for a wearable robotic hand orthosis. IEEE Int Conf Rehabil Robot 2017:615–621

    PubMed  Google Scholar 

  20. Sathiyanarayanan M, Raja S (2016) Myo armband for physiotherapy healtchare: a case study using gesture recognition application

  21. Kutafina E et al (2016) Wearable sensors for eLearning of manual tasks: using forearm EMG in hand hygiene training. Sensors (Basel) 16(8):1221

    Article  Google Scholar 

  22. Jimenez DA et al (2016) Human-computer interaction for image guided surgery systems using physiological signals: application to deep brain stimulation surgery. In VII Latin American Congress on Biomedical Engineering CLAIB 2016, Bucaramanga, Santander, Colombia, October 26th–28th, 2017. Springer

  23. Sanchez-Margallo FM et al (2017) Use of natural user interfaces for image navigation during laparoscopic surgery: initial experience. Minim Invasive Ther Allied Technol 26(5):253–261

    Article  Google Scholar 

  24. Romero P et al (2014) Intracorporal suturing—driving license necessary? J Pediatr Surg 49(7):1138–1141

    Article  CAS  Google Scholar 

  25. Munz Y et al (2007) Curriculum-based solo virtual reality training for laparoscopic intracorporeal knot tying: objective assessment of the transfer of skill from virtual reality to reality. Am J Surg 193(6):774–783

    Article  Google Scholar 

  26. Kowalewski K-F et al (2016) Development and validation of a sensor- and expert model-based training system for laparoscopic surgery: the iSurgeon. Surg Endosc 31:2155

    Article  Google Scholar 

  27. Chang OH et al (2015) Develo** an objective structured assessment of technical skills for laparoscopic suturing and intracorporeal knot tying. J Surg Educ 73:258

    Article  Google Scholar 

  28. Brown JD et al (2017) Using contact forces and robot arm accelerations to automatically rate surgeon skill at peg transfer. IEEE Trans Biomed Eng 64(9):2263–2275

    Article  Google Scholar 

  29. Sakoe H, Chiba S (1978) Dynamic programming algorithm optimization for spoken word recognition. IEEE Trans ASSP 26(1):43

    Article  Google Scholar 

  30. Wang Q (2013) Dynamic Time War** (DTW). MathWorks file exchange. https://www.mathworks.com/matlabcentral/fileexchange/43156-dynamic-time-war**-dtw

  31. Ahmidi N et al (2010) Surgical task and skill classification from eye tracking and tool motion in minimally invasive surgery. Med Image Comput Comput Assist Interv 13(Pt 3):295–302

    PubMed  Google Scholar 

  32. Fard MJ et al (2018) Automated robot-assisted surgical skill evaluation: predictive analytics approach. Int J Med Robot 14(1):e1850

    Article  Google Scholar 

  33. Rosen J et al (2001) Objective laparoscopic skills assessments of surgical residents using Hidden Markov Models based on haptic information and tool/tissue interactions. Stud Health Technol Inform 81:417–423

    CAS  PubMed  Google Scholar 

  34. Oropesa I et al (2013) EVA: laparoscopic instrument tracking based on endoscopic video analysis for psychomotor skills assessment. Surg Endosc 27(3):1029–1039

    Article  Google Scholar 

  35. Murdoch TB, Detsky AS (2013) The inevitable application of big data to health care. Jama 309(13):1351–1352

    Article  CAS  Google Scholar 

  36. Raghupathi W, Raghupathi V (2014) Big data analytics in healthcare: promise and potential. Health Inf Sci Syst 2(1):3

    Article  Google Scholar 

  37. Kenngott HG et al (2016) Intelligent operating room suite: from passive medical devices to the self-thinking cognitive surgical assistant. Chirurg 87(12):1033–1038

    Article  CAS  Google Scholar 

Download references

Author information

Author notes

  1. Karl-Friedrich Kowalewski and Carly R. Garrow contributed equally to the manuscript.

Authors and Affiliations

  1. Department of General, Visceral, and Transplantation Surgery, University of Heidelberg, Im Neuenheimer Feld 110, 69120, Heidelberg, Germany

    Karl-Friedrich Kowalewski, Carly R. Garrow, Mona W. Schmidt, Beat P. Müller-Stich & Felix Nickel

  2. Department of Medical Biometry and Informatics, University of Heidelberg, Im Neuenheimer Feld 130.3, 69120, Heidelberg, Germany

    Laura Benner

Authors
  1. Karl-Friedrich Kowalewski
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carly R. Garrow
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mona W. Schmidt
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Laura Benner
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Beat P. Müller-Stich
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Felix Nickel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Felix Nickel.

Ethics declarations

Disclosure

Felix Nickel reports receiving travel support for conference participation as well as equipment provided for laparoscopic surgery courses by KARL STORZ, Johnson & Johnson, Intuitive, and Medtronic. Karl-Friedrich Kowalewski, Carly R. Garrow, Mona W. Schmidt, Laura Benner and Beat Müller-Stich have no conflicts of interest or financial ties to disclose.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kowalewski, KF., Garrow, C.R., Schmidt, M.W. et al. Sensor-based machine learning for workflow detection and as key to detect expert level in laparoscopic suturing and knot-tying. Surg Endosc 33, 3732–3740 (2019). https://doi.org/10.1007/s00464-019-06667-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-019-06667-4

Keywords

Search

Navigation