SpringerLink
Log in

The development of a deep learning model for automated segmentation of the robotic pancreaticojejunostomy

  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Background

Minimally invasive surgery provides an unprecedented opportunity to review video for assessing surgical performance. Surgical video analysis is time-consuming and expensive. Deep learning provides an alternative for analysis. Robotic pancreaticoduodenectomy (RPD) is a complex and morbid operation. Surgeon technical performance of pancreaticojejunostomy (PJ) has been associated with postoperative pancreatic fistula. In this work, we aimed to utilize deep learning to automatically segment PJ RPD videos.

Methods

This was a retrospective review of prospectively collected videos from 2011 to 2022 that were in libraries at tertiary referral centers, including 111 PJ videos. Each frame of a robotic PJ video was categorized based on 6 tasks. A 3D convolutional neural network was trained for frame-level visual feature extraction and classification. All the videos were manually annotated for the start and end of each task.

Results

Of the 100 videos assessed, 60 videos were used for the training the model, 10 for hyperparameter optimization, and 30 for the testing of performance. All the frames were extracted (6 frames/second) and annotated. The accuracy and mean per-class F1 scores were 88.01% and 85.34% for tasks.

Conclusion

The deep learning model performed well for automated segmentation of PJ videos. Future work will focus on skills assessment and outcome prediction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (France)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Gietelink L, Wouters MW, Bemelman WA, Dekker JW, Tollenaar RA, Tanis PJ et al (2016) Reduced 30-day mortality after laparoscopic colorectal cancer surgery: a population based study from the Dutch Surgical Colorectal Audit (DSCA). Ann Surg 264(1):135–140

    Article  PubMed  Google Scholar 

  2. Semm K (1988) Pelviscopic appendectomy. Dtsch Med Wochenschr 113(1):3–5

    Article  CAS  PubMed  Google Scholar 

  3. Aquina CT, Probst CP, Becerra AZ, Iannuzzi JC, Hensley BJ, Noyes K et al (2016) Missed opportunity: laparoscopic colorectal resection is associated with lower incidence of small bowel obstruction compared to an open approach. Ann Surg 264(1):127–134

    Article  PubMed  Google Scholar 

  4. McCormack K, Scott NW, Go PM, Ross S, Grant AM (2003) Laparoscopic techniques versus open techniques for inguinal hernia repair. Cochrane Database Syst Rev 2003(1):CD001785

    PubMed  PubMed Central  Google Scholar 

  5. Reynolds W Jr (2001) The first laparoscopic cholecystectomy. J Soc Laparoendosc Surg 5(1):89–94

    Google Scholar 

  6. Shaligram A, Unnirevi J, Simorov A, Kothari VM, Oleynikov D (2012) How does the robot affect outcomes? A retrospective review of open, laparoscopic, and robotic Heller myotomy for achalasia. Surg Endosc 26(4):1047–1050

    Article  PubMed  Google Scholar 

  7. Zureikat AH, Postlewait LM, Liu Y, Gillespie TW, Weber SM, Abbott DE et al (2016) A multi-institutional comparison of perioperative outcomes of robotic and open pancreaticoduodenectomy. Ann Surg 264(4):640–649

    Article  PubMed  Google Scholar 

  8. Cameron JL, He J (2015) Two thousand consecutive pancreaticoduodenectomies. J Am Coll Surg 220(4):530–536

    Article  PubMed  Google Scholar 

  9. Cameron JL, Riall TS, Coleman J, Belcher KA (2006) One thousand consecutive pancreaticoduodenectomies. Ann Surg 244(1):10–15

    Article  PubMed  PubMed Central  Google Scholar 

  10. Al Abbas AI, Borrebach JD, Pitt HA, Bellon J, Hogg ME, Zeh HJ 3rd et al (2021) Development of a novel pancreatoduodenectomy-specific risk calculator: an analysis of 10,000 patients. J Gastrointest Surg 25(6):1503–1511

    Article  PubMed  Google Scholar 

  11. Cuschieri A (1994) Laparoscopic surgery of the pancreas. J R Coll Surg Edinb 39(3):178–184

    CAS  PubMed  Google Scholar 

  12. Gagner M, Pomp A (1994) Laparoscopic pylorus-preserving pancreatoduodenectomy. Surg Endosc 8(5):408–410

    Article  CAS  PubMed  Google Scholar 

  13. Kendrick ML, Cusati D (2010) Total laparoscopic pancreaticoduodenectomy: feasibility and outcome in an early experience. Arch Surg 145(1):19–23

    Article  PubMed  Google Scholar 

  14. Palanivelu C, Rajan PS, Rangarajan M, Vaithiswaran V, Senthilnathan P, Parthasarathi R et al (2009) Evolution in techniques of laparoscopic pancreaticoduodenectomy: a decade long experience from a tertiary center. J Hepatobiliary Pancreat Surg 16(6):731–740

    Article  CAS  PubMed  Google Scholar 

  15. Zeh HJ 3rd, Bartlett DL, Moser AJ (2011) Robotic-assisted major pancreatic resection. Adv Surg 45:323–340

    Article  PubMed  Google Scholar 

  16. Zureikat AH, Beane JD, Zenati MS, Al Abbas AI, Boone BA, Moser AJ et al (2021) 500 minimally invasive robotic pancreatoduodenectomies: one decade of optimizing performance. Ann Surg 273(5):966–972

    Article  PubMed  Google Scholar 

  17. Nassour I, Winters SB, Hoehn R, Tohme S, Adam MA, Bartlett DL et al (2020) Long-term oncologic outcomes of robotic and open pancreatectomy in a national cohort of pancreatic adenocarcinoma. J Surg Oncol 122(2):234–242

    Article  PubMed  Google Scholar 

  18. Zeh HJ, Zureikat AH, Secrest A, Dauoudi M, Bartlett D, Moser AJ (2012) Outcomes after robot-assisted pancreaticoduodenectomy for periampullary lesions. Ann Surg Oncol 19(3):864–870

    Article  PubMed  Google Scholar 

  19. Al Abbas AI, Jung JP, Rice MK, Zureikat AH, Zeh HJ 3rd, Hogg ME (2019) Methodology for develo** an educational and research video library in minimally invasive surgery. J Surg Educ 76(3):745–755

    Article  PubMed  Google Scholar 

  20. Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Nunn AR et al (2013) Surgical skill and complication rates after bariatric surgery. N Engl J Med 369(15):1434–1442

    Article  CAS  PubMed  Google Scholar 

  21. Brown JA, Jung JP, Zenati MS, Simmons RL, Al Abbas AI, Hogg ME et al (2021) Video review reveals technical factors predictive of biliary stricture and cholangitis after robotic pancreaticoduodenectomy. HPB 23(1):144–153

    Article  PubMed  Google Scholar 

  22. Hogg ME, Zenati M, Novak S, Chen Y, Jun Y, Steve J et al (2016) Grading of surgeon technical performance predicts postoperative pancreatic fistula for pancreaticoduodenectomy independent of patient-related variables. Ann Surg 264(3):482–491

    Article  PubMed  Google Scholar 

  23. Jung JP, Zenati MS, Dhir M, Zureikat AH, Zeh HJ, Simmons RL et al (2018) Use of video review to investigate technical factors that may be associated with delayed gastric emptying after pancreaticoduodenectomy. JAMA Surg 153(10):918–927

    Article  PubMed  PubMed Central  Google Scholar 

  24. Rice MK, Zenati MS, Novak SM, Al Abbas AI, Zureikat AH, Zeh HJ 3rd et al (2019) Crowdsourced assessment of inanimate biotissue drills: a valid and cost-effective way to evaluate surgical trainees. J Surg Educ 76(3):814–823

    Article  PubMed  Google Scholar 

  25. Hashimoto DA, Rosman G, Rus D, Meireles OR (2018) Artificial intelligence in surgery: promises and perils. Ann Surg 268(1):70–76

    Article  PubMed  Google Scholar 

  26. Hung AJ, Liu Y, Anandkumar A (2021) Deep learning to automate technical skills assessment in robotic surgery. JAMA Surg 156(11):1059–1060

    Article  PubMed  Google Scholar 

  27. Luongo F, Hakim R, Nguyen JH, Anandkumar A, Hung AJ (2021) Deep learning-based computer vision to recognize and classify suturing gestures in robot-assisted surgery. Surgery 169(5):1240–1244

    Article  PubMed  Google Scholar 

  28. Feichtenhofer C (2020) X3D: expanding architectures for efficient video recognition. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 200–210

  29. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN et al (2017) Attention is all you need. Adv Neur Inf Process Syst 30:5998–6008

    Google Scholar 

  30. Pele O, Werman M (2009) Fast and robust earth mover's distances. In: 2009 IEEE 12th international conference on computer vision, pp 460–467

  31. Garrow CR, Kowalewski KF, Li L, Wagner M, Schmidt MW, Engelhardt S et al (2021) Machine learning for surgical phase recognition: a systematic review. Ann Surg 273(4):684–693

    Article  PubMed  Google Scholar 

  32. Ahmadi SA, Sielhorst T, Stauder R, Horn M, Feussner H, Navab N (2006) Recovery of surgical workflow without explicit models. Med Image Comput Comput Assist Interv 9(Pt 1):420–428

    PubMed  Google Scholar 

  33. Sodergren MH, Orihuela-Espina F, Clark J, Darzi A, Yang GZ (2010) A hidden Markov model-based analysis framework using eye-tracking data to characterise re-orientation strategies in minimally invasive surgery. Cogn Process 11(3):275–283

    Article  PubMed  Google Scholar 

  34. Kawka M, Gall TMH, Fang C, Liu R, Jiao LR (2022) Intraoperative video analysis and machine learning models will change the future of surgical training. Intell Surg 1:13–5

    Article  Google Scholar 

  35. ** Y et al (2018) SV-RCNet: workflow recognition from surgical videos using recurrent convolutional network. IEEE Trans Med Imaging 37(5):1114–25

    Article  PubMed  Google Scholar 

  36. Hashimoto DA, Rosman G, Witkowski ER, Stafford C, Navarette-Welton AJ, Rattner DW et al (2019) Computer vision analysis of intraoperative video: automated recognition of operative steps in laparoscopic sleeve gastrectomy. Ann Surg 270(3):414–421

    Article  PubMed  Google Scholar 

  37. Kitaguchi D, Takeshita N, Matsuzaki H, Takano H, Owada Y, Enomoto T et al (2020) Real-time automatic surgical phase recognition in laparoscopic sigmoidectomy using the convolutional neural network-based deep learning approach. Surg Endosc 34(11):4924–4931

    Article  PubMed  Google Scholar 

  38. Morita S, Tabuchi H, Masumoto H, Yamauchi T, Kamiura N (2019) Real-time extraction of important surgical phases in cataract surgery videos. Sci Rep 9(1):16590

    Article  PubMed  PubMed Central  Google Scholar 

  39. Ward TM, Hashimoto DA, Ban Y, Rattner DW, Inoue H, Lillemoe KD et al (2021) Automated operative phase identification in peroral endoscopic myotomy. Surg Endosc 35(7):4008–4015

    Article  PubMed  Google Scholar 

  40. Eppler MB, Sayegh AS, Maas M, Venkat A, Hemal S, Desai MM et al (2023) Automated capture of intraoperative adverse events using artificial intelligence: a systematic review and meta-analysis. J Clin Med 12(4):1687

    Article  PubMed  PubMed Central  Google Scholar 

  41. Shi P, Zhao Z, Liu K, Li F (2022) Attention-based spatial–temporal neural network for accurate phase recognition in minimally invasive surgery: feasibility and efficiency verification. J Comput Des Eng 9(2):406–416

    Google Scholar 

  42. Namazi B, Sankaranarayanan G, Devarajan V (2018) Automatic detection of surgical phases in laparoscopic videos. In: International conference on artificial intelligence (ICAI'18), pp 124–130

Download references

Acknowledgements

The authors thank Dave Primm of the UT Southwestern Department of Surgery for help in editing this article.

Funding

None for this study.

Author information

Author notes

  1. Amr I. Al Abbas and Babak Namazi denotes co-first authors.

Authors and Affiliations

  1. Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9169, USA

    Amr I. Al Abbas, Babak Namazi, Imad Radi, Rodrigo Alterio, Andres A. Abreu, Benjamin Rail, Patricio M. Polanco, Herbert J. Zeh III & Ganesh Sankaranarayanan

  2. NorthShore University HealthSystem, Evanston, IL, USA

    Melissa E. Hogg

  3. University of Pittsburgh Medical Center, Pittsburgh, PA, USA

    Amer H. Zureikat

Authors
  1. Amr I. Al Abbas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Babak Namazi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Imad Radi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rodrigo Alterio
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Andres A. Abreu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Benjamin Rail
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Patricio M. Polanco
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Herbert J. Zeh III
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Melissa E. Hogg
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Amer H. Zureikat
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Ganesh Sankaranarayanan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ganesh Sankaranarayanan.

Ethics declarations

Disclosures

Amr Al Abbas and Ganesh Sankaranarayanan receive funding from the ASE. Ganesh Sankaranarayanan receives additional funding from a National Institutes of Health R01 (EB025247). Melissa Hogg receives funding from SAGES and Intuitive Surgical. Patricio Polanco is supported by the Eugene P. Frenkel Award at the Harold C. Simmon’s Comprehensive Cancer Center at UT Southwestern Medical Center. Herbert Zeh sits on the advisory board of Surgical Safety Technologies. Amr I. Al Abbas, Babak Namazi, Imad Radi, Rodrigo Alterio, Andres Arendu, Benjamin Rail, Patricio M. Polanco, Herbert J. Zeh and Amer Zureikat have no conflicts of interest to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Al Abbas, A.I., Namazi, B., Radi, I. et al. The development of a deep learning model for automated segmentation of the robotic pancreaticojejunostomy. Surg Endosc 38, 2553–2561 (2024). https://doi.org/10.1007/s00464-024-10725-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-024-10725-x

Keywords

Search

Navigation