Abstract
Purpose
We propose a framework for automatic and accurate detection of steeply inserted needles in 2D ultrasound data using convolution neural networks. We demonstrate its application in needle trajectory estimation and tip localization.
Methods
Our approach consists of a unified network, comprising a fully convolutional network (FCN) and a fast region-based convolutional neural network (R-CNN). The FCN proposes candidate regions, which are then fed to a fast R-CNN for finer needle detection. We leverage a transfer learning paradigm, where the network weights are initialized by training with non-medical images, and fine-tuned with ex vivo ultrasound scans collected during insertion of a 17G epidural needle into freshly excised porcine and bovine tissue at depth settings up to 9 cm and \(40^{\circ }\)–\(75^{\circ }\) insertion angles. Needle detection results are used to accurately estimate needle trajectory from intensity invariant needle features and perform needle tip localization from an intensity search along the needle trajectory.
Results
Our needle detection model was trained and validated on 2500 ex vivo ultrasound scans. The detection system has a frame rate of 25 fps on a GPU and achieves 99.6% precision, 99.78% recall rate and an \({F}_{1}\) score of 0.99. Validation for needle localization was performed on 400 scans collected using a different imaging platform, over a bovine/porcine lumbosacral spine phantom. Shaft localization error of \(0.82^{\circ }\pm 0.4^{\circ }\), tip localization error of \(0.23\pm 0.05\) mm, and a total processing time of 0.58 s were achieved.
Conclusion
The proposed method is fully automatic and provides robust needle localization results in challenging scanning conditions. The accurate and robust results coupled with real-time detection and sub-second total processing make the proposed method promising in applications for needle detection and localization during challenging minimally invasive ultrasound-guided procedures.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-018-1721-y/MediaObjects/11548_2018_1721_Fig1_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-018-1721-y/MediaObjects/11548_2018_1721_Fig2_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-018-1721-y/MediaObjects/11548_2018_1721_Fig3_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-018-1721-y/MediaObjects/11548_2018_1721_Fig4_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-018-1721-y/MediaObjects/11548_2018_1721_Fig5_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-018-1721-y/MediaObjects/11548_2018_1721_Fig6_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-018-1721-y/MediaObjects/11548_2018_1721_Fig7_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-018-1721-y/MediaObjects/11548_2018_1721_Fig8_HTML.jpg)
Similar content being viewed by others
References
Moore J, Clarke C, Bainbridge D, Wedlake C, Wiles A, Pace D, Peters T (2009) Image guidance for spinal facet injections using tracked ultrasound. Med Image Comput Comput Assist Interv 12(Pt 1):516–523
**a W, West S, Finlay M, Mari J, Ourselin S, David A, Desjardins A (2017) Looking beyond the imaging plane: 3D needle tracking with a linear array ultrasound probe. Sci Rep 7(1):3674
Ayvali E, Desai J (2014) Optical flow-based tracking of needles and needle-tip localization using circular hough transform in ultrasound images. Ann Biomed Eng 43(8):1828–1840
Barva M, Uhercik M, Mari JM, Kybic J, Duhamel JR, Liebgott H, Hlavac V, Cachard C (2008) Parallel integral projection transform for straight electrode localization in 3-D ultrasound images. IEEE Trans Ultrason Ferroelectr Freq Control 55(7):1559–1569
Mathiassen K, Dall’Alba D, Muradore R, Fiorini P, Elle O (2017) Robust real-time needle tracking in 2-D ultrasound images using statistical filtering. IEEE Trans Control Syst Technol 25(3):966–978
Beigi P, Rohling R, Salcudean S, Ng G (2017) CASPER: computer-aided segmentation of imperceptible motion-a learning-based tracking of an invisible needle in ultrasound. IJCARS 12(11):1857–1866
Hatt CR, Ng G, Parthasarathy V (2015) Enhanced needle localization in ultrasound using beam steering and learning-based segmentation. Comput Med Imaging Graph 41:46–54
Mwikirize C, Nosher JL, Hacihaliloglu I (2018) Signal attenuation maps for needle enhancement and localization in 2D ultrasound. Int J CARS 13(3):363–374
Hacihaliloglu I, Beigi P, Ng G, Rohling RN, Salcudean S, Abolmaesumi P (2015) Projection-based phase features for localization of a needle tip in 2D curvilinear ultrasound. In: Medical image computing and computer-assisted intervention. Lecture notes in computer science, vol 9349. Springer, pp 347–54
Litjens G, Kooi T, Bejnordi B, Setio A, Ciompi F, Ghafoorian M, van der Laak J, van Ginneken B, Snchez C (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88
Pourtaherian A, Zanjani FG, Zinger S, Mihajlovic N, Ng G, Korsten H, de With P (2017) Improving needle detection in 3D ultrasound using orthogonal-plane convolutional networks. In: Medical image computing and computer-assisted intervention. Lecture notes in computer science, vol 10434. Springer, pp 610–618
Ren S, He K, Girshick R, Sun J (2017) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149
Girshick R (2015) Fast R-CNN. In: Proceedings of IEEE international conference on computer vision, pp 1440–148
Krizhevsky A, Sutskever I, Hinton G (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90
Simonyan K, Zisserman A (2015) Very deep convolutional network for large-scale image recognition. In: Proceedings of international conference on learning representations
Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional neural networks. In: Proceedings of 13th European conference on computer vision, vol 81833
Krizhevsky A (2009) Learning multiple layers of features from tiny images. CIFAR10 dataset
Huynh B, Li H, Giger M (2016) Digital mammographic tumor classification using transfer learning from deep convolutional neural networks. J Med Imaging 3(3):034501
Torr PHS, Zisserman A (2000) MLESAC: a new robust estimator with application to estimating image geometry. J Comput Vis Image Underst 78(1):138–156
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Informed consent
This article does not contain patient data.
Rights and permissions
About this article
Cite this article
Mwikirize, C., Nosher, J.L. & Hacihaliloglu, I. Convolution neural networks for real-time needle detection and localization in 2D ultrasound. Int J CARS 13, 647–657 (2018). https://doi.org/10.1007/s11548-018-1721-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11548-018-1721-y