Abstract
Deformable image registration is a fundamental problem in computer vision and medical image computing. In this paper we investigate the use of graphical models in the context of a particular type of image registration problem, known as slice-to-volume registration. We introduce a scalable, modular and flexible formulation that can accommodate low-rank and high order terms, that simultaneously selects the plane and estimates the in-plane deformation through a single shot optimization approach. The proposed framework is instantiated into different variants seeking either a compromise between computational efficiency (soft plane selection constraints and approximate definition of the data similarity terms through pair-wise components) or exact definition of the data terms and the constraints on the plane selection. Simulated and real-data in the context of ultrasound and magnetic resonance registration (where both framework instantiations as well as different optimization strategies are considered) demonstrate the potentials of our method.
Similar content being viewed by others
References
Andres, B., Kappes, J. H., Beier, T., Köthe, U., & Hamprecht, F. A. (2012). The lazy flipper: Efficient depth-limited exhaustive search in discrete graphical models. In A. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato & C. Schmid (Eds.), Computer vision (ECCV 2012) (pp. 154–166). Berlin: Springer.
Baudin, P. Y., Goodman, D., Kumar, P., Azzabou, N., Carlier, P. G., Paragios, N., et al. (2013). Discriminative parameter estimation for random walks segmentation. In K. Mori, I. Sakuma, Y. Sato, C. Barillot & N. Navab (Eds.), Medical image computing and computer-assisted intervention (MICCAI 2013) (pp. 219–226). Berlin: Springer.
Besag, J. (1986). On the statistical analysis of dirty pictures. Journal of the Royal Statistical Society Series B (Methodological), 48, 259–302.
Birkfellner, W., Figl, M., Kettenbach, J., Hummel, J., Homolka, P., Schernthaner, R., et al. (2007). Rigid 2D/3D slice-to-volume registration and its application on fluoroscopic CT images. Medical Physics, 34(1), 246. doi:10.1118/1.2401661.
Chandler, A. G., Pinder, R. J., Netsch, T., Schnabel, J. A., Hawkes, D. J., Hill, D. L., et al. (2008). Correction of misaligned slices in multi-slice MR cardiac examinations by using slice-to-volume registration. Journal of Cardiovascular Magnetic Resonance, 2008(10), 13.
Eresen, A., Li, P., & Ji, J. X. (2014). Correlating 2D histological slice with 3D MRI image volume using smart phone as an interactive tool for muscle study. In 2014 36th Annual international conference of the IEEE engineering in medicine and biology society (EMBC) (pp. 6418–6421). IEEE.
Fei, B., Duerk, J. L., & Wilson, D. L. (2002). Automatic 3D registration for interventional MRI-guided treatment of prostate cancer. Computer Aided Surgery, 7(5), 257–267.
Ferrante, E., Fecamp, V., & Paragios, N. (2015a). Implicit planar and in-plane deformable map** in medical images through high order graphs. In IEEE international symposium on biomedical imaging: From nano to macro (ISBI).
Ferrante, E., Fecamp, V., & Paragios, N. (2015b). Slice-to-volume deformable registration: Efficient one shot consensus between plane selection and in-plane deformation. International Journal of Computer Assisted Radiology and Surgery (IJCARS), 10(6), 16.
Ferrante, E., & Paragios, N. (2013). Non-rigid 2D–3D medical image registration using Markov random fields. In Medical image computing and computer-assisted intervention (MICCAI 2013) (pp. 163–170). Berlin: Springer.
Ferrante, E., & Paragios, N. (2017). Slice-to-volume medical image registration: A survey. Medical Image Analysis, 39, 101–123.
Gill, S., Abolmaesumi, P., Vikal, S., Mousavi, P., & Fichtinger, G. (2008). Intraoperative prostate tracking with slice-to-volume registration in MRI. In Proceedings of the 20th international conference of the society for medical innovation and technology 2008 (pp. 154–158). Society for Medical Innovation and Technology SMIT.
Glocker, B. (2010). Random fields for image registration. PhD Thesis.
Glocker, B., Komodakis, N., Paragios, N., & Navab, N. (2009). Approximated curvature penalty in non-rigid registration using pairwise MRFs. In G. Bebis, R. Boyle, B. Parvin, D. Koracin, Y. Kuno, J. Wang, et al. (Eds.), Advances in visual computing (pp. 1101–1109). Berlin: Springer.
Glocker, B., Komodakis, N., Tziritas, G., Navab, N., & Paragios, N. (2008). Dense image registration through MRFs and efficient linear programming. Medical Image Analysis, 12(6), 731–741. doi:10.1016/j.media.2008.03.006. Special issue on information processing in medical imaging 2007.
Glocker, B., Sotiras, A., Komodakis, N., & Paragios, N. (2011). Deformable medical image registration: Setting the state of the art with discrete methods. Annual Review of Biomedical Engineering, 13, 219–244. doi:10.1146/annurev-bioeng-071910-124649.
Huang, X., Moore, J., Guiraudon, G., Jones, D. L., Bainbridge, D., Ren, J., et al. (2009). Dynamic 2D ultrasound and 3D ct image registration of the beating heart. IEEE Transactions on Medical Imaging, 28(8), 1179–1189.
Jiang, S., Xue, H., Counsell, S., Anjari, M., Allsop, J., Rutherford, M., et al. (2009). Diffusion tensor imaging (DTI) of the brain in moving subjects: Application to in-utero fetal and ex-utero studies. Magnetic Resonance in Medicine, 62(3), 645–655.
Kappes, J. H., Andres, B., Hamprecht, F. A., Schnörr, C., Nowozin, S., Batra, D., et al. (2013). A comparative study of modern inference techniques for discrete energy minimization problems. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2013 (pp. 1328–1335).
Kim, B., Boes, J. L., Bland, P. H., Chenevert, T. L., & Meyer, C. R. (1999). Motion correction in fMRI via registration of individual slices into an anatomical volume. Magnetic Resonance in Medicine, 41(5), 964–972.
Kohli, P., & Rother, C. (2012). Higher-order models in computer vision. Chapter 1, Image processing and analysis with graphs. Boca Raton: CRC Press.
Komodakis, N., Paragios, N., & Tziritas, G. (2011). MRF energy minimization and beyond via dual decomposition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(3), 531–552.
Komodakis, N., Tziritas, G., & Paragios, N. (2007). Fast, approximately optimal solutions for single and dynamic MRFs. In IEEE conference on computer vision and pattern recognition, 2007 (CVPR’07) (pp. 1–8). IEEE. http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=4269955.
Komodakis, N., **ang, B., & Paragios, N. (2015). A framework for efficient structured max-margin learning of high-order MRF models. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 37(7), 1425–1441.
Kschischang, F. R., Frey, B. J., & Loeliger, H. A. (2001). Factor graphs and the sum-product algorithm. IEEE Transactions on Information Theory, 47(2), 498–519.
Kwon, D., Lee, K. J., Yun, I. D., & Lee, S. U. (2008). Nonrigid image registration using dynamic higher-order MRF model. In D. Forsyth, P. Torr & A. Zisserman (Eds.), Computer Vision (ECCV 2008) (pp. 373–386). Berlin: Springer.
Leung, K. Y. E., van Stralen, M., Nemes, A., Voormolen, M. M., van Burken, G., Geleijnse, M. L., et al. (2008). Sparse registration for three-dimensional stress echocardiography. IEEE Transactions on Medical Imaging, 27(11), 1568–1579.
Liao, R., Zhang, L., Sun, Y., Miao, S., & Chefd’Hotel, C. (2013). A review of recent advances in registration techniques applied to minimally invasive therapy. IEEE Transactions on Multimedia, 15(5), 983–1000.
Markelj, P., Tomaževič, D., Likar, B., & Pernuš, F. (2012). A review of 3D/2D registration methods for image-guided interventions. Medical Image Analysis, 16(3), 642–661.
Mercier, L., Del Maestro, R. F., Petrecca, K., Araujo, D., Haegelen, C., & Collins, D. L. (2012). Online database of clinical MR and ultrasound images of brain tumors. Medical Physics, 39, 3253.
Murphy, K. P., Weiss, Y., & Jordan, M. I. (1999). Loopy belief propagation for approximate inference: An empirical study. In Proceedings of the fifteenth conference on uncertainty in artificial intelligence (UAI’99).
Nelder, J. A., & Mead, R. (1965). A simplex method for function minimization. The Computer Journal, 7(4), 308–313. doi:10.1093/comjnl/7.4.308.
Olesch, J., Beuthien, B., Heldmann, S., Papenberg, N., & Fischer, B. (2011). Fast intra-operative non-linear registration of 3D-ct to tracked, selected 2D-ultrasound slices. In SPIE medical imaging (pp. 79642R–79642R). International Society for Optics and Photonics.
Osechinskiy, S., & Kruggel, F. (2011). Slice-to-volume nonrigid registration of histological sections to MR images of the human brain. Anatomy Research International, 2011, 1–17. doi:10.1155/2011/287860.
Paragios, N., Ferrante, E., Glocker, B., Komodakis, N., Parisot, S., & Zacharaki, E. I. (2016). (Hyper)-graphical models in biomedical image analysis. Medical Image Analysis, 33, 102–106.
Paragios, N., Komodakis, N. (2014). Discrete visual perception. In 2014 22nd International conference on pattern recognition (ICPR) (pp. 18–25). IEEE.
Porchetto, R., Stramana, F., Paragios, N., & Ferrante, E. (2016). Rigid slice-to-volume medical image registration through Markov random fields. BAMBI Workshop, MICCAI 2016.
Ramalingam, S., Kohli, P., Alahari, K., & Torr, P. H. (2008). Exact inference in multi-label crfs with higher order cliques. In IEEE conference on computer vision and pattern recognition, 2008 (CVPR 2008) (pp. 1–8). IEEE.
Rueckert, D., Sonoda, L. I., Hayes, C., Hill, D. L., Leach, M. O., & Hawkes, D. J. (1999). Nonrigid registration using free-form deformations: Application to breast MR images. IEEE Transactions on Medical Imaging, 18(8), 712–721.
San José Estépar, R., Westin, C., & Vosburgh, K. (2009). Towards real time 2D to 3D registration for ultrasound-guided endoscopic and laparoscopic procedures. International Journal of Computer Assisted Radiology and Surgery, 4(6), 549–560.
Seshamani, S., Fogtmann, M., Cheng, X., Thomason, M., Gatenby, C., & Studholme, C. (2013). Cascaded slice to volume registration for moving fetal fMRI. In 2013 IEEE 10th international symposium on biomedical imaging (ISBI) (pp. 796–799). IEEE.
Shekhovtsov, A., Kovtun, I., & Hlaváč, V. (2008). Efficient MRF deformation model for non-rigid image matching. Computer Vision and Image Understanding, 112(1), 91–99.
Sotiras, A., Davatzikos, C., & Paragios, N. (2013). Deformable medical image registration: A survey. IEEE Transactions on Medical Imaging, 32(7), 1153–1190.
Sotiras, A., Ou, Y., Glocker, B., Davatzikos, C., & Paragios, N. (2010). Simultaneous geometric–iconic registration. In T. Jiang, N. Navab, J. P. W. Pluim & M. A. Viergever (Eds.), Medical image computing and computer-assisted intervention (MICCAI 2010) (pp. 676–683). Berlin: Springer.
Wang, C., Komodakis, N., & Paragios, N. (2013). Markov random field modeling, inference & learning in computer vision & image understanding: A survey. Computer Vision and Image Understanding, 117(11), 1610–1627.
Xu, H., Lasso, A., Fedorov, A., Tuncali, K., Tempany, C., & Fichtinger, G. (2014). Multi-slice-to-volume registration for MRI-guided transperineal prostate biopsy. International Journal of Computer Assisted Radiology and Surgery, 10(5), 1–10.
Zikic, D., Glocker, B., Kutter, O., Groher, M., Komodakis, N., Kamen, A., et al. (2010). Linear intensity-based image registration by Markov random fields and discrete optimization. Medical Image Analysis, 14(4), 550–562.
Acknowledgements
This research was partially supported by European Research Council Starting Grant Diocles (ERC-STG-259112). We thank Mihir Sahasrabudhe for proof-reading the paper, and Puneet Kumar Dokania, Vivien Fecamp and Jorg Kappes for helpful discussions.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Ron Kimmel.
Images in this work are better viewed in color.
Appendix 1: Continuous Slice-to-Volume Registration
Appendix 1: Continuous Slice-to-Volume Registration
In this appendix, we include a brief comparative study among alternative continuous models for slice-to-volume registration. Comparison is performed using the monomodal heart dataset (see Sect. 3.2.1 for a complete description). The aim of this experiment was to choose the most accurate method for deformable registration which (together with the standard rigid model) was then used as baseline for comparison with the discrete approaches proposed in this work (see Sect. 3.2).
Following the literature on slice-to-volume registration (for a complete survey on slice-to-volume registration see Ferrante and Paragios (2017)), we adopted a decoupled model where the transformation consists in a global 6-DOF rigid transformation for plane selection, and a 2D FFD to represent the deformation field. To account for smooth deformations, the FFD is regularized using the Jacobian of the deformation field, a common regularizer used in the deformable image registration community (Sotiras et al. 2013). Optimization was performed through the continuous Nelder-Mead simplex algorithm described in Sect. 3.2, adopted in many slice-to-volume registration studies (see Section 4.1.2 of Ferrante and Paragios 2017). The grid resolution for the 2D FFDs was set to be equivalent to the resolutions used in the discrete experiments (see Sects. 3.2.1, 3.2.2). We run simplex optimization until convergence or until a maximum of 10,000 simplex iterations were achieved. For the rigid model, convergence was always reached in a few seconds. In case of the deformable models, the algorithm did not converge in all the cases, achieving maximum running times of around 40 s for 10,000 iterations. We experimented with more iterations (100,000) but we did not reach significant improvements in the results.
Figure 16 summarizes the results for the comparative study including a simple 6-DOF rigid transformation and three variants of a decoupled model with a global 6-DOF rigid transformation and a 2D FFD. As it can be observed, the Cont Def-Two Steps outperforms the other models. That is why it was chosen as baseline for comparison with the discrete approaches proposed in this work.
Rights and permissions
About this article
Cite this article
Ferrante, E., Paragios, N. Graph-Based Slice-to-Volume Deformable Registration. Int J Comput Vis 126, 36–58 (2018). https://doi.org/10.1007/s11263-017-1040-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-017-1040-8