Abstract
We propose a new encoder-decoder framework based on transformer for electron microscopy (EM) image registration. Current image registration methods mostly require long running time and complicated parameter tuning or cannot model the correlation of pixels between adjacent images. In this paper, we propose an encoder-decoder framework based on transformer to consider the relationship of serial images. In the transformer encoder, we model the long-range dependencies of multiple reference images and focus on their relevant regions by self-attention mechanism. Then in the transformer decoder, we predict the feature of the undeformed source image by introducing a prediction query to interact with the features from the encoder. To our best knowledge, our method is the first to apply transformers to EM image registration tasks, which needs no extra parameter tuning and can produce more accurate deformation fields. Evaluated on two datasets, Cremi and FIB25, our method outperforms state-of-the-art methods with more precise registration results and competitive speed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Miccai 2017 challenge website. https://cremi.org/
Arganda-Carreras, I., Sorzano, C.O.S., Marabini, R., Carazo, J.M., Ortiz-de-Solorzano, C., Kybic, J.: Consistent and elastic registration of histological sections using vector-spline regularization. In: Beichel, R.R., Sonka, M. (eds.) CVAMIA 2006. LNCS, vol. 4241, pp. 85–95. Springer, Heidelberg (2006). https://doi.org/10.1007/11889762_8
Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. ar**v preprint ar**v:1607.06450 (2016)
Bailoni, A., Pape, C., Wolf, S., Beier, T., Kreshuk, A., Hamprecht, F.A.: A generalized framework for agglomerative clustering of signed graphs applied to instance segmentation. ar**v preprint ar**v:1906.11713 (2019)
Bay, H., Tuytelaars, T., Van Gool, L.: SURF: speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023_32
Bookstein, F.L.: Principal warps: thin-plate splines and the decomposition of deformations. IEEE Trans. Pattern Anal. Mach. Intell. 11(6), 567–585 (1989)
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13
Funke, J., et al.: Large scale image segmentation with structured loss based deep learning for connectome reconstruction. IEEE Trans. Pattern Anal. Mach. Intell. 41(7), 1669–1680 (2018)
Hu, B., Zhou, S., **ong, Z., Wu, F.: Self-recursive contextual network for unsupervised 3d medical image registration. In: Liu, M., Yan, P., Lian, C., Cao, X. (eds.) MLMI 2020. LNCS, vol. 12436, pp. 60–69. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59861-7_7
Hu, X., Kang, M., Huang, W., Scott, M.R., Wiest, R., Reyes, M.: Dual-stream pyramid registration network. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11765, pp. 382–390. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32245-8_43
Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. In: Advances in Neural Information Processing Systems, vol. 28 (2015)
Januszewski, M., et al.: High-precision automated reconstruction of neurons with flood-filling networks. Nat. Methods 15(8), 605–610 (2018)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. ar**v preprint ar**v:1412.6980 (2014)
Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)
Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)
Paszke, A., et al.: Automatic differentiation in pytorch (2017)
Saalfeld, S., Fetter, R., Cardona, A., Tomancak, P.: Elastic volume reconstruction from series of ultra-thin microscopy sections. Nat. Methods 9(7), 717–720 (2012)
Shu, C., **n, T., Zhou, F., Chen, X., Han, H.: Dual networks for high-precision and high-speed registration of brain electron microscopy images. Brain Sci. 10(2), 86 (2020)
Sokooti, H., de Vos, B., Berendsen, F., Lelieveldt, B.P.F., Išgum, I., Staring, M.: Nonrigid image registration using multi-scale 3D convolutional neural networks. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 232–239. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66182-7_27
Takemura, S.Y., et al.: Synaptic circuits and their variations within different columns in the visual system of drosophila. Proc. Nat. Acad. Sci. 112(44), 13711–13716 (2015)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Wang, W., et al.: Pyramid vision transformer: a versatile backbone for dense prediction without convolutions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 568–578 (2021)
Wang, X., Zhang, S., Qing, Z., Shao, Y., Zuo, Z., Gao, C., Sang, N.: OadTR: online action detection with transformers. ar**v preprint ar**v:2106.11149 (2021)
Wu, G., Kim, M., Wang, Q., Munsell, B.C., Shen, D.: Scalable high-performance image registration framework by unsupervised deep feature representations learning. IEEE Trans. Biomed. Eng. 63(7), 1505–1516 (2015)
Yoo, I., Hildebrand, D.G.C., Tobin, W.F., Lee, W.-C.A., Jeong, W.-K.: ssEMnet: serial-section electron microscopy image registration using a spatial transformer network with learned features. In: Cardoso, M.J., et al. (eds.) DLMIA/ML-CDS -2017. LNCS, vol. 10553, pp. 249–257. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67558-9_29
Zheng, Z., et al.: A complete electron microscopy volume of the brain of adult drosophila melanogaster. Cell 174(3), 730-743.e22 (2018)
Zhou, S., et al.: Fast and accurate electron microscopy image registration with 3D convolution. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 478–486. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32239-7_53
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Feng, F., Zhang, T., Sun, R., He, J., **ong, Z., Wu, F. (2023). Electron Microscopy Image Registration with Transformers. In: Tanveer, M., Agarwal, S., Ozawa, S., Ekbal, A., Jatowt, A. (eds) Neural Information Processing. ICONIP 2022. Lecture Notes in Computer Science, vol 13625. Springer, Cham. https://doi.org/10.1007/978-3-031-30111-7_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-30111-7_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-30110-0
Online ISBN: 978-3-031-30111-7
eBook Packages: Computer ScienceComputer Science (R0)