Abstract
Deep learning approaches are state-of-the-art for semantic segmentation of medical images, but unlike many deep learning applications, medical segmentation is characterized by small amounts of annotated training data. Thus, while mainstream deep learning approaches focus on performance in domains with large training sets, researchers in the medical imaging field must apply new methods in creative ways to meet the more constrained requirements of medical datasets. We propose a framework for incrementally fine-tuning a multi-class segmentation of a high-resolution multiplex (multi-channel) immuno-flourescence image of a rat brain section, using a minimal amount of labelling from a human expert. Our framework begins with a modified Swin-UNet architecture that treats each biomarker in the multiplex image separately and learns an initial “global” segmentation (pre-training). This is followed by incremental learning and refinement of each class using a very limited amount of additional labeled data provided by a human expert for each region and its surroundings. This incremental learning utilizes the multi-class weights as an initialization and uses the additional labels to steer the network and optimize it for each region in the image. In this way, an expert can identify errors in the multi-class segmentation and rapidly correct them by supplying the model with additional annotations hand-picked from the region. In addition to increasing the speed of annotation and reducing the amount of labelling, we show that our proposed method outperforms a traditional multi-class segmentation by a large margin.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs12021-024-09651-z/MediaObjects/12021_2024_9651_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs12021-024-09651-z/MediaObjects/12021_2024_9651_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs12021-024-09651-z/MediaObjects/12021_2024_9651_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs12021-024-09651-z/MediaObjects/12021_2024_9651_Fig4_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs12021-024-09651-z/MediaObjects/12021_2024_9651_Fig5_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs12021-024-09651-z/MediaObjects/12021_2024_9651_Fig6_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs12021-024-09651-z/MediaObjects/12021_2024_9651_Fig7a_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs12021-024-09651-z/MediaObjects/12021_2024_9651_Fig7b_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs12021-024-09651-z/MediaObjects/12021_2024_9651_Fig8_HTML.png)
Similar content being viewed by others
Data Availability
Dataset was collected following the procedure described in this paper: https://www.nature.com/articles/s41467-021-21735-x The dataset is available at the following link: https://figshare.com/articles/dataset/Whole-brain_tissue_map**_toolkit_using_large-scale_highly_multiplexed_immunofluorescence_imaging_and_deep_neural_networks_Data_/13731585/1.
References
Azad, R., Aghdam, E. K., Rauland, A., Jia, Y., Avval, A. H., Bozorgpour, A., Karimijafarbigloo, S., Cohen, J. P., Adeli, E., & Merhof, D. (2022). Medical image segmentation review: the success of U-Net. ar** toolkit using large-scale highly multiplexed immunofluorescence imaging and deep neural networks. Nature Communications, 12, 1550 (2021). https://doi.org/10.1038/s41467-021-21735-x. Accessed September 28, 2023.
napari. (2023). A fast, interactive viewer for multi-dimensional images in Python napari. Accessed September 30, 2023, from https://napari.org/stable/
Oktay, O., Schlemper, J., Folgoc, L. L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N. Y., Kainz, B., Glocker, B., & Rueckert, D. (2022). Attention U-Net: learning where to look for the pancreas. Accessed September 29, 2023, from https://openreview.net/forum?id=Skft7cijM
Paxinos, G., & Watson, C. (2013). The rat brain in stereotaxic coordinates. Academic Press. Google-Books-ID: RiHLCQAAQBAJ.
Phellan, R., Lindner, T., Helle, M., Falco, A. X., & Forkert, N. D. (2018). Automatic temporal segmentation of vessels of the brain using 4D ASL MRA Images. IEEE Transactions on Biomedical Engineering, 65(7), 1486–1494. https://doi.org/10.1109/TBME.2017.2759730. Conference Name: IEEE Transactions on Biomedical Engineering. Accessed November 16, 2023.
Rahimpour, M., Bertels, J., Radwan, A., Vandermeulen, H., Sunaert, S., Vandermeulen, D., Maes, F., Goffin, K., & Koole, M. (2022). Cross-modal distillation to improve MRI-based brain tumor segmentation with missing MRI sequences. IEEE Transactions on Biomedical Engineering, 69(7), 2153–2164. https://doi.org/10.1109/TBME.2021.3137561. Conference Name: IEEE Transactions on Biomedical Engineering. Accessed November 16, 2023.
Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Lecture Notes in Computer Science, pp. 234–241. Springer, Cham. https://doi.org/10.1007/978-3-319-24574-4_28
Stringer, C., Wang, T., Michaelos, M., & Pachitariu, M. (2021). Cellpose: a generalist algorithm for cellular segmentation. Nature Methods, 18(1), 100–106. https://doi.org/10.1038/s41592-020-01018-x. Number: 1 Publisher: Nature Publishing Group. Accessed September 28, 2023.
Tang, X., Wang, X., Yan, N., Fu, S., **ong, W., & Liao, Q. (2022). A new ore image segmentation method based on Swin-Unet. In: 2022 China Automation Congress (CAC), pp. 1681–1686. https://doi.org/10.1109/CAC57257.2022.10055952. ISSN: 2688-0938. Accessed September 29, 2023, from https://ieeexplore.ieee.org/document/10055952
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jegou, H. (2021). Training data-efficient image transformers & distillation through attention. In: Proceedings of the 38th International Conference on Machine Learning, pp. 10347–10357. PMLR. ISSN: 2640-3498. Accessed September 30, 2023, from https://proceedings.mlr.press/v139/touvron21a.html
Valverde, J. M., Shatillo, A., De Feo, R., & Tohka, J. (2023). Automatic cerebral hemisphere segmentation in Rat MRI with ischemic lesions via attention-based convolutional neural networks. Neuroinformatics, 21(1), 57–70. https://doi.org/10.1007/s12021-022-09607-1. Accessed December 19, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, U., & Polosukhin, I. (2017). Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. Accessed September 29, 2023, from https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
Wang, X., Girshick, R., Gupta, A., & He, K. (2018). Non-local Neural Networks. ar**v. ar**v:1711.07971 [cs]. https://doi.org/10.48550/ar**v.1711.07971. http://arxiv.org/abs/1711.07971. Accessed September 30, 2023.
Wang, Y., Gu, L., Jiang, T., & Gao, F. (2023). MDE-UNet: A multitask deformable UNet combined enhancement network for farmland boundary segmentation. IEEE Geoscience and Remote Sensing Letters, 20, 1–5. https://doi.org/10.1109/LGRS.2023.3252048. Conference Name: IEEE Geoscience and Remote Sensing Letters. Accessed September 29, 2023.
Wang, W., **e, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021). Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 548–558. https://doi.org/10.1109/ICCV48922.2021.00061. ISSN: 2380-7504. Accessed September 29, 2023, from https://ieeexplore.ieee.org/document/9711179
Wang, W., **e, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2022). PVT v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3), 415–424. https://doi.org/10.1007/s41095-022-0274-8. ar**v:2106.13797 [cs]. Accessed January 1, 2024.
**ao, T., Liu, Y., Zhou, B., Jiang, Y., & Sun, J. (2018). Unified perceptual parsing for scene understanding. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. Lecture Notes in Computer Science, pp. 432–448. Springer, Cham. https://doi.org/10.1007/978-3-030-01228-1_26
Yin, M., Yao, Z., Cao, Y., Li, X., Zhang, Z., Lin, S., & Hu, H. (2020). Disentangled non-local neural networks. In: Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV, pp. 191–207. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-030-58555-6_12. Accessed September 29, 2023.
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F.E., Feng, J., & Yan, S. (2021). Tokens-to-Token ViT: Training vision transformers from scratch on ImageNet. ar**v. ar**v:2101.11986 [cs]. https://doi.org/10.48550/ar**v.2101.11986. http://arxiv.org/abs/2101.11986. Accessed September 30, 2023.
Zhang, Y., Liu, H., & Hu, Q. (2021). TransFuse: Fusing transformers and CNNs for medical image segmentation. In: Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., Essert, C. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. Lecture Notes in Computer Science, pp. 14–24. Springer, Cham. https://doi.org/10.1007/978-3-030-87193-2_2
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., **ang, T., Torr, P. H. S., & Zhang, L. (2021). Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6877–6886. https://doi.org/10.1109/CVPR46437.2021.00681. ISSN: 2575-7075. Accessed September 29, 2023, from https://ieeexplore.ieee.org/document/9578646
Zhou, Z., Rahman Siddiquee, M. M., Tajbakhsh, N., & Liang, J. (2018). UNet++: A nested U-Net architecture for medical image segmentation. In: Stoyanov, D., Taylor, Z., Carneiro, G., Syeda-Mahmood, T., Martel, A., Maier-Hein, L., Tavares, J.M.R.S., Bradley, A., Papa, J.P., Belagiannis, V., Nascimento, J.C., Lu, Z., Conjeti, S., Moradi, M., Greenspan, H., Madabhushi, A. (eds.) Deep learning in medical image analysis and multimodal learning for clinical decision support. Lecture Notes in Computer Science, pp. 3–11. Springer, Cham. https://doi.org/10.1007/978-3-030-00889-5_1
Funding
National Institutes of Health, Grant No. 5R01NS109118.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception. R.F. conducted the experiments, generated the results and wrote the manuscript. S.P. supervised the research. B.R. and D.M. provided context on the neuroscience tasks. All authors read and approved the final version of the manuscript.
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Faulkenberry, R., Prasad, S., Maric, D. et al. Visual Prompting Based Incremental Learning for Semantic Segmentation of Multiplex Immuno-Flourescence Microscopy Imagery. Neuroinform 22, 147–162 (2024). https://doi.org/10.1007/s12021-024-09651-z
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12021-024-09651-z