Abstract
One-shot federated learning (FL) has emerged as a promising solution in scenarios where multiple communication rounds are not practical. Notably, as feature distributions in medical data are less discriminative than those of natural images, robust global model training with FL is non-trivial and can lead to overfitting. To address this issue, we propose a novel one-shot FL framework leveraging Image Synthesis and Client model Adaptation (FedISCA) with knowledge distillation (KD). To prevent overfitting, we generate diverse synthetic images ranging from random noise to realistic images. This approach (i) alleviates data privacy concerns and (ii) facilitates robust global model training using KD with decentralized client models. To mitigate domain disparity in the early stages of synthesis, we design noise-adapted client models where batch normalization statistics on random noise (synthetic images) are updated to enhance KD. Lastly, the global model is trained with both the original and noise-adapted client models via KD and synthetic images. This process is repeated till global model convergence. Extensive evaluation of this design on five small- and three large-scale medical image classification datasets reveals superior accuracy over prior methods. Code is available at https://github.com/myeongkyunkang/FedISCA.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Baradad Jurjo, M., Wulff, J., Wang, T., Isola, P., Torralba, A.: Learning to see by looking at noise. Adv. Neural. Inf. Process. Syst. 34, 2556–2569 (2021)
Chen, H., et al.: Data-free learning of student networks. In: International Conference on Computer Vision, pp. 3514–3522 (2019)
Chikontwe, P., Nam, S.J., Go, H., Kim, M., Sung, H.J., Park, S.H.: Feature re-calibration based multiple instance learning for whole slide image classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 420–430. Springer (2022). https://doi.org/10.1007/978-3-031-16434-7_41
Codella, N.C., et al.: Skin lesion analysis toward melanoma detection: a challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 168–172. IEEE (2018)
Combalia, M., et al.: Bcn20000: dermoscopic lesions in the wild. ar**v preprint ar**v:1908.02288 (2019)
Dennis, D.K., Li, T., Smith, V.: Heterogeneity for the win: one-shot federated clustering. In: International Conference on Machine Learning, pp. 2611–2620. PMLR (2021)
EyePACS: Diabetic retinopathy detection (2015)
Guha, N., Talwalkar, A., Smith, V.: One-shot federated learning. ar**v preprint ar**v:1902.11175 (2019)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. ar**v preprint ar**v:1503.02531 (2015)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456. PMLR (2015)
Jiang, M., Yang, H., Li, X., Liu, Q., Heng, P.A., Dou, Q.: Dynamic bank learning for semi-supervised federated image diagnosis with class imbalance. In: Medical Image Computing and Computer Assisted Intervention. pp. 196–206. Springer (2022). https://doi.org/10.1007/978-3-031-16437-8_19
Jung, E., Luna, M., Park, S.H.: Conditional gan with 3d discriminator for MRI generation of Alzheimer’s disease progression. Pattern Recogn. 133, 109061 (2023)
Kim, S., An, S., Chikontwe, P., Park, S.H.: Bidirectional rnn-based few shot learning for 3d medical image segmentation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 1808–1816 (2021)
Li, Q., He, B., Song, D.: Practical one-shot federated learning for cross-silo setting. In: International Joint Conference on Artificial Intelligence (2020)
Li, Y., Wang, N., Shi, J., Liu, J., Hou, X.: Adaptive batch normalization for practical domain adaptation. Pattern Recogn. 80, 109–117 (2018)
Lin, T., Kong, L., Stich, S.U., Jaggi, M.: Ensemble distillation for robust model fusion in federated learning. Adv. Neural. Inf. Process. Syst. 33, 2351–2363 (2020)
Liu, X., Li, W., Yuan, Y.: Intervention & interaction federated abnormality detection with noisy clients. In: Medical Image Computing and Computer Assisted Intervention, pp. 309–319. Springer (2022). https://doi.org/10.1007/978-3-031-16452-1_30
Liu, Y., Zhang, W., Wang, J., Wang, J.: Data-free knowledge transfer: a survey. ar**v preprint ar**v:2112.15278 (2021a
Mahendran, A., Vedaldi, A.: Understanding deep image representations by inverting them. In: Computer Vision and Pattern Recognition, pp. 5188–5196 (2015)
McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics, pp. 1273–1282. PMLR (2017)
Micaelli, P., Storkey, A.J.: Zero-shot knowledge transfer via adversarial belief matching. In: Advances in Neural Information Processing Systems 32 (2019)
Qi, X., Yang, G., He, Y., Liu, W., Islam, A., Li, S.: Contrastive re-localization and history distillation in federated cmr segmentation. In: Medical Image Computing and Computer Assisted Intervention, pp. 256–265. Springer (2022). https://doi.org/10.1007/978-3-031-16443-9_25
Raikwar, P., Mishra, D.: Discovering and overcoming limitations of noise-engineered data-free knowledge distillation. In: Advances in Neural Information Processing Systems (2022)
RSNA: Rsna pneumonia detection challenge (2018)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2014)
Ogier du Terrail, J., et al.: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings. In: Advances in Neural Information Processing Systems (2022)
Tschandl, P., Rosendahl, C., Kittler, H.: The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data 5(1), 1–9 (2018)
Yang, J., et al.: Medmnist v2: A large-scale lightweight benchmark for 2d and 3d biomedical image classification. ar**v preprint ar**v:2110.14795 (2021)
Yin, H., et al.: Dreaming to distill: data-free knowledge transfer via deepinversion. In: Computer Vision and Pattern Recognition, pp. 8715–8724 (2020)
Yurochkin, M., Agarwal, M., Ghosh, S., Greenewald, K., Hoang, N., Khazaeni, Y.: Bayesian nonparametric federated learning of neural networks. In: International Conference on Machine Learning, pp. 7252–7261. PMLR (2019)
Zagoruyko, S., Komodakis, N.: Wide residual networks. In: British Machine Vision Conference (BMVC) (2016)
Zhang, J., et al.: Dense: data-free one-shot federated learning. In: Advances in Neural Information Processing Systems (2022)
Zhang, S., Liu, M., Yan, J.: The diversified ensemble neural network. Adv. Neural. Inf. Process. Syst. 33, 16001–16011 (2020)
Zhou, Y., Pu, G., Ma, X., Li, X., Wu, D.: Distilled one-shot federated learning. ar**v preprint ar**v:2009.07999 (2020)
Zhu, W., Luo, J.: Federated medical image analysis with virtual sample synthesis. In: Medical Image Computing and Computer Assisted Intervention, pp. 728–738. Springer (2022). https://doi.org/10.1007/978-3-031-16437-8_70
Acknowledgments
This work was supported by funding from the DGIST R &D program of the Ministry of Science and ICT of KOREA (22-KUJoint-02) and the framework of international cooperation program managed by the National Research Foundation of Korea (NRF-2022K2A9A1A01097840) and the NRF grant funded by the Korean Government (MSIT)(No. 2019R1C1C1008727) and the National Institute of Health (MH113406, DA057567, AA021697) and by the Stanford HAI Google Cloud Credit.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kang, M. et al. (2023). One-Shot Federated Learning on Medical Data Using Knowledge Distillation with Image Synthesis and Client Model Adaptation. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14221. Springer, Cham. https://doi.org/10.1007/978-3-031-43895-0_49
Download citation
DOI: https://doi.org/10.1007/978-3-031-43895-0_49
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43894-3
Online ISBN: 978-3-031-43895-0
eBook Packages: Computer ScienceComputer Science (R0)
