Abstract
Federated Learning (FL) is a well-known framework for distributed machine learning that enables mobile phones and IoT devices to build a shared machine learning model via only transmitting model parameters to preserve sensitive data. However, existing Non-IID FL methods always assume data distribution of clients are under a single imbalance scenario, which is nearly impossible in the real world. In this work, we first investigate the performance of the existing FL methods under double imbalance distribution. Then, we present a novel FL framework, called Federated Learning with Gravitation Regulation (FedGR), that can efficiently deal with the double imbalance distribution scenario. Specifically, we design an unbalanced softmax to deal with the quantity imbalance in a client by adjusting the forces of positive and negative features adaptively. Furthermore, we propose a gravitation regularizer to effectively tackle the label imbalance among clients by facilitating collaborations between clients. At the last, extensive experimental results show that FedGR outperforms state-of-the-art methods on CIFAR-10, CIFAR-100, and Fashion-MNIST real-world datasets. Our code is available at https://github.com/Guosy-wxy/FedGR.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Chen, M., et al.: Federated learning of N-gram language models. ar**v preprint ar**v:1910.03432 (2019)
Deng, Y., Kamani, M.M., Mahdavi, M.: Adaptive personalized federated learning. ar**v preprint ar**v:2003.13461 (2020)
Duan, M., Liu, D., Chen, X., Liu, R., Tan, Y., Liang, L.: Self-balancing federated learning with global imbalanced data in mobile systems. IEEE Trans. Parallel Distrib. Syst. (2021)
Fallah, A., Mokhtari, A., Ozdaglar, A.: Personalized federated learning with theoretical guarantees: a model-agnostic meta-learning approach. In: Advances in Neural Information Processing Systems, vol. 33, pp. 3557–3568 (2020)
Ghosh, A., Chung, J., Yin, D., Ramchandran, K.: An efficient framework for clustered federated learning. In: Advances in Neural Information Processing Systems, vol. 33, pp. 19586–19597 (2020)
Gupta, O., Raskar, R.: Distributed learning of deep neural network over multiple agents. J. Netw. Comput. Appl. 116, 1–8 (2018)
Huang, H., Shang, F., Liu, Y., Liu, H.: Behavior mimics distribution: combining individual and group behaviors for federated learning. ar**v preprint ar**v:2106.12300 (2021)
Huang, Y., et al.: Personalized cross-silo federated learning on non-IID data. In: AAAI, pp. 7865–7873 (2021)
Kairouz, P., et al.: Advances and open problems in federated learning. Found. Trends® Mach. Learn. 14(1–2), 1–210 (2021)
Karimireddy, S.P., Kale, S., Mohri, M., Reddi, S., Stich, S., Suresh, A.T.: Scaffold: stochastic controlled averaging for federated learning. In: International Conference on Machine Learning, pp. 5132–5143. PMLR (2020)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)
Li, Q., He, B., Song, D.: Model-contrastive federated learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10713–10722 (2021)
Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., Smith, V.: Federated optimization in heterogeneous networks. In: Proceedings of Machine Learning and Systems, vol. 2, pp. 429–450 (2020)
Li, X.C., Zhan, D.C.: FedRS: federated learning with restricted softmax for label distribution non-IID data. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 995–1005 (2021)
Van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(11) (2008)
McMahan, B., Moore, E., Ramage, D., Hampson, S., Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics, pp. 1273–1282. PMLR (2017)
Niu, Y., Deng, W.: Federated learning for face recognition with gradient correction. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 1999–2007 (2022)
Reddi, S., et al.: Adaptive federated optimization. ar**v preprint ar**v:2003.00295 (2020)
T Dinh, C., Tran, N., Nguyen, J.: Personalized federated learning with Moreau envelopes. In: Advances in Neural Information Processing Systems, vol. 33, pp. 21394–21405 (2020)
Tan, A.Z., Yu, H., Cui, L., Yang, Q.: Towards personalized federated learning. IEEE Trans. Neural Netw. Learn. Syst. (2022)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Wang, J., Liu, Q., Liang, H., Joshi, G., Poor, H.V.: Tackling the objective inconsistency problem in heterogeneous federated optimization. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems, vol. 33, pp. 7611–7623. Curran Associates, Inc. (2020)
**ao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. ar**v preprint ar**v:1708.07747 (2017)
Xu, J., Xu, Z., Walker, P., Wang, F.: Federated patient hashing. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 6486–6493 (2020)
Yang, Q., Liu, Y., Chen, T., Tong, Y.: Federated machine learning: concept and applications. ACM Trans. Intell. Syst. Technol. (TIST) 10(2), 1–19 (2019)
Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., Chandra, V.: Federated learning with non-IID data. ar**v preprint ar**v:1806.00582 (2018)
Zheng, W., Yan, L., Gou, C., Wang, F.Y.: Federated meta-learning for fraudulent credit card detection. In: Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 4654–4660 (2021)
Acknowledgement
This work was supported by National Key R & D Program of China (No. 2022YFF0606303) and National Natural Science Foundation of China (No. 62076079).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Guo, S. et al. (2023). FedGR: Federated Learning with Gravitation Regulation for Double Imbalance Distribution. In: Wang, X., et al. Database Systems for Advanced Applications. DASFAA 2023. Lecture Notes in Computer Science, vol 13943. Springer, Cham. https://doi.org/10.1007/978-3-031-30637-2_47
Download citation
DOI: https://doi.org/10.1007/978-3-031-30637-2_47
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-30636-5
Online ISBN: 978-3-031-30637-2
eBook Packages: Computer ScienceComputer Science (R0)