Abstract
Federated learning (FL) is a collaborative machine learning paradigm in which clients with limited data collaborate to train a single “best” global model based on consensus. One major challenge facing FL is the statistical heterogeneity among the data for each of the local clients. Clients trained with non-IID or imbalanced data whose models are aggregated using averaging schemes such as FedAvg may result in a biased global model with a slow training convergence. To address this challenge, we propose a novel and robust aggregation scheme, FedMan, which assigns each client a weighting factor based on its statistical consistency with other clients. Such statistical consistency is measured on a Riemannian manifold spanned by the covariance of the local client output logits. We demonstrate the superior performance of FedMAN over several FL baselines (FedAvg, FedProx, and Fedcurv) as applied to various benchmark datasets (MNIST, Fashion-MNIST, and CIFAR-10) under a wide variety of degrees of statistical heterogeneity.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Amari, S.: Information Geometry and Its Applications. AMS, vol. 194. Springer, Tokyo (2016). https://doi.org/10.1007/978-4-431-55978-8
Fletcher, P.T., Joshi, S.: Principal geodesic analysis on symmetric spaces: statistics of diffusion tensors. In: Sonka, M., Kakadiaris, I.A., Kybic, J. (eds.) CVAMIA/MMBIA -2004. LNCS, vol. 3117, pp. 87–98. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-27816-0_8
Förstner, W., Moonen, B.: A metric for covariance matrices (2003)
Fréchet, M.: Les éléments aléatoires de nature quelconque dans un espace distancié. In: Annales de l’institut Henri Poincaré, vol. 10, pp. 215–310 (1948)
Goodfellow, I.J., Pouget-Abadie, J., et al.: Generative adversarial nets. In: NIPS (2014)
Hsu, T., Qi, H., Brown, M.: Measuring the effects of non-identical data distribution for federated visual classification. Ar**v abs/1909.06335 (2019)
Kairouz, P., McMahan, H.B., Avent, B., et. al.: Advances and open problems in federated learning. Found. Trends Mach. Learn. 14, 1–210 (2021)
Karcher, H.: Riemannian center of mass and mollifier smoothing. Commun. Pure Appl. Math. 30(5), 509–541 (1977)
Karimireddy, S.P., Kale, S., Mohri, M., et al.: SCAFFOLD: Stochastic controlled averaging for federated learning. In: ICML, pp. 5132–5143 (13–18 Jul 2020)
Li, T., Sahu, A.K., et al.: FedDANE: a federated newton-type method. In: 2019 53rd Asilomar Conference on Signals, Systems, and Computers, pp. 1227–1231 (2019)
Li, T., Sahu, A.K., et al.: Federated optimization in heterogeneous networks. In: Dhillon, I., Papailiopoulos, D., Sze, V. (eds.) Proceedings of Machine Learning and Systems, vol. 2, pp. 429–450 (2020)
McMahan, H.B., Moore, E., et al.: Communication-efficient learning of deep networks from decentralized data. In: AISTATS (2017)
Miolane, N., Guigui, N., et. al.: Geomstats: a python package for Riemannian geometry in machine learning. J. Mach. Learn. Res. 21(223), 1–9 (2020). http://jmlr.org/papers/v21/19-027.html
Mitra, A., Jaafar, R.H., Pappas, G.J., Hassani, H.: Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients. In: Neural Information Processing Systems (2021)
Pennec, X.: Probabilities and statistics on Riemannian manifolds: basic tools for geometric measurements. In: Proceedings of Nonlinear Signal and Image Processing (NSIP1999), vol. 1, pp. 194–198 (1999)
Shoham, N., et al.: Overcoming forgetting in federated learning on non-IID data. In: FL-NeurIPS (2019). ar**v:1910.07796
Wang, J., Liu, Q., Liang, H., Joshi, G., Poor, H.V.: Tackling the objective inconsistency problem in heterogeneous federated optimization. Adv. Neural. Inf. Process. Syst. 33, 7611–7623 (2020)
Zhao, Y., Li, M., Lai, L., et al.: Federated learning with non-IID data. Ar**v abs/1806.00582 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ahmad, A., Luo, W., Robles-Kelly, A. (2023). Federated Learning Under Statistical Heterogeneity on Riemannian Manifolds. In: Kashima, H., Ide, T., Peng, WC. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2023. Lecture Notes in Computer Science(), vol 13935. Springer, Cham. https://doi.org/10.1007/978-3-031-33374-3_30
Download citation
DOI: https://doi.org/10.1007/978-3-031-33374-3_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-33373-6
Online ISBN: 978-3-031-33374-3
eBook Packages: Computer ScienceComputer Science (R0)