Abstract
Federated learning (FL) involves collaboration between clients with limited data to produce a single optimal global model through consensus. One of the difficulties with FL is the differences in data statistics between local clients. Clients with statistically heterogeneous data deviate from the global target, resulting in a slower convergence rate and increased communication resource consumption. To address this problem, we propose a new approach, FedH, that maintains the proximity of local models to the global target while maximizing communication efficiency and computational resources. We use the Hessian matrix to constrain client updates that deviate from the global target. Our results demonstrate the superiority of FedH over FL baselines such as FedAvg, FedProx, and Fedcurv when applied to benchmark datasets such as MNIST, Fashion-MNIST, and CIFAR-10 across a range of statistical heterogeneity levels.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
McMahan, H.B., Moore, E., et al.: Communication-efficient learning of deep networks from decentralized data. In: AISTATS (2017)
Kairouz, P., McMahan, H.B., Avent, B., Bellet, A., Bennis, M., et al.: Advances and open problems in federated learning. Found. Trends Mach. Learn. 14, 1–210 (2019)
Karimireddy, S., Kale, S., Mohri, M., et al.: SCAFFOLD: stochastic controlled averaging for federated learning. In: ICML, 13–18 Jul 2020, pp. 5132–5143 (2020)
Li, T., Sahu, A.K., et al.: Federated optimization in heterogeneous networks. In: Dhillon, I., Papailiopoulos, D., Sze, V. (eds.) Proceedings of Machine Learning and Systems, vol. 2, pp. 429–450 (2020)
Shoham, N., et al.: Overcoming forgetting in federated learning on non-IID data. In: FL-NeurIPS (2019). ar**v:1910.07796
Pennington, J., Bahri, Y.: Geometry of neural network loss surfaces via random matrix theory. In: ICML (2017)
Pennington, J., Worah, P.: The spectrum of the fisher information matrix of a single-hidden-layer neural network. In: Conference on Neural Information Processing Systems (2018)
Hsu, T., Qi, H., Brown, M.: Measuring the effects of non-identical data distribution for federated visual classification. ar**v:abs/1909.06335 (2019)
Zhao, Y., Li, M., Lai, L., et al.: Federated learning with non-IID data. ar**v:abs/1806.00582 (2018)
Guha, N., Talwalkar, A., Smith, V.: One-shot federated learning. CoRR, abs/1902.11175 (2019)
Lin, T., Kong, L., Stich, S.U., Jaggi, M.: Ensemble distillation for robust model fusion in federated learning. Adv. Neural. Inf. Process. Syst. 33, 2351–2363 (2020)
Chen, H.-Y., Chao, W.-L.: FedBE: making Bayesian model ensemble applicable to federated learning. In: ICLR (2021)
Maddox, W., Garipov, T., Izmailov, P., et al.: A simple baseline for Bayesian uncertainty in deep learning. In: NeurIPS (2019)
Li, T., Sahu, A.K., Zaheer, M., et al.: Feddane: a federated newton-type method. In: 2019 53rd Asilomar Conference on Signals, Systems, and Computers, pp. 1227–1231 (2019)
Islamov, R., Qian, X., Richtárik, P.: Distributed second order methods with fast rates and compressed communication. In: ICML (2021)
Safaryan, M., Islamov, R., et al.: FedNL: making newton-type methods applicable to federated learning. In: ICML, ser. Workshop on Federated Learning for User Privacy and Data Confidentiality (2021)
Qian, X., et al.: Basis matters: better communication-efficient second order methods for federated learning. In: AISTATS (2022)
Liu, Y., Zhu, Y., James, J.: Resource-constrained federated learning with heterogeneous data: formulation and analysis. IEEE Trans. Network Sci. Eng. (2021)
Liu, D.C., Nocedal, J.: On the limited memory BFGs method for large scale optimization. Math. Program. 45, 503–528 (1989)
Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., Smith, V.: Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, vol. 2, pp. 429–450 (2020)
Nocedal, J., Wright, S.J.: Numerical Optimization. Springer, New York (1999). https://doi.org/10.1007/0-387-22742-3
LeCun,Y.A., Bottou, L., Orr, G.B., Müller, K.-R.: Efficient backprop. In: Neural Networks: Tricks of the Trade (2012)
Becker, S., LeCun, Y.: Improving the convergence of back-propagation learning with second-order methods. In: Technical Report CRG-TR-88-5 (1989)
Schraudolph, N.N.: Fast curvature matrix-vector products. In: ICANN (2001)
Chen, P.: Hessian matrix vs. gauss-newton hessian matrix. SIAM J. Numer. Anal. 49, 1417–1435 (2011)
Nocedal, J., Wright, S.: Numerical Optimization. Springer, New York (2006). https://doi.org/10.1007/978-0-387-40065-5
Shamir, O., Srebro, N., Zhang, T.: Communication-efficient distributed optimization using an approximate newton-type method. In: International Conference on Machine Learning. PMLR 2014, pp. 1000–1008 (2014)
Oren, S.S., Luenberger, D.G.: Self-scaling variable metric (SSVM) algorithms. Part I: criteria and sufficient conditions for scaling a class of algorithms. Manage. Sci. 20(5), 845–862 (1974)
Deng, L.: The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Process. Mag. 29(6), 141–142 (2012)
Cohen, G., Afshar, S., Tapson, J., Van Schaik, A.: Emnist: extending mnist to handwritten letters. In: International Joint Conference on Neural Networks (IJCNN). IEEE 2017, pp. 2921–2926 (2017)
Krizhevsky, A., Hinton, G.: Convolutional deep belief networks on cifar-10. Unpublished manuscript, 40(7), 1–9 (2010)
Li, X., Huang, K., et al.: On the convergence of fedavg on non-IID data. In: ICLR (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ahmad, A., Luo, W., Robles-Kelly, A. (2023). A Hessian-Based Federated Learning Approach to Tackle Statistical Heterogeneity. In: Yang, X., et al. Advanced Data Mining and Applications. ADMA 2023. Lecture Notes in Computer Science(), vol 14177. Springer, Cham. https://doi.org/10.1007/978-3-031-46664-9_28
Download citation
DOI: https://doi.org/10.1007/978-3-031-46664-9_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46663-2
Online ISBN: 978-3-031-46664-9
eBook Packages: Computer ScienceComputer Science (R0)