Heterogeneous coding-based federated learning

doi:10.3969/j.issn.1000-5641.2023.05.010

Journal of East China Normal University(Natural Science) ›› 2023, Vol. 2023 ›› Issue (5): 110-121.doi: 10.3969/j.issn.1000-5641.2023.05.010

• System for Learning from Data • Previous Articles

Heterogeneous coding-based federated learning

Hongwei SHI¹(), Daocheng HONG^2,*(), Lianmin SHI^3,⁴, Yingyao YANG³

1. School of Information Engineering, Suqian University, Suqian, Jiangsu　223800, China
2. Shanghai Institute of AI for Education & School of Computer Science and Technology, East China Normal University, Shanghai　200062, China
3. School of Computer Science and Technology, Soochow University, Suzhou, Jiangsu　215008, China
4. The Key Laboratory of Cognitive Computing and Intelligent Information Processing of Fujian Education Institutions, Wuyi University, Wuyishan, Fujian　354300, China

Received:2023-07-19 Accepted:2023-07-19 Online:2023-09-25 Published:2023-09-15
Contact: Daocheng HONG E-mail:shwtongxin@squ.edu.cn;hongdc@dase.ecnu.edu.cn

Abstract

Abstract:

In heterogeneous federated learning systems, among a variety of edge devices such as personal computers and embedded devices, resource-constrained devices, i.e. stragglers, reduce the training efficiency of the federated learning system. This paper proposes a heterogeneous coded federated learning (HCFL) system to ① improve the training efficiency of the system and speed up the training of heterogeneous federated learning (FL) for multiple stragglers, ② provide a certain level of data privacy protection. The HCFL scheme designs scheduling strategies from the perspective of client and server to satisfy the accelerated calculation of multiple stragglers model in the general environment. In addition, a linear coded computing (LCC) scheme is designed to provide data protection for task distribution. The experimental results show that HCFL can reduce training time by 89.85% when the performance difference between devices is large.

Key words: federated learning, linear coding, heterogeneous system, scheduling algorithm

CLC Number:

TP181

Hongwei SHI, Daocheng HONG, Lianmin SHI, Yingyao YANG. Heterogeneous coding-based federated learning[J]. Journal of East China Normal University(Natural Science), 2023, 2023(5): 110-121.

Figures/Tables 6

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

References 21

1	KONECNY J, MCMAHAN H B, YU F, et al. Federated learning: Strategies for improving communication efficiency [EB/OL]. (2017-10-30)[2023-07-06]. https://arxiv.org/abs/1610.05492.
2	KAIROUZ P, MCMAHAN H B, AVENT B, et al.. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 2021, 14 (1/2): 1- 210.
3	MCMAHAN H B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data [C] // Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS). 2017, 54: 1273-1282.
4	LI T, SAHU A K, ZAHEER M, et al.. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2020, (2): 429- 450.
5	LIU R, WU F, WU C, et al. No one left behind: Inclusive federated learning over heterogeneous devices [C] // Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022: 3398-3406.
6	ZHAO Z, FENG C, YANG H H, et al.. Federated-learning-enabled intelligent fog radio access networks: Fundamental theory, key techniques, and future trends. IEEE Wireless Communications, 2020, 27 (2): 22- 28.
7	YANG H, ARAFA A, QUEK T, et al. Age-based scheduling policy for federated learning In mobile edge networks [C] // Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2020: 8743-8747.
8	刘汪根, 郑淮城, 荣国平.. 云环境下大规模分布式计算数据感知的调度系统. 大数据, 2020, 6 (1): 81- 98.
9	SHEN S, ZHU T, WU D, et al. From distributed machine learning to federated learning: In the view of data privacy and security [J]. Concurrency and Computation: Practice and Experience, 2020: 1-19.
10	刘俊旭, 孟小峰.. 机器学习的隐私保护研究综述. 计算机研究与发展, 2020, 57 (2): 346- 362.
11	PAPERNOT N, MCDANIEL P, SINHA A, et al. Towards the science of security and privacy in machine learning [EB/OL]. (2016-11-11)[2023-07-06]. https://arxiv.org/abs/1611.03814.
12	刘艺璇, 陈红, 刘宇涵, 等.. 联邦学习中的隐私保护技术. 软件学报, 2022, 33 (3): 1057- 1092.
13	YU X, YAN Z, VASILAKOS A.. A survey of verifiable computation. Mobile Networks and Applications, 2017, 22 (3): 438- 453.
14	李宁波, 周潭平, 车小亮, 等.. 多密钥全同态加密研究. 密码学报, 2020, 7 (6): 713- 734.
15	TRUEX S, BARACALDO N, ANWAR A, et al. A hybrid approach to privacy-preserving federated learning [C] // Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security. 2019: 1-11.
16	BERNSTEIN G, SHELDON D. Differentially private Bayesian linear regression [C] // Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019: 525-535.
17	LEE K, LAM M, PEDARSANI R, et al.. Speeding up distributed machine learning using codes. IEEE Transactions on Information Theory, 2017, 64 (3): 1514- 1529.
18	郑腾飞, 周桐庆, 蔡志平, 等.. 编码计算研究综述. 计算机研究与发展, 2021, 58 (10): 2187- 2212.
19	CAO C M, WANG J, WANG J P, et al. Optimal task allocation and coding design for secure coded edge computing [C] // Proceedings of the 39th IEEE International Conference on Distributed Computing Systems (ICDCS). 2019: 1083-1093.
20	DHAKAL S, PRAKASH S, YONA Y, et al. Coded federated learning [C] // Proceedings of the IEEE Globecom Workshops (GC Wkshps). 2019: 1-6.
21	CHELLAPILLA K, PURI S, SIMARD P. High performance convolutional neural networks for document processing [EB/OL]. (2006-11-09)[2023-07-06]. https://inria.hal.science/inria-00112631/document.

[1]	Qimin BAN, Wen WU, Wenxin HU, Hui LIN, Wei ZHENG, Liang HE. Personalized course recommendations based on a learner’s knowledge and personality [J]. Journal of East China Normal University(Natural Science), 2022, 2022(6): 87-101.
[2]	CHEN Lei-hui, KUANG Jun, CHEN Hui, ZENG Wei, ZHENG Jian-bing, GAO Ming. Techniques for cross-domain recommendation:A survey [J]. Journal of East China Normal University(Natural Sc, 2017, 2017(5): 101-116,137.
[3]	FANG Juan, LIU Hong-ying, LI Qing-li. Learning distance metrics with dimension constraints [J]. Journal of East China Normal University(Natural Sc, 2017, 2017(2): 69-74,88.
[4]	YUAN Yu-Ping, AN Zeng-Long. Support vector machine in the primal space based on\\ the ramp loss function\\[6mm [J]. Journal of East China Normal University(Natural Sc, 2016, 2016(2): 20-29.

Heterogeneous coding-based federated learning

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 6

References 21

Related Articles 4

Recommended Articles

Metrics

Comments