Journal of East China Normal University(Natural Science) >

2023 , Vol. 2023 >Issue 5: 77 - 89

DOI: https://doi.org/10.3969/j.issn.1000-5641.2023.05.007

System for Learning from Data

Privacy-preserving cloud-end collaborative training

  • Xiangyun GAO ,
  • Dan MENG ,
  • Mingkai LUO ,
  • Jun WANG ,
  • Liping ZHANG ,
  • Chao KONG
Expand
  • 1. School of Computer and Information, Anhui Polytechnic University, Wuhu, Anhui　241000, China
    2. OPPO Research Institute, Shenzhen, Guangdong　518000, China
    3. College of Electronic and Information Engineering, Tongji University, Shanghai　201804, China
    4. Reconfigurable and Intelligent Computing Laboratory, Anhui Polytechnic University, Wuhu, Anhui　241000, China

Received date: 2023-06-30

  Online published: 2023-09-20

Fold

Abstract

China has the advantages of scale and diversity in data resources, and mobile internet data applications, which generate massive amounts of data in diverse application scenarios, recommendation systems have the capability to extract valuable information from this massive amounts of data, thereby mitigating the problem of information overload. Most existing research on recommendation systems focused on centralized recommender systems, training the data on the cloud centrally. However, with increasingly prominent data security and privacy protection issues, collecting user data has become increasingly difficult, making centralized recommendation methods infeasible. This study focuses on privacy-preserving cloud-end collaborative training in a decentralized manner for personalized recommender systems. To fully utilize the advantages of end devices and cloud servers while considering privacy and security issues, a cloud-end collaborative training method named FedMNN (federated machine learning and mobile neural network) is proposed for recommender systems based on federated machine learning (FedML) and a mobile neural network (MNN). The proposed method was divided into three parts: First, cloud-based models implemented in various deep learning frameworks were converted into general MNN models for end-device training using the ONNX (open neural network exchange) intermediate framework and a MNN model conversion tool. Second, the cloud server sends the model to the end-side devices, which initialized and obtain local data for training and loss calculation, followed by gradient back-propagation. Finally, the end-side models are fed back to the cloud server for model aggregation and updating. Depending on different requirements, the cloud model was deployed on end-side devices as required, achieving end-cloud collaboration. Experiments comparing power consumption of the proposed FedMNN and FLTFlite (flower and TensorFlow lite) frameworks on benchmark tasks identified that FedMNN is 32% to 51% lower than FLTFlite. Using DSSM (deep structured semantic model) and deep and wide recommendation models, the experimental results demonstrated the effectiveness of the proposed cloud-end collaborative training method.

Key words： privacy protection; federated learning; machine learning; cloud-end collaborative training

Cite this article

Xiangyun GAO , Dan MENG , Mingkai LUO , Jun WANG , Liping ZHANG , Chao KONG . Privacy-preserving cloud-end collaborative training[J]. Journal of East China Normal University(Natural Science), 2023 , 2023(5) : 77 -89 . DOI: 10.3969/j.issn.1000-5641.2023.05.007

References

1 HAFFAR R, SANCHEZ D, DOMINGO-FERRER J.. Explaining predictions and attacks in federated learning via random forests. Applied Intelligence, 2023, 53 (1): 169- 185.
2 ZHAO B, MOPURI K R, BILEN H. IDGL: Improved deep leakage from gradients [EB/OL]. (2020-01-08)[2023-05-18]. https://arxiv.org/pdf/2001.02610.pdf.
3 WEI K, LI J, DING M, et al.. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security, 2020, 15, 3454- 3469.
4 FANG H K, QIAN Q.. Privacy preserving machine learning with homomorphic encryption and federated learning. Future Internet, 2021, 13 (4): 94.
5 LI T, SAHU A K, TALWALKAR A, et al.. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 2020, 37 (3): 50- 60.
6 KAIROUZ P, MCMAHAN H B, AVENT B, et al.. Advances and open problems in federated learning. Foundations and Trends? in Machine Learning, 2021, 14 (1/2): 1- 210.
7 KARIMIREDDY S P, KALE S, MOHRI M, et al. SCAFFOLD: Stochastic controlled averaging for federated learning [C]// Proceedings of the 37th International Conference on Machine Learning. 2020: 5132-5143.
8 HE C Y, LI S Z, SO J, et al. FedML: A research library and benchmark for federated machine learning [EB/OL]. (2020-11-08)[2023-05-18]. https://arxiv.org/pdf/2007.13518.pdf.
9 MATHUR A, BEUTEL D J, DE GUSMÃO P P B, et al. On-device federated learning with flower [EB/OL]. (2021-04-07)[2023-05-18]. https://arxiv.org/pdf/2104.03042.pdf.
10 JIANG X T, WANG H, CHEN Y L, et al. MNN: A universal and efficient inference engine[EB/OL]. (2020-02-27)[2023-06-01]. https://arxiv.org/pdf/2002.12418.pdf.
11 ZHOU P, WANG K H, GUO L K, et al.. A privacy-preserving distributed contextual federated online learning framework with big data support in social recommender systems. IEEE Transactions on Knowledge and Data Engineering, 2019, 33 (3): 824- 838.
12 YANG Q, LIU Y, CHEN T J, et al.. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology, 2019, 10 (2): 12.
13 CHEN Q, YAO L, WU Y L, et al. PyHENet: A generic framework for privacy-preserving DL inference based on fully homomorphic encryption [C]// Proceeding of the 4th International Conference on Data Intelligence and Security. 2022: 127-133.
14 LIU Y, FAN T, CHEN T J, et al.. FATE: An industrial grade platform for collaborative learning with data protection. Journal of Machine Learning Research, 2021, 22, 226.
15 马艳军, 于佃海, 吴甜, 等.. 飞桨: 源于产业实践的开源深度学习平台. 数据与计算发展前沿, 2019, 1 (1): 105- 115.
16 MCMAHAN B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data [C]// Proceedings of the 20th International Conference on Artificial intelligence and statistics. 2017: 1273-1282.
17 LI Q B, WEN Z Y, HE B S. Practical federated gradient boosting decision trees [C]// Proceedings of the AAAI Conference on Artificial Intelligence. 2020: 4642-4649.
18 林伟伟, 石方, 曾岚, 等. 联邦学习开源框架综述 [J]. 计算机研究与发展, 2023, 60(7): 1551-1580.
19 WU D P, SUN M Y, ZHANG P, et al.. Personalized secure demand-oriented data service toward edge-cloud collaborative IoT. IEEE Internet of Things Journal, 2022, 10 (1): 378- 390.
20 WANG T, MEI Y X, JIA W J, et al.. Edge-based differential privacy computing for sensor-cloud systems. Journal of Parallel and Distributed Computing, 2020, 136, 75- 85.
21 YAN Y K, NIU C Y, GU R J, et al. On-device learning for model personalization with large-scale cloud-coordinated domain adaption [C]// Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022: 2180-2190.
22 YAO J C, WANG F, DING X C, et al. Device-cloud collaborative recommendation via meta controller [C]// Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022: 4353-4362.
23 YAO J C, WANG F, JIA K Y, et al. Device-cloud collaborative learning for recommendation [C]// Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2021: 3865-3874.
Options
Outlines

/

〈 〉