AMBLE: Adjusting Mini-Batch and Local Epoch Adaptively for Federated Learning with Serverless Parameter Server
AMBLE: 서버리스 파라미터 서버 기반의 연합학습을 위한 적응형 Mini-Batch 및 Local Epoch 조정 기법
- 주제(키워드) Federated Learning , Serverless Computing , System Heterogeneity , Local SGD
- 주제(DDC) 006.31
- 발행기관 아주대학교
- 지도교수 오상윤
- 발행년도 2021
- 학위수여년월 2021. 8
- 학위명 석사
- 학과 및 전공 일반대학원 인공지능학과
- 실제URI http://www.dcollection.net/handler/ajou/000000031174
- 본문언어 영어
- 저작권 아주대학교 논문은 저작권에 의해 보호받습니다.
초록/요약
As data privacy becomes increasingly important, federated learning is a technique for training deep learning models while ensuring the data privacy of devices. Because federated learning updates the global model through a centralized server, it is challenging to reduce communication overhead. It is also challenging to consider the system heterogeneity of devices. In this paper, we present a new architecture for federated learning with a serverless parameter server. Because gradients and global model updates in federated learning are event-driven, a serverless environment can be utilized to decouple the synchronization process of the parameter server from the device’s model communication process. In addition, we propose AMBLE, which adaptively adjusts the local mini-batch and local epoch size for heterogeneous devices in federated learning, synchronously updating the parameters. Our proposed scheme, AMBLE, can increase computation during the waiting time caused by stragglers and can scale the local learning rate to improve the model convergence rate and accuracy. We confirm that federated learning with AMBLE can be stably trained with a faster convergence speed and higher accuracy than FedAvg, in both the non-IID and IID cases.
more목차
I INTRODUCTION １
A. BACKGROUND AND MOTIVATION １
B. PROBLEM STATEMENT AND CONTRIBUTION ４
II BACKGROUND AND RELATED WORKS ７
A. BACKGROUND ７
1 Federated Learning ７
2 Serverless Computing １０
B. RELATED WORKS １３
1 Local Mini-batch SGD １３
2 Adaptive Batch Size Approach １７
III AMBLE WITH SERVERLESS PARAMETER SERVER ２０
A. ARCHITECTURE OF SERVERLESS PARAMETER SERVER ２０
B. PROCESS OF AMBLE IN FEDERATED LEARNING ２２
IV EXPERIMENT AND EVALUATION ２７
A. EXPERIMENT SETUP ２７
1 Experiment environment ２７
B. EXPERIMENT RESULT ２９
1 MNIST with IID Case ２９
2 MNIST with Non-IID Case ３２
3 CIFAR-10 with IID case ３４
4 CIFAR-10 with Non-IID case ３５
5 Reusing Serverless Instance ３６
V CONCLUSION AND FUTURE WORKS ３８
REFERENCE ３９
