Dynamic Gradient Sparsification Exploiting Aggregated Gradients for Scalable Distributed Deep Learning
고확장성 분산 딥 러닝을 위한 동적 기울기 희소화 기법
- 주제(키워드) gradient sparsification , distributed deep learning , scalability
- 주제(DDC) 006.31
- 발행기관 아주대학교 일반대학원
- 지도교수 Sangyoon Oh
- 발행년도 2024
- 학위수여년월 2024. 2
- 학위명 박사
- 학과 및 전공 일반대학원 인공지능학과
- 실제URI http://www.dcollection.net/handler/ajou/000000033298
- 본문언어 영어
- 저작권 아주대학교 논문은 저작권에 의해 보호받습니다.
초록/요약
Communication overhead is a major obstacle to scaling distributed training systems. Gradient sparsification is a potential optimization approach to reduce the communication volume without significant loss of model fidelity. However, existing gradient sparsification methods have low scalability owing to inefficient design of their algorithms, which raises the communication overhead significantly. In particular, gradient build-up and inadequate sparsity control methods degrade the sparsification performance considerably. Moreover, communication traffic increases drastically owing to workload imbalance of gradient selection between workers. In this paper, we propose ExDyna to address above challenges. In ExDyna, the gradient tensor of the model comprises fined-grained blocks, and contiguous blocks are grouped into non-overlapping partitions. Each worker selects gradients in its exclusively allocated partition so that gradient build-up never occurs. To balance the workload of gradient selection between workers, ExDyna adjusts the topology of partitions by comparing the workloads of adjacent partitions. In addition, ExDyna supports online threshold scaling, which estimates the accurate threshold of gradient selection on-the-fly. Accordingly, ExDyna can satisfy the user-required sparsity level during a training period regardless of models and datasets. Therefore, ExDyna can enhance the scalability of distributed training systems by preserving near-optimal gradient sparsification cost. In experiments, ExDyna outperformed state-of-the-art sparsifiers in terms of training speed and sparsification performance while achieving high accuracy.
more목차
1 Introduction 1
2 Preliminaries 9
3 Limitations of State-of-the-Art Methods 11
4 ExDyna Design 14
4.1 Block-based gradient vector partitioning 14
4.2 Dynamic partition allocation 16
4.3 Partition-wise exclusive gradient selection 19
4.4 Online threshold scaling 20
5 Evaluation 23
5.1 Methodology 23
5.2 Performance evaluation 24
5.3 Efficiency evaluation 35
6 Conclusion 44
Bibliography 45
