Semantic Prompting with Image Token for Continual Learning
- 주제(키워드) Deep learning , Continual learning , Task-agnostic , Prompt-based learning
- 주제(DDC) 006.31
- 발행기관 아주대학교 일반대학원
- 지도교수 황원준
- 발행년도 2024
- 학위수여년월 2024. 8
- 학위명 석사
- 학과 및 전공 일반대학원 인공지능학과
- 실제URI http://www.dcollection.net/handler/ajou/000000033939
- 본문언어 한국어
- 저작권 아주대학교 논문은 저작권에 의해 보호받습니다.
초록/요약
Continual learning aims to adapt model parameters for new tasks while preserving knowledge from previous tasks. Recently, prompt-based learning has leveraged pre-trained models to facilitate learning new tasks through prompts, avoiding the need for a rehearsal buffer. While this approach has shown exceptional results, existing methods rely on a prior task-selection process to choose suitable prompts. However, inaccuracies in task selection can adversely affect performance, especially when dealing with a large number of tasks or imbalanced task distributions. To overcome this challenge, we present I-Prompt, a task-agnostic method that focuses on the visual semantic information of image tokens, thus eliminating the need for task prediction. Our approach features semantic prompt matching, which selects prompts based on the similarities between tokens, and image token-level prompting, which applies prompts directly to image tokens at intermediate layers. As a result, our method delivers competitive performance on four benchmarks while significantly reducing training time compared to state-of-the-art methods. Furthermore, we demonstrate the effectiveness of our method in various scenarios through extensive experiments.
more목차
Abstract 1
1 Introduction 6
2 Related Work 11
2.1 Continual Learning 11
2.2 Prompt-based Continual Learning 12
2.3 Token Similarity in Transformer 13
3 Proposed Method 15
3.1 Preliminary 15
3.1.1 Continual learning protocol 15
3.1.2 Vision transformer 16
3.1.3 Traditional prompt matching 17
3.2 Semantic Prompting with Image-token 19
3.2.1 Semantic prompt matching 19
3.2.2 Image token-level prompting 21
3.2.3 Objective function 22
4 Experiments 24
4.1 Experimental settings 24
4.1.1 Dataset 24
4.1.2 Evaluation scenarios 25
4.2 Comparison with State-of-the-Arts 26
4.2.1 Task-imbalanced scenario 26
4.2.2 Task-balanced scenario 27
4.2.3 Online continual learning scenario 29
4.3 Ablation studies 29
4.3.1 Effects of each component 29
4.3.2 Hyperparameter analysis 30
4.3.3 Efficiency comparison 31
4.3.4 Various task distribution 32
4.3.5 Random increase scenario 34
5 Conclusion 35
Bibliography 37
