Embedding Dimensionality Estimation for Autoencoder with Lazy Node Dropout
초록/요약
Autoencoders are widely used for nonlinear dimension reduction. However, determining the number of nodes in the autoencoder embedding spaces is still a challenging task. The number of nodes in the bottleneck layer, which is an encoded representation, is estimated and determined as a hyperparameter. Therefore, the number of bottleneck nodes are needed to automatically select as an indicator to maintain embedding performance and reduce the complexity of the model. This study proposes a method for automatically estimating the adapted number of nodes in the bottleneck layer for the autoencoder training process. The basic idea of the proposed method is to eliminate lazy nodes rarely affect to the model performance based on the weight distribution of the bottleneck layer. With lazy node dropout, we reduce the number of bottleneck nodes. The following two methods are the main tasks of this paper. The first one is verifying Informative dropout reducing inactive nodes with poorly updated weights, not randomly decreasing the bottleneck nodes. The second one is verifying Online dropout reducing nodes in the online learning process rather than repeating batch learning process. The autoencoder with the number of nodes determined by the proposed method showed better or similar performance in classification accuracy compared to random dropout with online process. Since the proposed method takes place in the learning process of the autoencoder, it has the advantage of accelerating the training speed.
more목차
제1장 Introduction 1
제1절 Contributions 4
제2장 Fundamental 6
제1절 Autoencoder 6
제2절 Weight update in neural networks 8
제3절 Dropout and Dropconnect 8
제3장 Proposed Methods 9
제1절 Informative dropout: measurement of node activity 10
제2절 Online dropout: model training 15
제4장 Experiments 22
제1절 Dimensionality estimation 24
제2절 Performance of embedding vectors 28
제3절 Computation time comparison 34
제5장 Conclusion 36
References 37
