Improved Knowledge Distillationwith Dynamic Network Pruning

Authors : Eren ŞENER, Emre AKBAŞ

Pages : 650-665

Doi:10.29109/gujsc.1141648

View : 25 | Download : 27

Publication Date : 2022-09-30

Article Type : Research Paper

Abstract :Deploying convolutional neural networks to mobile or embedded devices is often prohibited by limited memory and computational resources. This is particularly problematic for the most successful networks, which tend to be very large and require long inference times. Many alternative approaches have been developed for compressing neural networks based on pruning, regularization, quantization or distillation. In this paper, we propose the “Knowledge Distillation with Dynamic Pruning” (KDDP), which trains a dynamically pruned compact student network under the guidance of a large teacher network. In KDDP, we train the student network with supervision from the teacher network, while applying L1 regularization on the neuron activations in a fully-connected layer. Subsequently, we prune inactive neurons. Our method automatically determines the final size of the student model. We evaluate the compression rate and accuracy of the resulting networks on an image classification dataset, and compare them to results obtained by Knowledge Distillation (KD). Compared to KD, our method produces better accuracy and more compact models.
Keywords : Knowledge Distillation, Neural Network Compression, Image Classification, Deep Neural Networks