Abstract—Structured pruning for deep neural networks has been proposed for network model compression. Because earlier structured pruning methods assign pruning rate manually, finding appropriate pruning rate to suppress the degradation of pruned model accuracy is difficult. In this paper, we propose a structured pruning method by deriving pruning rate for each layer adaptively based on gradient and loss function. The proposed method first calculates a threshold of L1-norm of the pruned weight for each layer using loss function and gradient per layer. The threshold guarantees no degradation of the loss function by pruning when the L1-norm of pruned weight is less than that threshold. Then, by comparing the L1-norm of the pruned weight and the calculated threshold while changing the pruning rate, we derive the pruning rate for each layer, which does not degrade the loss function. By applying the derived pruning rate per layer, the accuracy degradation of the pruned model is suppressed. We evaluate the proposed method on CIFAR-10 task with VGG-16 and ResNet in iterative pruning method. For example, our proposed method reduces model parameters of ResNet-56 by 66.3% with 93.71% accuracy.
 

Index Terms—neural networks, structured pruning, automatic pruning rate search

Cite: Yasufumi Sakai, Yu Eto, and Yuta Teranishi, "Structured Pruning for Deep Neural Networks with Adaptive Pruning Rate Derivation Based on Connection Sensitivity and Loss Function," Journal of Advances in Information Technology, Vol. 13, No. 3, pp. 295-300, June 2022.

Copyright © 2022 by the authors. This is an open access article distributed under the Creative Commons Attribution License (CC BY-NC-ND 4.0), which permits use, distribution and reproduction in any medium, provided that the article is properly cited, the use is non-commercial and no modifications or adaptations are made.