Progressive block knowledge distillation method for neural network acceleration

Abstract

The invention discloses a progressive block knowledge distillation method for neural network acceleration. The method specifically comprises the following steps of: inputting an original complex network and related parameters; dividing the original complex network into a plurality of sub-network blocks, designing student sub-network blocks and randomly initializing the parameters; taking the inputoriginal complex network as a teacher network of the first block distillation process and obtaining a student network after the block distillation process is completed, wherein the first student sub-network block has the optimum parameters; taking the student network obtained in the last block distillation process as a teacher network of the next block distillation process so as to obtain a nextstudent network, wherein the student sub-network blocks, the block distillation of which is finished, have the optimum parameters; and obtaining a final simple student network and optimum parameters after all the sub-network block distillation processes are completed. The method is capable of achieving the effect of accelerating model compression on common hardware architecture, is simple to realize, and is an effective, practical and simple deep network model compression acceleration algorithm.

Description

Technical field [0001] The invention relates to the field of deep network model compression and acceleration, in particular to a progressive block knowledge distillation method for neural network acceleration. Background technique [0002] Since 2016, the artificial intelligence boom has swept the world. Major domestic and foreign companies, including Google, Microsoft, Baidu, Alibaba, Tencent, etc., have put a lot of effort into research on artificial intelligence; the Chinese government has also released the "New Generation Artificial Intelligence Development Plan" not long ago, formulating the future The goal of China's artificial intelligence development. In the past few years, the rapid development of deep learning has enabled the most advanced algorithm performance in a series of fields such as computer vision and natural language processing to make leapfrog progress. In the field of artificial intelligence, the traditional chip computing architecture cannot support the n...

AI Technical Summary

Problems solved by technology

Most of the existing strategies use only one-step mechanism to obtain the student model from the teacher model. Finding the student network function that approximates the teacher network function in a huge search space requires a lot of network configurations. In practice, this non-joint optimization Process is also unwieldy and unstable

Distillation schemes in the form of sub-network blocks are very easy to optimize, but cannot effectively preserve the sequential dependencies between layer-specific sub-network blocks

In addition, existing design guidelines for student sub-network blocks cannot well protect the receptive field information of atomic network blocks in feature extraction

Images