Knowledge distillation method and device fusing channel and relation feature learning, and equipment

Abstract

The invention relates to the technical field of knowledge distillation, and discloses a knowledge distillation method and device fusing channel and relation feature learning, and equipment, and the method comprises the steps: constructing an untrained student network and a pre-trained teacher network; respectively inputting the training data into a student network and a teacher network to obtain an output result of the student network and an output result of the teacher network, wherein the training data further comprises corresponding real label data; determining a distillation loss function based on the channel data of the student network and the teacher network, the output result of the student network, the output result of the teacher network, and the relationship between the learning network and the teacher network migration sample; and performing iterative training on the student network based on the distillation loss function. According to the method and device, the student network model can be effectively compressed, and the performance of the student network can be further improved and even exceeds the performance of a large teacher network.

Description

Technical field [0001] The present invention relates to the field of knowledge distillation, and specifically, a knowledge distillation method, apparatus, and apparatus for fusion channels and relationship characteristics. Background technique [0002] Deep learning technology has developed very rapidly in recent years, and there is also a breakthrough in computer vision and natural language processing. From 2012, AlexNet proposed in 2016, these neural networks have powerful performance and break through traditional algorithms in the image classification task. More deeper neural networks can extract more information and have a better representation. However, deeper, more complex neural networks require more power and reason to meet real-time responses in practical industrial applications, and in real time in real-time in mobile terminals cannot accept a large number of network parameters and calculations. AlexNet and Densenet neural networks only consider how to improve accuracy,...

AI Technical Summary

Problems solved by technology

However, the above method does not take into account the relevant knowledge of the channel on the convolutional layer, and also ignores the relationship feature knowledge of different samples in the channel.

Moreover, during the learning process, the student network will be affected by the wrong knowledge of the teacher, so that the performance of the student network cannot be effectively improved in the later stage of training the student network.

Images