Neural network model training method and device and storage medium

Abstract

The invention discloses a neural network model training method and device and a storage medium, and the method comprises the steps: respectively storing the teacher networks of different time nodes selected in advance as process models in the training process of the teacher networks; integrating the plurality of stored process models to form a new teacher network; and training the student network by using the new teacher network.

Description

technical field [0001] This article relates to the field of neural network model compression, in particular to a neural network model training method, device and storage medium. Background technique [0002] Compared with large-scale deep neural network models, the performance of lightweight neural network models is generally poor, and it is difficult to meet some applications with high performance requirements. Model compression is the most common method for this problem, and generally includes methods such as model pruning, parameter quantization, and knowledge distillation. [0003] Knowledge distillation is a concept proposed by Hinton in 2015. It aims to introduce the knowledge of a pre-trained teacher network (generally a large network with superior performance and high complexity) as a light-weight model for constructing a student network (to be deployed on the application side). Network, poor performance and low complexity) training part of the loss function to achi...

AI Technical Summary

Problems solved by technology

However, in these methods, the knowledge learned by the student network is only the knowledge of the trained teacher network, and does not include the knowledge in the training process of the teacher network itself, and the knowledge transfer is not complete.

Images