Method and apparatus for neural network quantization

Abstract

The invention provides a method and apparatus for neural network quantization. Apparatuses and methods of manufacturing same, systems, and methods for performing network parameter quantization in deepneural networks are described. In one aspect, diagonals of a second-order partial derivative matrix (a Hessian matrix) of a loss function of network parameters of a neural network are determined andthen used to weight (Hessian-weighting) the network parameters as part of quantizing the network parameters. In another aspect, the neural network is trained using first and second moment estimates ofgradients of the network parameters and then the second moment estimates are used to weight the network parameters as part of quantizing the network parameters. In yet another aspect, network parameter quantization is performed by using an entropy-constrained scalar quantization (ECSQ) iterative algorithm. In yet another aspect, network parameter quantization is performed by quantizing the network parameters of all layers of a deep neural network together at once.

Description

[0001] This application claims priority to U.S. Provisional Patent Application No. 62 / 409,961, filed October 19, 2016, and U.S. Nonprovisional Patent Application No. 15 / 433,531, filed February 15, 2017, the entire contents of which applications Incorporated herein by reference. technical field [0002] The present disclosure relates generally to deep neural networks, and more particularly, to a method and apparatus for neural network quantization. Background technique [0003] Machine learning technology is constantly developing and has begun to support many aspects of modern society, from web search, content filtering, automatic recommendation of commercial websites, automated games, object detection, image classification, speech recognition, machine translation, drug discovery and genomics. The current state-of-the-art in the field of machine learning are deep neural networks, which use computational models consisting of multiple processing layers that learn representation...

AI Technical Summary

Problems solved by technology

[0007] This makes it difficult to deploy deep neural networks on devices with limited storage, such as mobile / portable devices

Images