Compression method and system used for neural network language model (NN LM)

Abstract

The invention discloses a compression method and system used for a neural network language model (NN LM). The method includes: inputting training data into the neural network language model for pre-training; respectively carrying out base decomposition and clustering quantization operations on word vector matrices of input and / or output of the language model to compress the word vector matrices; and inputting the training data again into the neural network language model after compression of the word vector matrices to finely tune the language model. The invention provides a novel and efficient structured word embedding framework based on product quantization, the framework is used for compressing the input / output word vector matrices, and a significant memory reducing rate can be obtainedin a case of not damaging NN LM performance.

Description

technical field [0001] The invention belongs to the technical field of language model compression, in particular to a compression method and system for neural network language models. Background technique [0002] In Automatic Speech Recognition (ASR, Automatic Speech Recognition), the language model (LM, Language Model) is the core component that combines the syntactic and semantic constraints of a given language. Although the traditional N-gram backoff language model with smoothing has been widely used in ASR, its context length is limited, and the memory requirement for a large vocabulary is also large. Recently, neural network-based language models (NN LM, Neural Network Language Model) have attracted great interest due to their efficient encoding of word context history and memory efficiency. In neural network-based language models, both word contexts and target words are projected into a continuous space. Projections represented by transformation matrices are learned...

AI Technical Summary

Problems solved by technology

It solves the above two problems, but the memory reduction rate is negligible (20% when |V|=10K) with tiny vocabulary size

Also, the architecture is complicated because it calls an additional layer in the output layer

Images