Method for processing out-of-set words of Chinese-over-neural machine translation integrated with classification dictionary

Abstract

The invention relates to a method for processing out-of-set words of Chinese-over-neural machine translation integrated with a classification dictionary, and belongs to the technical field of naturallanguage processing. Out-of-set words are classified. Different types of out-of-set words can be processed by adopting different methods. A classification dictionary is built in a targeted manner; wherein a bilingual dictionary is used for solving a translation problem of rare words outside a word list; an entity dictionary is used for solving the problem of inaccurate entity word translation; anda rule dictionary is used for solving the translation problem of numbers, symbols, time, dates and other words. Then, in the preprocessing stage of the model, out-of-set word recognition is performedby querying a classification dictionary. Label replacement is performed on out-of-set words at the encoding end of the model. A translation result with labels is acquired after model translation andtranslation recovery is carried out on the labels by querying the classification dictionary. The classification dictionary is fused into neural machine translation. The out-of-set words can be more accurately translated and the performance and effect of a neural machine translation system are improved.

Description

technical field [0001] The invention relates to a method for processing foreign words in a Chinese-Vietnamese neural machine translation set integrated into a classification dictionary, and belongs to the technical field of natural language processing. Background technique [0002] Neural machine translation is a machine translation method proposed in recent years. It has achieved good results in resource-rich translation tasks, but the effect in low-resource language neural machine translation is not ideal. To control the computational complexity that grows in proportion to the size of the target vocabulary, most neural machine translation systems limit the vocabulary to only 30,000 to 80,000 common words in the parallel data. Words, when translating, convert out-of-set words into UNK symbols. The obvious problem with this method is that the neural machine translation model cannot effectively translate out-of-set words, and meaningless UNK symbols will increase the ambigui...

AI Technical Summary

Problems solved by technology

[0004] The present invention provides a method for processing out-of-set words in Chinese-Vietnamese neural machine translation integrated into a classification dictionary, in order to solve the following problem: most of the out-of-set word processing methods do not consider to the universal applicability of the language; finding other resources to alleviate such problems will affect the translation effect of the words around the out-of-set words, resulting in incomplete context translation; replacing out-of-set words by building a general dictionary will result in more than one word The translation problem is difficult to effectively deal with the out-of-collection words

[0005] The present invention studies the characteristics of different out-of-set words, proposes a classification idea, classifies out-of-set words to build a classification dictionary, and integrates it into the neural network Dealing with out-of-set words in the machine translation model solves the negative impact of out-of-set words in neural machine translation on the translation effect

Images