Machine translation model training method and device, medium and electronic equipment

Abstract

The invention belongs to the technical field of computers, is applied to the field of natural language processing, and particularly relates to a machine translation model training method and device, a computer readable medium and electronic equipment. The machine translation model training method comprises the steps of obtaining a training data set and a candidate data set; the machine translation model is trained through the corpora in the training data set, and word lists corresponding to all the corpora in the training data set are updated in the training process; updating the word list corresponding to the low-resource corpus by adopting the word list corresponding to the high-resource corpus, and calculating the updating completion degree of the word list corresponding to the low-resource corpus; and adding the low-resource corpus of which the update completion degree of the word list is greater than a preset threshold value into the training data set. Based on the method, the problem of data sparsity in the training process of the low-resource corpus can be avoided, the rich corpus of the high-resource corpus is fully utilized, and then the training effect of the to-be-trained language corresponding to the low-resource corpus is improved.

Description

technical field [0001] The application belongs to the field of computer technology, and in particular relates to a machine translation model training method, a machine translation model training device, a computer-readable medium and electronic equipment. Background technique [0002] The general neural machine translation model is based on the end-to-end Encoder-Decoder (encoding-decoding) framework, and generally requires large-scale parallel corpus for model training. However, due to the high cost of manual annotation, large-scale parallel corpus cannot be obtained for languages ​​with scarce corpus resources, which makes the translation quality of neural machine translation systems for languages ​​​​with scarce corpus resources poor. [0003] High-quality parallel corpora often exist only in a small number of languages. For some languages ​​that lack resources, it is difficult to find or obtain available parallel corpora from the Internet. Lack of corpus resources will ...

AI Technical Summary

Problems solved by technology

However, due to the high cost of manual labeling, large-scale parallel corpus cannot be obtained for languages ​​with scarce corpus resources, which makes the translation quality of neural machine translation systems for languages ​​​​with scarce corpus resources poor.

[0003] High-quality parallel corpora often exist only in a small number of languages. For some languages ​​that lack resources, it is difficult to find or obtain available parallel corpora from the Internet

Lack of corpus resources will cause the problem of data sparsity in the model training process, resulting in poor translation effect of the model for languages ​​with scarce corpus resources

Images