Multi-modal machine translation data enhancement method based on image description generation

Abstract

The invention discloses a multi-modal machine translation data enhancement method based on image description generation, which comprises the following steps of: training an attention mechanism-based image description generation model by using pre-trained image coding information and corresponding image description; encoding and decoding pictures in the existing multi-modal training data by using the trained image description generation model to generate a corresponding source language image description text; translating the generated source language image description text into a target language, and constructing pseudo data; and adding the constructed pseudo data into the multi-modal training data, fusing the picture information in the multi-modal training data with the source language description information, sending the fused information into a multi-modal machine translation model, and generating a target language translation assisted by the image context information in an autoregressive manner. The diversity of the pseudo data is enriched, the performance can be improved from knowledge refinement, and compared with a common data enhancement method adopting a random replacementmode and the like, the invention has great advantages.

Description

technical field [0001] The invention relates to a machine translation data enhancement technology, in particular to a multimodal machine translation data enhancement method based on image description generation. Background technique [0002] Machine Translation (MT for short) is an experimental discipline that uses computers to translate between natural languages. Using machine translation technology, a source language can be automatically converted into a target language. Machine translation, as a key technology to eliminate barriers in people's cross-language communication, has always been an important part of natural language processing research. Compared with human translation, machine translation is more efficient and lower cost, which is of great significance for promoting national unity and cultural exchanges. Machine translation technology can be summarized as two methods based on rationalism and methods based on empiricism. Since it was proposed in the 1940s, mach...

AI Technical Summary

Problems solved by technology

[0006] Aiming at the scarcity of existing multimodal translation training data and the insufficient effects of traditional data enhancement techniques, the present invention proposes a multimodal machine translation data enhancement method based on image description generation, which uses an image description generation model to construct a pseudo data, expand the training data

Images