Parallelization word alignment method based on bilingual word embedding technology

Abstract

The invention discloses a parallelization word alignment method based on a bilingual word embedding technology. The method comprises the steps of obtaining a bilingual word vector table through an MPS-Neg bilingual word embedding technology on a Spark platform, obtaining a word alignment model through a bilingual word vector table, then conducting a distributed word alignment task, updating the bilingual word vector table through the word alignment result and MPS-Neg, and repeating the operations of word alignment and bilingual word vector table updating till the specific iteration frequency is reached. The parallelization word alignment method based on the bilingual word embedding technology solves the problem that an existing word alignment method cannot well adapt to a large-scale corpus word alignment task.

Description

technical field [0001] The invention belongs to the field of computer natural language processing and parallel computing, and specifically relates to a word alignment method based on bilingual word embedding technology implemented on a Spark platform. Background technique [0002] As one of the key technologies in the field of machine translation, word alignment plays an important role in many natural language processing tasks. In 1993, Brown and others proposed to divide the basic equation of machine translation into language model and translation model, and proposed five translation models from IBM 1 to 5. Since then, the IBM series of models have become the de facto standard for word alignment models, and most of the subsequent word alignment research is based on this series of models. Among them, the word alignment method based on the HMM model has improved the IBM model 2, which is commonly used in practical applications HMM word alignment model to replace IBM Model 2....

AI Technical Summary

Problems solved by technology

However, a large-scale word translation probability table needs to be generated in the traditional word alignment algorithm. This data structure will generate huge network overhead during the parallelization process and slow down the overall word alignment efficiency.

Images