A Chinese word segmentation method based on bi-directional long-short time memory network model

Abstract

The invention discloses a Chinese word segmentation method based on a bi-directional long-short time memory network model. Firstly, the data set of any field is obtained, and the data set is divided into training set and test set. Then, the training set is preprocessed, the preprocessed training set and the open data set msr of Microsoft Research Asia are embedded separately, and the processed training set and data set msr are input into Bi-LSTM neural network model is trained to get the training set model X_Bi-LSTM Model and msr_Bi-LSTM model, and then using X_Bi-LSTM model, msr_Bi-LSTM modelpredicts the label of the test suite, and weights the forecasting probabilities of the two models, the probability of each Chinese character label after combination is obtained, then, the Viterbe algorithm is used to calculate the probability of each label after the combination to obtain the final probability of each Chinese character belonging to each label, and the label with the maximum probability value belongs to each Chinese character as the label of each Chinese character, thereby completing Chinese word segmentation. The invention can obtain better word segmentation result, and improves the accuracy of word segmentation.

Description

technical field [0001] The invention relates to a Chinese word segmentation method based on a bidirectional long-short-term memory network model, belonging to the field of natural language processing. Background technique [0002] In Chinese, there are no separators between words, and the words themselves lack obvious morphological marks. Therefore, the unique problem of Chinese information processing is how to divide Chinese word strings into reasonable word sequences, that is, Chinese word segmentation. Word segmentation is the first step in Chinese natural language processing. This is an important feature of natural language processing systems different from other languages, and it is also an important factor affecting the application of natural language processing in Chinese information processing. In recent years, many scholars at home and abroad have done a lot of research work in the field of Chinese word segmentation and achieved certain research results. However, f...

AI Technical Summary

Problems solved by technology

The neural network-based word segmentation method uses the neural network to automatically learn data features, avoiding the limitations of the traditional word segmentation method due to artificial settings, but the neural network model is greatly affected by the size of the context window, and it is easy to introduce too many features when the window is large The impurity information brought is prone to overfitting problems, and the traditional cyclic neural network (such as RNN) only relies on the above information in the sentence order, and cannot use the future text information in the sentence

Images