Mixed-corpus word segmentation method based on Bi-LSTM

Abstract

The invention discloses a mixed-corpus word segmentation method based on Bi-LSTM. The method includes the steps that training mixed corpus data is converted into character-level corpus data; characters of the corpus data are subjected to statistics, a character set is obtained, the characters are numbered, and a character number set is obtained; labels of the characters are subjected to statistics, a label set is obtained, the labels are numbered, and a label number set is obtained; corpus is segmented according to sentence length, obtained sentences are grouped according to the sentence length, and a data set is obtained; a sentence group is randomly selected from the data set without releasing, multiple sentences are extracted from the sentence group, the characters of each sentence formdata w, and the corresponding label set is y; the data w is converted into corresponding numbers and labels y to be sent into a model Bi-LSTM, and parameters of the depth learning model Bi-LSTM are trained; data to be predicted is converted into data matched with the depth learning model, the data is sent into the trained depth learning model Bi-LSTM, and a word segmentation result is obtained.

Description

technical field [0001] The invention belongs to the technical field of computer software, and relates to a Bi-LSTM-based mixed corpus word segmentation method. Background technique [0002] Bi-LSTM, the full name in English is: Bi-directional Long Short-Term Memory, which means in Chinese, bidirectional long and short-term memory neural network. [0003] Mixed corpus, in this patent, refers to training or prediction data that includes corpus data in at least two languages. [0004] Word segmentation (Word Segment) refers to marking the input continuous string into a continuous label sequence according to the semantic information. In this patent, it refers to segmenting the sequence data of Asian type characters (simplified Chinese, traditional Chinese, Korean and Japanese) into individual words, and using spaces as the segmentation between words. [0005] The word segmentation method of mixed corpus involves two aspects of professional knowledge: on the one hand, the data ...

AI Technical Summary

Problems solved by technology

[0015] Disadvantage 1: The detection granularity of multiple languages ​​is not easy to distinguish, and the word segmentation accuracy will be lost because a certain language is not detected

[0016] Disadvantage 2: The dictionary-based method is too dependent on the dictionary, and cannot identify unregistered words that have not appeared in the dictionary based on semantic information

[0017] Disadvantage 3: The current statistics-based methods are mainly HMM (Hidden Markov) model and CRF (Conditional Random Field) model, because of the complexity of calculation, it only considers the correlation between the current word and the previous word , the rest of the words are conditionally independent, which is inconsistent with the reality, so there is room for further improvement in the accuracy of word segmentation

Images