A mixed corpus word segmentation method based on lstm-cnn

Abstract

The invention discloses a mixed corpus word segmentation method based on LSTM-CNN. The method is as follows: converting the training mixed corpus data into character-level mixed corpus data; counting the characters of the mixed corpus data to obtain a character set and numbering each character to obtain a character number set; counting character labels to obtain a label set, and calculating the labels numbering to obtain a set of label numbers; divide the corpus according to the sentence length, and group the obtained sentences according to the sentence length to obtain a data set; randomly select a sentence group from the data set without replacement, and extract multiple sentences from it. The characters constitute a data w, and the corresponding label set is y; the data w is converted into the corresponding number and label y and sent to the model LSTM‑CNN to train the parameters of the deep learning model; the mixed corpus data to be predicted is converted into The data matched by the model is sent to the trained deep learning model to obtain word segmentation results.

Description

technical field [0001] The invention belongs to the technical field of computer software, and relates to a mixed corpus word segmentation method based on LSTM-CNN. Background technique [0002] Mixed corpus, in this patent application, refers to training or prediction data that includes corpus data in at least two languages. [0003] Word segmentation (Word Segment) refers to marking the input continuous string into a continuous label sequence according to the semantic information. In this patent application, the sequence data of Asian characters (simplified Chinese, traditional Chinese, Korean and Japanese) is divided into individual words, and spaces are used as the division between words. Registered words, in this patent, refer to words that have already appeared in the corpus vocabulary. Unregistered words refer to words that do not appear in the corpus vocabulary. [0004] The word segmentation method of mixed corpus involves two aspects of professional knowledge: on...

AI Technical Summary

Problems solved by technology

[0009] Disadvantage 1: The detection granularity of multiple languages ​​is not easy to distinguish, and there is a loss of participle accuracy because a certain language is not detected

[0010] Disadvantage 2: The dictionary-based method is too dependent on the dictionary, and cannot identify unregistered words that have not appeared in the dictionary based on semantic information

[0011] Disadvantage 3: The current statistics-based methods are mainly HMM (Hidden Markov) model and CRF (Conditional Random Field) model, because of the degree of calculation, it only considers the correlation between the current word and the previous word , the rest are conditionally independent, which is inconsistent with the reality, so there is room for further improvement in the accuracy of word segmentation

Images