LSTM-CNN-based word segmentation method

Abstract

The invention discloses an LSTM-CNN-based word segmentation method. The method comprises the steps of converting training corpus data into character-level corpus data; performing statistics on characters of the corpus data to obtain a character set and numbering the characters to obtain a character number set; performing statistics on character tags to obtain a tag set, and numbering the tags to obtain a tag number set; dividing corpora according to sentence lengths, and grouping obtained sentences according to the sentence lengths to obtain a data set comprising n groups of sentences; randomly selecting a sentence group from the data set without replacement, wherein the characters of each sentence form a piece of data w, and the corresponding tag set is y; converting the data w into the corresponding number and tag y to be input to a LSTM-CNN of a model, and training parameters of the deep learning model; and converting to-be-predicted data into data matched with the deep learning model, and inputting the data to the trained deep learning model to obtain a word segmentation result.

Description

technical field [0001] The invention belongs to the technical field of computer software, and relates to a word segmentation method based on LSTM-CNN. Background technique [0002] In natural language processing, Asian texts do not have natural space separators like Western texts. Many Western text processing methods cannot be directly used for processing Asian texts (Chinese, Korean, and Japanese), because Asian texts (Chinese , Korean and Japanese) must go through the process of word segmentation in order to maintain consistency with Western languages. Therefore, word segmentation is the basis of information processing in the processing of Asian text, and its application scenarios include: [0003] 1. Search engine: An important function of a search engine is to do full-text indexing of documents. Its content is to segment the text, and then form an inverted index with the word segmentation results of the document and the document. When users query, they also first query ...

AI Technical Summary

Problems solved by technology

The knowledge-based artificial neural network model has a small number of network layers in the actual application due to the gradient disappearance problem during model training, and the final word segmentation result has no obvious advantage

[0007] The word segmentation method based on the dictionary relies heavily on the dictionary library, the efficiency is relatively low, and cannot identify unregistered words; among the present invention, registered words refer to words that have appeared in the corpus vocabulary, and unregistered words refer to words that do not appear in word in corpus vocabulary

[0008] Based on the word frequency statistical word segmentation method (such as N-Gram), it can only associate the semantics of the first N-1 words of the current word, and the recognition accuracy is not high enough. When N increases, the efficiency is very low

And the recognition rate for unlogged is low

Images