A mixed corpus named entity recognition method based on bi-lstm-cnn

Abstract

The invention relates to a Bi-LSTM-CNN-based mixed corpus named entity recognition method. This method converts the labeled training mixed corpus data into character-level mixed corpus data in the training phase, then trains the deep learning model based on Bi-LSTM-CNN, and converts the unlabeled test mixed corpus data into characters in the prediction phase Level mixed corpus data, and then use the deep learning model trained in the training phase to make predictions. The present invention uses character-level rather than word-level vectors, which can avoid the impact of participle accuracy, and can also avoid the problem of unregistered words; in addition, it adopts a combination model of bidirectional long-term short-term memory neural network Bi-LSTM and convolutional neural network CNN Compared with the traditional algorithm, the accuracy is much improved; directly using the mixed corpus for model training does not need to detect and separate each language of the mixed corpus, and finally achieves the purpose of identifying the mixed corpus.

Description

technical field [0001] The invention belongs to the field of information technology, and in particular relates to a Bi-LSTM-CNN-based mixed corpus named entity recognition method. Background technique [0002] Named entity recognition refers to the process of identifying specified entity nouns with specific meanings for a given dataset. The practical scenarios of the named entity recognition method include: [0003] Scenario 1: Event detection. Place, time, and person are several basic components of time. When constructing an event summary, relevant persons, places, units, etc. can be highlighted. In the event search system, related people, time, and places can be used as index keywords. The relationship between several components of an event describes the event in more detail at the semantic level. [0004] Scenario 2: Information retrieval. Named entities can be used to enhance and improve the effect of the retrieval system. When the user enters "major", it can be fou...

AI Technical Summary

Problems solved by technology

[0012] Disadvantage 1: The detection granularity of multiple languages ​​is not easy to distinguish, and there is a loss of participle accuracy because a certain language is not detected

[0013] Disadvantage 2: HMM (Hidden Markov) and CRF (Conditional Random Field) methods based on word frequency statistics can only relate to the semantics of the previous word of the current word, and the recognition accuracy is not high enough, especially the recognition rate of unregistered words is low;

[0014] Disadvantage 3: The method based on the artificial neural network model has the problem of gradient disappearance during training, and in actual applications, the number of network layers is small, and the final named entity recognition results have no obvious advantages

Images