Industry text entity extraction method based on distributed platform

Abstract

The invention discloses an industry text entity extraction method based on a distributed platform. The method comprises the following steps: obtaining a relation feature model by using a deep learningneural network training text data set, generating multiple elastic distributed relation feature data set RDD through the extracted relation features; extracting class features from a data set in theRDD through a class feature model obtained through improved non-linear SVM classification algorithm; finding a corresponding context entity model according to the extracted class feature, and extracting entity data in the text of the corresponding class feature through the trained entity model; judging whether the text amount of the corresponding context exceeds a set threshold, retraining the context entity model if the text amount exceeds the threshold, and extracting the entity data in the text of the corresponding class feature by using the retrained entity model; otherwise, saving the text entity feature and the text data. The text feature entities under different contexts can be processed, the entity extraction efficiency and the entity extraction accuracy rate are effectively improved.

Description

technical field [0001] The invention relates to a method for extracting text entities, in particular to a method for extracting industry text entities based on a distributed platform. Background technique [0002] Traditional text extraction methods use pattern matching relationship extraction methods, dictionary-driven relationship extraction methods, and machine learning-based relationship extraction methods. Most of these methods first extract words with relatively high word frequencies in the text as effective entities through word segmentation. These methods are suitable for the scene where the entities in the text are relatively single, but in different contexts, these methods cannot effectively distinguish entities in different contexts, and will segment and merge entities that do not need to be segmented or merged incorrectly. [0003] At the same time, it is difficult for traditional detection methods to extract words that have not appeared in previous texts through...

AI Technical Summary

Problems solved by technology

[0005] Chinese patent document CN2017100036859 discloses an online TCM text named entity recognition method based on deep learning. The entity extraction method enriches the text training sample set by crawlers, and at the same time extracts text features by neural network, which can extract samples to a certain extent. The accuracy of the entity, but as the training sample increases, the corresponding extracted entity model also increases, and the training time will gradually increase, and the feature extraction time will also increase.

Images