A subject term extraction method and system based on sequence labeling model

Abstract

The invention discloses a subject term extraction method and system based on a sequence labeling model, and belongs to the technical field of data extraction. The method includes the steps that firstly, labeling and class label setting are performed on subject terms in training linguistic data to obtain a labeling sequence, a subject term extraction model is obtained through training with the training linguistic data serving as an observation sequence and the labeling sequence serving as a state sequence, and the subject terms in the linguistic data to be extracted are preliminarily extracted with the model serving as an extractor; then, preliminary extraction results are screened according to the similarity between the subject terms to obtain the true subject terms belonging to corresponding subject fields. According to the extraction method and system, when the subject terms are extracted, by performing labeling on the subject terms in a small quantity of training linguistic data, rapid and accurate extraction of the subject terms in the linguistic data is achieved, meanwhile, existing knowledge hierarchy structures of the subject fields can be gradually improved, and the defects of a traditional subject term extraction method are overcome.

Description

technical field [0001] The invention relates to the technical field of data extraction, in particular to a method and system for extracting subject terms based on a sequence labeling model. Background technique [0002] Subject terminology is the most basic unit in a field. In order to better describe the knowledge system in this field, all subject terms in this field need to be obtained. With the development of the Internet and information technology, knowledge in the same field is expanding rapidly, new fields are emerging one after another, and the extraction and application of subject terms are getting more and more attention. For example, many online education companies (such as Mingbo Education, etc.) Using the corpus that the user is reading, the subject terms in the corpus will be marked for the user, and relevant educational resources can be recommended for the user based on these subject terms, which better meets the needs of the user. With the continuous increase...

AI Technical Summary

Problems solved by technology

[0005] These extraction methods mentioned above save a lot of labor costs, but these methods do not consider the context information of the term occurrence, and require a large amount of data to train the extractor, and some emerging fields lack relevant corpus, so the extraction results are relatively poor; at the same time, these The methods are all done on the basis of English and are not suitable for various languages; finally, after extracting new subject terms, no further term denoising is considered, and there are still some irrelevant words in the terms

Images