Method and device for constructing text classifier with reference to external knowledge

Abstract

The invention provides a method and equipment for constructing a text classifier by referencing external knowledge. The method comprises the steps of: inputting a label text set; extracting internal characteristics of the label text set; constructing external characteristics of the label text set by referencing an external knowledge source (such as a dictionary); comprehensively considering the internal characteristics and the external characteristics of the label text set, and selecting training texts from the label text set; and learning the generation of the text classifier by using the selected training texts. According to the invention, sample distribution deviation generated by the label text set can be possibly regulated by the external characteristics automatically generated by the external knowledge source, and therefore, the finally trained classifier has better generalization capability and robustness.

Description

technical field [0001] The present invention relates generally to information retrieval and text classification. More specifically, the present invention relates to methods and devices for constructing text classifiers with reference to external knowledge. Background technique [0002] With the rapid development of electronic office and the Internet, the amount of electronic text information has exploded, and large-scale automatic information processing has become a necessary means and challenge for people to make better use of this large-scale information. [0003] Information retrieval refers to the process and technology of organizing information in a certain way and finding relevant information according to the needs of information users. Automatic text classification is one of the main supporting technologies for information retrieval. Its basic purpose is to divide text into predefined categories, which is an effective means to help people search, query, filter and ut...

AI Technical Summary

Problems solved by technology

Therefore, the final text classifier constructed must be completely influenced by the given labeled text, resulting in poor generalization and robustness of this classifier

[0012] Although there are other training text selection methods in the prior art, the current training text selection methods are mainly realized by using the internal knowledge of a given labeled text set, that is, the features and weights used are completely dependent on the given Determine the data distribution of the labeled text set, so that the selected training text will have a strong bias

This bias will be propagated to the classification orientation of the final constructed classifier, which greatly affects its generalization ability and robustness, and finally causes the performance of the classifier to be unsatisfactory.

Images