Text categorization feature selection and weight computation method based on field knowledge

Abstract

The invention relates to the artificial intelligence technical field, in particular to a text classification feature selection and weigh calculation method based on field knowledge. The method combines sample statistics and field glossaries to construct a filed classification feature space, utilizes internal knowledge relations in the field, calculates the similarity between the glossaries, and then adjusts the corresponding feature weight of classification feature vectors. Moreover, the method adopts a learning algorithm of a support vector machine to construct a field text classification model and then realize field text classification. As shown by text classification laboratory results of the Yunan tourist field and the non-tourist field, the classification accuracy of the method is improved by 4 percent compared with the text classification effect of the improved TFIDF feature weigh method.

Description

technical field [0001] The invention relates to the technical field of artificial intelligence, in particular to a text classification feature selection and weight calculation method based on domain knowledge. Background technique [0002] Text classification is a hot issue in the current natural language processing research. How to identify whether a text belongs to a specific field of text is a key issue in the current research on vertical search engines and question answering systems. Usually in text classification, feature selection is the most important part, which directly affects the accuracy of text classification. Conventional feature selection methods mostly use various evaluation functions such as document frequency (Document Frequency, DF), information gain (Information Gain, IG), mutual information (Mutual Information, MI), statistics (CHI), etc. for feature extraction. These feature selection methods are all based on statistical algorithms. A large amount of c...

AI Technical Summary

Problems solved by technology

These feature selection methods may cause some selected statistical features to contribute less to the classification, which will reduce the accuracy of the classification; and for domain text classification, some domain terms often appear in the text, and these domain terms have great influence on the domain text. The classification has a high degree of discrimination. However, using conventional feature selection methods, these features that play an important role in the classification effect may obtain lower weights, or even be removed as noise, which will greatly affect the classification accuracy.

Images