Text classification method and system based on class perception feature selection framework

Abstract

The invention discloses a text classification method. The text classification method comprises the following steps: preprocessing a plurality of category clusters, obtaining a set of feature words, calculating a class relevancy score and a class discrimination score between each feature word and each class cluster; allocating each feature word to a category cluster with a corresponding highest category relevancy value; and reordering words in each category cluster, selecting feature subsets from each category cluster, reordering the feature subsets in the total feature set to obtain a final feature set, inputting the to-be-classified text subjected to vector representation into a classifier, outputting a classification result and the like. According to the text classification method, the data processed by the classifier contains the respective properties of different types of clusters, the intra-class correlation degree and the inter-class distinguishing degree of the feature words andother information at the same time, thus overcoming the one-sidedness of the prior art, and being able to achieve a better text classification effect. The text classification method is widely appliedto the technical field of text classification.

Description

technical field [0001] The invention relates to the technical field of text classification, in particular to a text classification method and system based on a class-aware feature selection framework. Background technique [0002] Text classification technology is widely used in practical application scenarios such as information retrieval, text mining, public opinion analysis, and spam identification. Most text classification technologies are implemented based on classifiers, and the training set used to train the classifiers contains as many as hundreds of thousands of feature words. Therefore, feature extraction is an important part of text classification technology. [0003] The purpose of feature extraction is to extract feature words that are more capable of identifying cluster categories. Most of the existing feature extraction methods extract feature words that can best identify cluster categories from a global perspective. Taking information gain as an example, its...

AI Technical Summary

Problems solved by technology

However, this approach does not perform well in discriminating between imbalanced datasets

This is because, when the data set has a large number of categories and is unbalanced data, the traditional feature extraction method only considers the features with the highest global class discrimination, resulting in sparse features extracted for some small sample category clusters, resulting in The classification accuracy rate for small sample clusters is reduced

At the same time, the feature extraction methods that the existing text classification methods rely on only consider the class inclination of the feature words but not the inter-class discrimination ability of the feature words. This one-sidedness limits the classification accuracy of the existing text classification methods.

Images