A Text Feature Selection Method Based on Imbalanced Dataset

Abstract

The invention relates to a text feature selection method based on unbalanced data sets. Feature sets of unbalanced documents are calculated on a computer; and modelling is carried out by selecting a classification algorithm model. The text feature selection method specifically comprises the following steps of: (1), dividing the data sets into majority classes and minority classes, stipulating the minority classes as positive classes represented by ci, and stipulating the majority classes as negative classes represented by a formula shown in the specification; (2), pre-processing texts in the data sets, and executing operations, such as word segmentation and removing of stop words, so as to form a set T of features t; (3), respectively calculating parameters A, B, C, D and N corresponding to various features t in the unbalanced class documents; (4), respectively calculating new X2(t,ci) of various features t under different classes in the unbalanced class documents; (5), respectively setting threshold values for screening features in the unbalanced class documents, according to the X2(t,ci) calculated by various features, arranging according to the size order; and taking out a feature set T' including an appointed number of features according to the classes; and (6), selecting a proper classification algorithm model (such as a decision tree, a support vector machine and Bayes) to model according to the feature set T' after the features are selected.

Description

technical field [0001] The invention relates to the classification problem of unbalanced data sets in data mining and the field of text feature selection, and is an improved CHI method suitable for unbalanced text classification. Background technique [0002] With the rapid development of the Internet, the number of electronic documents has increased dramatically, making text classification a core technology for processing large amounts of text data. Due to the large number of dimensions that contain features in text, feature selection techniques are often used in data dimensionality reduction. An effective feature selection algorithm can not only reduce the dimensionality of features, remove redundant features, but also avoid overfitting of the classifier, thereby improving the classification accuracy of the model. [0003] Currently commonly used feature selection algorithms mainly include mutual information (Mutual Information, MI), information gain (Information Gain, IG...

AI Technical Summary

Problems solved by technology

These algorithms have their own advantages and disadvantages. For example, mutual information (MI) tends to select low-frequency features, information gain (IG) has a large amount of calculation, and is suitable for global feature selection rather than specific to a certain category; CHI is a stable and efficient algorithm. Feature selection algorithm, showing better accuracy in experiments

There are also some effective algorithms for the text imbalance problem, such as: CTD, SCIW, etc., but the disadvantage of these algorithms is that they do not fully consider all the important factors that affect feature selection hidden in the imbalanced text dataset.

Images