Feature selection method based on document frequency of within-class and between-class and term frequency statistics

Abstract

The invention discloses a feature selection method based on document frequency of within-class and between-class and term frequency statistics. The document frequency, the word frequency, the between-class concentration ratio and the within-class dispersity of a feature term are comprehensively considered to construct a feature selection assessment function based on DFCTFS (Document Frequency of within-class and between-class and Term Frequency Statistics); and the original feature space, which is subjected to text preprocessing, of a training set uses a feature selection assessment function which is put forward by the invention to select a certain ratio of feature terms in each class of the training set to form the feature term bank of the class, and the feature term bank of the trainingset is the union set of each class of feature term bank of the training set. The invention puts forward the feature selection method based on the DFCTFS, feature terms which are intensively distributed in the certain class of document, are evenly distributed in the class of documents and frequently appear can be diagnostically selected, and a Chinese text classification effect can be improved.

Description

technical field [0001] The invention belongs to the technical field of Chinese text classification, and relates to a feature selection method, in particular to a feature selection method based on statistics of document frequency and word frequency within a class and between classes. Background technique [0002] The overall idea of ​​Chinese text classification is roughly: text preprocessing, feature selection, establishment of text representation model, classification using classification algorithm, classification model evaluation. Feature selection is a key step in Chinese text classification. It refers to selecting some important features from the high-dimensional original feature space to form a low-dimensional space, thereby improving classification accuracy and classification efficiency. [0003] Traditional feature selection methods include: document frequency (DF), mutual information (MI), information gain (IG), chi-square statistics (CHI), etc. The method of featur...

AI Technical Summary

Problems solved by technology

However, there are deficiencies in the traditional IG method. (1) The word frequency distribution of feature words in each category is not considered; (2) The negative correlation of feature words interferes; (3) It can only do global feature selection (referring to the training set All categories use the same feature set), and cannot do local feature selection (referring to each category in the training set has its own feature set)

Images