An Adaptive Feature Selection Method Based on Chi-Square Statistics

Abstract

The invention discloses a chi square statistic based self-adaption feature selection method and relates to the field of computer text data processing. Firstly, preprocessing of a training text set and a test text set is performed and comprises participle processing and stop word processing, then, self-adaption text feature selection based on chi square statistic is performed, a word frequency factor and interclass variance are defined and introduced into a CHI algorithm, an appropriate scaling factor is added for the CHI algorithm, finally, the scaling factor is automatically adjusted in combination of classical KNN algorithm evaluation indexes, improved CHI is adapted to different text corpora, and higher classification accuracy is guaranteed. An experimental result proves that by comparison with a conventional CHI method, the classification accuracy of a balanced corpus and a non-balanced corpus is improved.

Description

Technical field [0001] The present invention relates to the field of computer text data processing, in particular to a method based on chi-square statistics (χ 2 , CHI) adaptive text feature selection method. Background technique [0002] In today's era of big data, mining the potential value of data is essential. As a technology to discover the potential value of data, data mining has attracted great attention. Text data accounts for a large proportion of big data, and text classification, as a data mining method for effective organization and management of text data, has gradually become a hot spot. It is widely used in information filtering, information organization and management, information retrieval, digital libraries, and spam filtering. Text Classification (TC) refers to the process of automatically classifying unknown texts into one or more categories based on their content under a predetermined category system. Commonly used text classification methods, such as K-Ne...

AI Technical Summary

Problems solved by technology

[0004] As one of the commonly used text feature selection methods, the CHI method has the characteristics of simple implementation and low time complexity; but there are also many shortcomings, so that the classification effect is not ideal

The shortcomings of the CHI algorithm mainly include two aspects: first, CHI only considers the document frequency of feature items, and ignores the word frequency of feature items, resulting in the weight of low-frequency words being amplified; The weight of feature items that are many and often appear in other classes

Aiming at the deficiencies of the CHI algorithm, many researchers have improved it, and summarized the improvement methods into the following two aspects: First, several adjustment parameters are introduced to reduce the reliance on low-frequency words, but this method does not consider the relationship between feature items and categories. Positive and negative correlation between

Images