Data resampling method based on repeated editing nearest neighbor and clustering oversampling

Abstract

The invention relates to a data resampling method based on repeated editing nearest neighbor and clustering oversampling. The method comprises the steps: calculating the Euclidean distance between each to-be-sampled book and a nearby sample, selecting the sample with the smallest distance as the nearby sample of the to-be-sampled book, comparing whether the labels of the sample and the nearby sample are the same or not, and deleting the sample if the labels of the sample and the nearby sample are different; dividing the remaining samples into k clusters by using K-means, and filtering out theclusters of which the ratio of the number of majority class samples to the number of minority class samples is less than an imbalance rate threshold c; calculating an Euclidean distance between minority class samples in each cluster, constructing a distance matrix of the cluster, summing all off-diagonal elements in the matrix, and dividing the sum by the number of the off-diagonal elements to obtain an average distance of the cluster; calculating a sparse factor of each cluster; and calculating a resampling weight value of each cluster, and determining the number of generated new samples according to the weight values by using an SMOTE method. According to the method, the problem of class imbalance in the data is solved, so that the classifier can obtain a better classification effect.

Description

technical field [0001] The invention belongs to the technical field of data sampling, in particular to a data resampling method based on repeated editing nearest neighbor and clustering oversampling. Background technique [0002] In machine learning, an unbalanced training set may cause the trained model to be more inclined to identify samples as the majority class. Although the number of samples in the minority class is small, the importance is higher, and the cost of misclassifying the minority class is much higher than the cost of misclassifying the majority class. Therefore, addressing the class imbalance problem is very important for software defect prediction. Current methods for dealing with imbalanced datasets include algorithm-level methods, cost-sensitive methods, and data-level methods. Compared with algorithm-level methods that are bound to specific classifiers and cost-sensitive methods for specific problems, data-level methods are more general. Data preproce...

AI Technical Summary

Problems solved by technology

However, the SMOTE method amplifies the noise in the data and does not take into account the imbalance and small separation problems within the clusters

Images