A Method for Removing Redundancy of Information from Sample Sets

Abstract

The present invention provides an information de-redundancy method for a sample set. The method includes: obtaining samples to be processed and corresponding trainable labels to obtain the original sample set to be processed; Feature extraction, to obtain the feature vector set of the original sample set; input the feature vector set to the learnable sample selector model, perform sample selection on the feature vector set, and obtain a representative feature vector subset according to the preset threshold; obtain the feature vector The original sample corresponding to the quantum set is used as a sub-sample set after removing redundant information. The technical solution of the present invention can efficiently streamline the original sample set, remove redundant information and retain valuable information samples, and improve the training efficiency of the algorithm on the sample set.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to an information de-redundancy method for a sample set. Background technique [0002] With the development of deep learning technology, machine learning methods based on large-scale data sets have been continuously proposed. However, in reality, large-scale data sets often have a large amount of redundant information and data, for example, excessive single-category samples, repeated or approximate samples, etc.; on the other hand, large-scale data sets make the training process of machine learning models more complex. A lot of computing power and computing time consume a lot of resources. Therefore, in the face of large-scale training tasks in different scenarios, for example, ultra-large-scale computational vision classification tasks often use tens of millions of image samples for training, or ultra-large-scale natural language processing tasks often use hundreds of mill...

AI Technical Summary

Problems solved by technology

Considering the large scale of data sets, the complex relationship between samples, and the comparison and analysis of samples based on pairs relies on huge computing power, there is currently no technical solution that can be directly used to deal with the information redundancy of large-scale data sets

Images