Ultra-high-dimensional data dimension reduction algorithm based on information entropy

Abstract

The invention provides an ultra-high-dimensional data dimension reduction algorithm based on information entropy, belongs to the field of high-dimensional data preprocessing, and aims to solve the following problems existing in actual application of a conventional PCA algorithm: when data dimensionality (feature) is high enough, all data characteristic read values cannot be stored in memory in one time for analysis and calculation; a partitioning processing method, without depending on a cloud platform and a distributed type calculation platform, is used for experiment; however, the method is long in time consumption and cannot satisfy actual application requirement. On the basis, the thought of the information entropy is introduced for improving the PCA algorithm; the improved algorithm can be used for processing the ultra-high-dimensional data dimension reduction; and experiment results prove that the operating time consumption of the improved algorithm is shortened by 60 times compared with that of the partitioning processing algorithm when it is ensured that the same proportion of the original data information is reserved.

Description

technical field [0001] The invention belongs to the field of high-dimensional data preprocessing, and more specifically, is a dimensionality reduction algorithm for ultra-high-dimensional data based on information entropy improvement. Background technique [0002] With the rapid development of information science and technology, the representation of information is becoming more and more comprehensive, it is becoming easier for people to obtain data, and the data objects of concern are becoming more and more complex. The industry has the most urgent demand for data analysis and processing technology, especially for high-dimensional data. analysis and processing techniques. Directly dealing with high-dimensional data will face the following difficulties: curse of dimensionality, empty space, ill-posedness, and algorithm failure. Aiming at the problem that the data feature dimension is too high, the memory is limited, and cannot be read into the memory for analysis and calcul...

AI Technical Summary

Problems solved by technology

However, the results show that the running time is too long to meet the application requirements. On this basis, information entropy is introduced, and feature screening is performed first, which greatly reduces the number of features, and then dimensionality reduction is performed. The specific process is as follows: figure 2 As shown, the specific algorithm is as image 3 As shown, the running time of the whole process is reduced several times, and the dimensionality reduction results retain most of the principal components, which can still meet the application requirements.

Images